Determine support path for RPMs that install into `/opt`, `/usr/local` etc

[cgwalters]

RPMs like Puppet and Google Chrome go in /opt.

For the package layering case, in this PR we made it an obvious error.

The core problem here is: OSTree defines /opt (really /var/opt) as system administrator territory - it's never rolled forward/backwards etc. Content in there isn't protected by the ro bind mount covering /usr right now.

One approach would be to - for these RPMs, automatically rewrite the content into /usr. Chrome I don't believe actually stores any persistent state in /var, so simply rewriting /opt/google/chrome -> /usr/lib/opt/google/chrome with a compatibility symlink would likely work.

Puppet probably stores state in /var and hence would be harder.

[copumpkin]

I'd support the /usr/lib/opt approach!

[copumpkin]

@cgwalters any hints about what I can do in the meantime to get a sensible /opt to work? I was thinking of manually putting things into /usr/lib/opt during my install script, and then setting up a bind mount to /opt in my /etc/fstab, but I don't know if that'll be incompatible with the existing mount mechanism under OSTree. Is there a better way to weasel my way into the standard OSTree mount mechanism?

[copumpkin]

(the issue here is proprietary software with hardcoded assumptions about being in /opt, by the way; otherwise I'd just patch it)

[cgwalters]

We can change rpm-ostree to do this postprocessing automatically.

[copumpkin]

@cgwalters yeah, sorry, I realize that; was just asking about what to do between now and having native support for /opt in rpm-ostree

[cgwalters]

The only thing I can think of is to postprocess the RPM outside of rpm-ostree, basically RPMs are just cpio blobs with metadata and scripts. One could rpm2cpio foo.rpm to unpack it, then mv opt/foo usr/lib/foo, then turn it into a "source" tarball and make a binary-only RPM from that.

It might sound complex at first but it'd probably be a 4 line script + 15 line spec file.

[copumpkin]

@cgwalters but then I also need something to add an /opt bind mount back at runtime, right? The RPMs are still hardcoded to think they're in /opt. Would I have my RPM-wrangler add an entry to fstab as well?

[cgwalters]

Add a systemd tmpfiles.d file which creates the symlink /opt/foo -> /usr/lib/foo. This is pretty similar to what rpm-ostree automatically generates for directories in /var. See also https://github.com/projectatomic/rpm-ostree/blob/master/src/app/tmpfiles-ostree-integration.conf#L18

[copumpkin]

Oh, excellent, thanks!

[copumpkin]

Ah, it looks like https://github.com/projectatomic/rpm-ostree/blob/master/src/libpriv/rpmostree-postprocess.c#L102 is already putting a symlink for /opt -> /var/opt, so I think I need to use an L+ tmpfiles entry.

[copumpkin]

Turns out even with L+, systemd-tmpfiles-setup.service doesn't have the power to unlink /opt. Not does root. I'm confused what kind of thing /opt is to prevent me from changing it like this, given that it's on a RW filesystem and isn't a special mount point or anything weird.

[cgwalters]

I don't think you want to replace the full /opt symlink, just create a sub-symlink inside it, no? I.e your tmpfiles.d snippet should be

L /opt/appname - - - - ../usr/share/appname

or so.

In the current model, the /opt -> /var/opt symlink is part of the ostree commit (see the init_rootfs() function), it isn't created on boot. The reason you're getting EPERM is because of the immutable bit.

[copumpkin]

Ah yes, I just figured out the immutable bit on /. The reason I was trying to redirect all of /opt, rather than just subdirectories within it is that I have a few different packages that want to put things into /opt, and I was just going to have a general purpose "/opt fixer" in my postprocessing script rather than teaching my individual RPMs how to add to tmpfiles.

So my postprocess script currently just copies /opt into /usr/lib/opt and adds a tmpfiles.d entry that (tries to) symlink /opt to /usr/lib/opt.

I can do it for individual packages, but that starts putting a lot more burden on different subprojects, so I was hoping to avoid that 😔