Determine support path for RPMs that install into /opt, /usr/local etc

[cgwalters]

RPMs like Puppet and Google Chrome go in /opt.

For the package layering case, in this PR we made it an obvious error.

The core problem here is: OSTree defines /opt (really /var/opt) as system administrator territory - it's never rolled forward/backwards etc. Content in there isn't protected by the ro bind mount covering /usr right now.

One approach would be to - for these RPMs, automatically rewrite the content into /usr. Chrome I don't believe actually stores any persistent state in /var, so simply rewriting /opt/google/chrome -> /usr/lib/opt/google/chrome with a compatibility symlink would likely work.

Puppet probably stores state in /var and hence would be harder.

[copumpkin]

I'd support the /usr/lib/opt approach!

[copumpkin]

@cgwalters any hints about what I can do in the meantime to get a sensible /opt to work? I was thinking of manually putting things into /usr/lib/opt during my install script, and then setting up a bind mount to /opt in my /etc/fstab, but I don't know if that'll be incompatible with the existing mount mechanism under OSTree. Is there a better way to weasel my way into the standard OSTree mount mechanism?

[copumpkin]

(the issue here is proprietary software with hardcoded assumptions about being in /opt, by the way; otherwise I'd just patch it)

[cgwalters]

We can change rpm-ostree to do this postprocessing automatically.

[copumpkin]

@cgwalters yeah, sorry, I realize that; was just asking about what to do between now and having native support for /opt in rpm-ostree

[cgwalters]

The only thing I can think of is to postprocess the RPM outside of rpm-ostree, basically RPMs are just cpio blobs with metadata and scripts. One could rpm2cpio foo.rpm to unpack it, then mv opt/foo usr/lib/foo, then turn it into a "source" tarball and make a binary-only RPM from that.

It might sound complex at first but it'd probably be a 4 line script + 15 line spec file.

[copumpkin]

@cgwalters but then I also need something to add an /opt bind mount back at runtime, right? The RPMs are still hardcoded to think they're in /opt. Would I have my RPM-wrangler add an entry to fstab as well?

[cgwalters]

Add a systemd tmpfiles.d file which creates the symlink /opt/foo -> /usr/lib/foo. This is pretty similar to what rpm-ostree automatically generates for directories in /var. See also https://github.com/projectatomic/rpm-ostree/blob/master/src/app/tmpfiles-ostree-integration.conf#L18

[copumpkin]

Oh, excellent, thanks!

[copumpkin]

Ah, it looks like https://github.com/projectatomic/rpm-ostree/blob/master/src/libpriv/rpmostree-postprocess.c#L102 is already putting a symlink for /opt -> /var/opt, so I think I need to use an L+ tmpfiles entry.

[copumpkin]

Turns out even with L+, systemd-tmpfiles-setup.service doesn't have the power to unlink /opt. Not does root. I'm confused what kind of thing /opt is to prevent me from changing it like this, given that it's on a RW filesystem and isn't a special mount point or anything weird.

[cgwalters]

I don't think you want to replace the full /opt symlink, just create a sub-symlink inside it, no? I.e your tmpfiles.d snippet should be

L /opt/appname - - - - ../usr/share/appname

or so.

In the current model, the /opt -> /var/opt symlink is part of the ostree commit (see the init_rootfs() function), it isn't created on boot. The reason you're getting EPERM is because of the immutable bit.

[copumpkin]

Ah yes, I just figured out the immutable bit on /. The reason I was trying to redirect all of /opt, rather than just subdirectories within it is that I have a few different packages that want to put things into /opt, and I was just going to have a general purpose "/opt fixer" in my postprocessing script rather than teaching my individual RPMs how to add to tmpfiles.

So my postprocess script currently just copies /opt into /usr/lib/opt and adds a tmpfiles.d entry that (tries to) symlink /opt to /usr/lib/opt.

I can do it for individual packages, but that starts putting a lot more burden on different subprojects, so I was hoping to avoid that 😔

[copumpkin]

Interesting effect of having /opt -> /var/opt and then using tmpfiles.d to populate /opt: I need to ensure that my tmpfiles.d conf file sorts lexicographically later than rpm-ostree-autovar.conf (or tmpfiles-ostree-integration.conf, which also defines the same entry) because otherwise tmpfiles.d tries to create my /opt/myapp while /opt is still a broken link, before /var/opt has been created.

[cgwalters]

Yeah, this would all be better if rpm-ostree handled it internally. I'll get around to this at some point if no one else does.

[copumpkin]

How would you envision the "ostree-native" version of this working? My concern with the obvious thing is that a lot of programs use it to store all their files, including ones they expect to mutate. Simply making a symlink from /opt/foo to /usr/lib/opt/foo will work until the program tries to mutate one of those, and then we'll run into other issues. What I'm doing today with my horrible hack is copying stuff from /usr/lib/opt into /opt, which ostree sets up to point to /var/opt so it's mutable anyway. This leads to some duplication and more runtime mutability than I'd like, but I don't have a cleaner solution in mind other than handling /opt entries on a case-by-case basis.

[tobiashochguertel]

@cgwalters I would suggest an migration of /opt to /usr/lib/opt 233#issue-140670748. From treefile we can set remove-from-packages to remove files.

I'm not sure if the way to say we drop all files from /opt/ except for the following files and directories which are definied from treefile option include-from-opt (Array)*.The Entries from include-from-opt are moved to /usr/lib/opt. The previous behavior of rpm-ostree does not change (version compatibility).

*New treefile option introduction: include-from-opt (Array).

What do you think?

[tobiashochguertel]

I hope this will help others to workaround until there is an solution, thanks to @cgwalters and @copumpkin

Rebuild puppet-agent rpm with new path

Regular Expression to change /opt/ -to-> /var/opt:

:%s/"\/opt/"\/var\/opt/g

Copy Conent from /opt/ to the new location: /var/opt/

cp -rfa /opt/puppetlabs /var/opt/puppetlabs
rpmrebuild -e puppet-agent

   opens vi/vim, run the regular expression... and write close the change ( !wq ).

ls -lha /root/rpmbuild/RPMS/x86_64/puppet-agent-1.4.1-1.el7.x86_64.rpm

yum install createrepo
createrepo --database /root/customrpms

File: sig-atomic-buildscripts/puppetlabs-pc1.repo

[puppetlabs-pc1]
name=Puppet Labs PC1 Repository el 7 - $basearch
#baseurl=http://yum.puppetlabs.com/el/7/PC1/$basearch
baseurl=http://192.168.99.102:6060
gpgcheck=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-puppetlabs-PC1

Start Custom Yum repository:

• http-server: ( npm install -g http-server )

cd /root/customrpms/
http-server -p 6060

via rpm-ostree upgrade I've got the changes in ostree and /opt/puppetlabs -> /var/opt/puppetlabs e.g. I can run now puppet agent on centos-atomic host. The above documented solution is upgradeable. I have no blog or website where I can put this and link from here.

HTH
Tobias

[tobiashochguertel]

by the way this does not work with rpm-ostree -> I get the following log error: https://nopaste.me/view/dd563d20

A test installation of my rpmrebuild rpm on my ostree-build system was just fine, files & binaries are located in /var/opt/puppetlabs/. With sudo rpm-ostree-toolbox treecompose -c ~/sig-atomic-buildscripts/config.ini --ostreerepo /srv/repo/ -p hochguertel.local and testing the custom ostree via virtualmachine shows me that there something went wrong. /var/opt/puppetlabs and directory structure and even some files are there - but also files are missing, binaries are missing.

Can someone help?

[tobiashochguertel]

After some research I was able to understand that I have to modify my custom rpm so that is will be located under /usr/ instead /var/. /var stays empty in ostree. After changing the location to /usr/opt things went fine and I was able to get the complete content of the custom rpm on my custom ostree.

I changed the regular expression from above to: :%s/"\/opt/"\/var\/opt/g.

As next I have to add or change the tmpfiles.d file (system.d tmpfiles.d) of puppet-agent rpm (my custom rpm) before rpmrebuild it.

cat <<EOF > /usr/lib/tmpfiles.d/puppet-agent.conf
d /var/run/puppetlabs 0755 root root -
d /tmp/puppetlabs 0755 root root -
d /tmp/puppetlabs/puppet 755 root root -
d /tmp/puppetlabs/puppet/cache 755 root root -
L+ /usr/opt/puppetlabs/puppet/cache - - - - /tmp/puppetlabs/puppet/cache
L+ /opt/puppetlabs - - - - /usr/opt/puppetlabs
EOF

On my testing centos-atomic-host it is now possible to start the puppet-agent systemd service, the original location of /opt/puppetlabs is a symLink to /usr/opt/puppetlabs.

There are some addtional adjustments todo at the custom rpm - which are not yet done by me, but the informations here are already useful so I hope.

HTH
Tobias

[cgwalters]

Thanks for providing the workarounds. I would like to support this better, however, it would touch a lot of code that is in flux for other work (e.g. the just-landed package layering).

[cgwalters]

See also https://lists.projectatomic.io/projectatomic-archives/atomic-devel/2014-December/msg00026.html

[jlebon]

I'm a bit curious as with puppet it puts some files under /opt/puppetlabs/facter/cache/cached_facts which are created by the embedded facter application depending on the config.

Yes, that should work. /opt is writable in this model but packaged content is still updated as expected during host updates.

Definitely interested in getting this into the hands of people so that real uses of /opt and /usr/local can be tested.

[LKD-PIX]

Is there a timeframe for this feature coming to "next" stream ?😢

[giesse]

This thread is super long and I admit to have not read it all.
My issue is related to drivers for a Lexmark printer/scanner combo. I was able to solve the problem with the scanner by installing the drivers in a container and running simple-scan from the container, so that part is not critical, but it would be nice to have access to the scanner system-wide. I don't think I can do this for the printer though as I'd like to be able to print from a bunch of programs rather than just one.
The printer drivers are released either as a tarball with an install.sh inside, or a RPM. The install.sh will of course try to write to /usr and fail. The RPM will of course try to install stuff in /usr/local and so rpm-ostree won't install it. It would have been nice if Lexmark had made their mind up about where to put stuff but apparently they put files all over the place.
Is there anything I can do about this or do I need to wait for rpm-ostree to support this natively in some way?