rpm --rebuilddb in an overlayfs without redirect_dir=on

[th-hummel]

Hello,

[I'm not sure this is the right place to post such a question, feel free to redirect me if needed.]

using RPM version 4.19.1 on Fedora release 39 (Thirty Nine) or RPM version 4.14.3 on Red Hat Enterprise Linux release 8.8 (Ootpa), I experienced that I could not successfully run rpm --rebuilddb in an Apptainer or Docker container. I would get the following error msg:

Apptainer> rpm --rebuilddb
error: failed to replace old database with new database!
error: replace files in /var/lib/rpm with files from /var/lib/rpmrebuilddb.53312 to recover

My understanding it that rpm does not handle the EXDEV error when rename() is called in an overlayfs without the redirect_dir option turned to on

As a matter of fact, if I turn this on via:

echo "Y" > /sys/module/overlay/parameters/redirect_dir

the command succeeds

My question was, as I think some tools like mv do workaround this error, what is the reason for that (my guess is there is then no atomicity) and/or could it fit a legit request to implement this feature (beeing able to run rpm --rebuilddb when into a container or maybe just to give a hint about possible cause) or not ?

Context: in some provisioning environment, I happen to have to rebuild the database inside a Docker container.

Thanks for your help

--
Thomas HUMMEL

[dmnks]

Yup, this is a known issue, see #2355.

The thing is, even if this would be possible to work around, we don't want to have any such filesystem-specific code in RPM.

Now, thinking about it more, this might in fact be something to handle in an RPM plugin.

[dmnks]

Right, that is a valid question, although I'm no expert on OverlayFS so can't really answer that.

The only "explanation" (as to why EXDEV is issued on a rename(2) call) that I've found is the following excerpt from a comment in the OverlayFS code:

[...] This means, when copying up a directory we can rely on it and ancestors being stable.

There's also some more documentation on redirect_dir here with some hints on the "why" (although no clear explanation either).

[th-hummel]

Well, I'm no expert either but my understanding is that for instance a tool like mv would first try rename() and if it returns EXDEV it will workaround by copying data.

So, to me the main difference is the atomicity: when you set an xattr to the orignal dir then rename() the copied up dir, though this means 2 actions, the rename() is atomic since no data is actually copied.
My understanding is that such actions are somehow "prepared" in the workdir.

I don't know if in the rpm --rebuilddb you need this atomicity or not but that would be the first question to be answered I think ?

Thanks for your help

--
Thomas HUMMEL

[dmnks]

Well, I'm no expert either but my understanding is that for instance a tool like mv would first try rename() and if it returns EXDEV it will workaround by copying data.

That's correct, I posted a separate comment below covering this part.

So, to me the main difference is the atomicity: when you set an xattr to the orignal dir then rename() the copied up dir, though this means 2 actions, the rename() is atomic since no data is actually copied. My understanding is that such actions are somehow "prepared" in the workdir.

Indeed, the redirect_dir feature looks like an optimization. That said, I'm not sure what happens internally in OverlayFS when you actually do a "copy-up". I'd assume no data is physically copied either, unless it changes in the upper layer, but that's again just a guess.

I don't know if in the rpm --rebuilddb you need this atomicity or not but that would be the first question to be answered I think ?

I think atomicity is not a concern here - that's hopefully ensured by the (OverlayFS) filesystem, regardless of whether we choose to do a regular copy of the old directory (to trigger a copy-up) or use the redirect_dir feature. However, I believe we should do (if we decide to implement this) the former since that's reasonably filesystem-agnostic, unlike the latter (which is, again, OverlayFS specific).

In any case, thanks for pointing out redirect_dir, I wasn't aware of it before!

[dmnks]

Hmm, it seems like OverlayFS indeed does a full copy up (there's the metacopy feature that only does that for the metadata).

Anyway, using an OverlayFS feature would still be a no-go in my opinion, but like said, it's good to know about it.

[th-hummel]

Well, my understanding is that data is actually copied when copy-up is triggered (which of course happens only if you somehow modify lower or merge data or metadata). Overlayfs has some xattr tricks to present to the user in the merged layer the most "transparent/consistent" things as possible.

If atomicity is not a concern for the rpm database rebuild I think (if you decide to implement the feature) you maybe take the same route as mv if you (legitimately) don't want to fall into fs specificity.

Note: I don't know if the need to rebuild database in a container occurs much so it may not be worth implementing it I don't know. On the othe side, it may be great to have the --rebuilddb work no matter the environement.

Thanks for your reply again.

--
Thomas HUMMEL

[dmnks]

Looking at the OverlayFS docs some more, specifically at the section covering redirect_dir, it mentions the following (emphasis mine):

return EXDEV error: this error is returned by rename(2) when trying to move a file or directory across filesystem boundaries. Hence applications are usually prepared to handle this error (mv(1) for example recursively copies the directory tree). This is the default behavior.

Obviously, since mv(1) falls back to copying when crossing filesystem boundaries, it handles this property of OverlayFS transparently. That makes sense for a utility like mv(1) where you expect users to want to move stuff across filesystems, however in our code, we typically operate on the same filesystem when backing up the current rpmdb instance in the same directory.

Historically, before OverlayFS was a thing, there was no reason to assume otherwise, as you wouldn't normally have an "artificial" filesystem boundary within the same directory.

Since OverlayFS is now a mature and widely deployed filesystem (especially in the container space), adjusting our --rebuilddb logic to handle the EXDEV error actually seems quite reasonable, after all.

[th-hummel]

Sorry I think my replies somehow crossed yours.

[th-hummel]

I would agree about the redirect_dir nogo in the rpm case as there must be a reason why it's disabled by default (I cannot figure out what's the tradeoff) and you don't have control on what mount options are used anyway.

--
Thomas HUMMEL

[dmnks]

OK, having discussed this with the team, let's reopen #2355 and see what we can do on that front. This issue certainly is something that comes up regularly so we might as well bite the bullet now.

[th-hummel]

Hello,

that's great news!

Thanks again for your help

--
Thomas HUMMEL

[pmatilai]

As a matter of fact, if I turn this on via:

echo "Y" > /sys/module/overlay/parameters/redirect_dir

the command succeeds

This is an interesting bit that I haven't seen before. That clue reveals what is no doubt another wonderful rabbithole...
https://www.kernel.org/doc/Documentation/filesystems/overlayfs.txt

[th-hummel]

I still have to figure out what's the treadoff/drawback (if any) on turning this on...

--
Thomas HUMMEL