Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

adds a fully safe way how apps can pin files into /tmp temporarily, excepting them from the tmpfiles aging algorithm, based on BSD file locks on dirs we descend into #11482

Merged
merged 6 commits into from Feb 20, 2019

Conversation

@poettering
Copy link
Member

poettering commented Jan 18, 2019

Not sure why we never thought about this before.

This allows us to fix systemd/mkosi#252

@poettering poettering added the tmpfiles label Jan 18, 2019
poettering added a commit to poettering/mkosi that referenced this pull request Jan 18, 2019
poettering added a commit to poettering/mkosi that referenced this pull request Jan 18, 2019
This makes use of the new logic here:

systemd/systemd#11482

(And on older systems has no effect, but that shouldn't hurt either)

Fixes: systemd#252
@poettering

This comment has been minimized.

Copy link
Member Author

poettering commented Jan 18, 2019

An mkosi counterpart PR that makes use of this is here:

systemd/mkosi#319

@larskarlitski

This comment has been minimized.

Copy link
Contributor

larskarlitski commented Jan 19, 2019

We stumbled across the same issue yesterday (nice timing) in lorax: weldr/lorax#570. Would be cool to see this land soon, so that we can use the same trick. Thanks!

@poettering poettering added this to the v242 milestone Jan 21, 2019
@poettering

This comment has been minimized.

Copy link
Member Author

poettering commented Jan 21, 2019

let's postpone this to v242, and let's make v241 purely about bugfixing.

bcl added a commit to bcl/lorax that referenced this pull request Jan 21, 2019
If systemd's tmpfiles.d timer is executed while lorax is running it will
remove any files and directories older than 30 days. This is what has
been causing the occasional error where /proc/ would seem to vanish
during the install.

Upstream has proposed this solution, systemd/systemd#11482
but until that is released we need a work-around to protect the lorax
files.

This commit does several things:

* Move the default tmpdir from /var/tmp/ to /var/tmp/lorax/
* Add a lorax.conf tmpfiles.d file that
  1) removes everything under /var/tmp/lorax/ at boot time.
  2) prevents systemd-tmpfiles from removing anything under /var/tmp/lorax/
     (except at boot time).
* Add an exit handler to lorax so that temporary directories are removed on
  exit or on a python traceback.

Note that this will not remove the tempdir if lorax is killed with a
signal -- those are being caught by dnf and prevent the exit handler
from running.

If you want to keep the working directory around for debugging purposes
use --workdir /var/tmp/lorax/my-workdir and it won't be removed up by
lorax.
bcl added a commit to bcl/lorax that referenced this pull request Jan 22, 2019
If systemd's tmpfiles.d timer is executed while lorax is running it will
remove any files and directories older than 30 days. This is what has
been causing the occasional error where /proc/ would seem to vanish
during the install.

Upstream has proposed this solution, systemd/systemd#11482
but until that is released we need a work-around to protect the lorax
files.

This commit does several things:

* Move the default tmpdir from /var/tmp/ to /var/tmp/lorax/
* Add a lorax.conf tmpfiles.d file that prevents systemd-tmpfiles from
  removing anything under /var/tmp/lorax/
* Add an exit handler to lorax so that temporary directories are removed on
  exit or on a python traceback.
* Use flock to lock access to the tempdir while lorax is running.
* Remove any unlocked tempdirs named /var/tmp/lorax/lorax.* at startup

Note that the exit handler will not remove the tempdir if lorax is
killed with a signal -- those are being caught by dnf and prevent the
exit handler from running.

systemd-tmpfiles cannot clean up the tempdirs at boot time because they
contain files labeled as shadow_t, so we have to remove those when lorax
runs. It uses the flock to prevent removing any directories created by
parallel instances of lorax and only removes ones that are unlocked.
Worst case they will be around until the first run of lorax after a
reboot.

If you want to keep the working directory around for debugging purposes
use --workdir /var/tmp/lorax/my-workdir and it won't be removed by
lorax.
@poettering poettering force-pushed the poettering:tmpfiles-bsd-lock branch from 1cec55f to 39ea781 Jan 23, 2019
bcl added a commit to bcl/lorax that referenced this pull request Jan 29, 2019
If systemd's tmpfiles.d timer is executed while lorax is running it will
remove any files and directories older than 30 days. This is what has
been causing the occasional error where /proc/ would seem to vanish
during the install.

Upstream has proposed this solution, systemd/systemd#11482
but until that is released we need a work-around to protect the lorax
files.

This commit does several things:

* Move the default tmpdir from /var/tmp/ to /var/tmp/lorax/
* Add a lorax.conf tmpfiles.d file that prevents systemd-tmpfiles from
  removing anything under /var/tmp/lorax/
* Add an exit handler to lorax so that temporary directories are removed on
  exit or on a python traceback.
* Use flock to lock access to the tempdir while lorax is running.
* Remove any unlocked tempdirs named /var/tmp/lorax/lorax.* at startup

Note that the exit handler will not remove the tempdir if lorax is
killed with a signal -- those are being caught by dnf and prevent the
exit handler from running.

systemd-tmpfiles cannot clean up the tempdirs at boot time because they
contain files labeled as shadow_t, so we have to remove those when lorax
runs. It uses the flock to prevent removing any directories created by
parallel instances of lorax and only removes ones that are unlocked.
Worst case they will be around until the first run of lorax after a
reboot.

If you want to keep the working directory around for debugging purposes
use --workdir /var/tmp/lorax/my-workdir and it won't be removed by
lorax.
bcl added a commit to weldr/lorax that referenced this pull request Jan 29, 2019
If systemd's tmpfiles.d timer is executed while lorax is running it will
remove any files and directories older than 30 days. This is what has
been causing the occasional error where /proc/ would seem to vanish
during the install.

Upstream has proposed this solution, systemd/systemd#11482
but until that is released we need a work-around to protect the lorax
files.

This commit does several things:

* Move the default tmpdir from /var/tmp/ to /var/tmp/lorax/
* Add a lorax.conf tmpfiles.d file that prevents systemd-tmpfiles from
  removing anything under /var/tmp/lorax/
* Add an exit handler to lorax so that temporary directories are removed on
  exit or on a python traceback.
* Use flock to lock access to the tempdir while lorax is running.
* Remove any unlocked tempdirs named /var/tmp/lorax/lorax.* at startup

Note that the exit handler will not remove the tempdir if lorax is
killed with a signal -- those are being caught by dnf and prevent the
exit handler from running.

systemd-tmpfiles cannot clean up the tempdirs at boot time because they
contain files labeled as shadow_t, so we have to remove those when lorax
runs. It uses the flock to prevent removing any directories created by
parallel instances of lorax and only removes ones that are unlocked.
Worst case they will be around until the first run of lorax after a
reboot.

If you want to keep the working directory around for debugging purposes
use --workdir /var/tmp/lorax/my-workdir and it won't be removed by
lorax.

(cherry picked from commit e4fe1aa)
bcl added a commit to weldr/lorax that referenced this pull request Jan 29, 2019
If systemd's tmpfiles.d timer is executed while lorax is running it will
remove any files and directories older than 30 days. This is what has
been causing the occasional error where /proc/ would seem to vanish
during the install.

Upstream has proposed this solution, systemd/systemd#11482
but until that is released we need a work-around to protect the lorax
files.

This commit does several things:

* Move the default tmpdir from /var/tmp/ to /var/tmp/lorax/
* Add a lorax.conf tmpfiles.d file that prevents systemd-tmpfiles from
  removing anything under /var/tmp/lorax/
* Add an exit handler to lorax so that temporary directories are removed on
  exit or on a python traceback.
* Use flock to lock access to the tempdir while lorax is running.
* Remove any unlocked tempdirs named /var/tmp/lorax/lorax.* at startup

Note that the exit handler will not remove the tempdir if lorax is
killed with a signal -- those are being caught by dnf and prevent the
exit handler from running.

systemd-tmpfiles cannot clean up the tempdirs at boot time because they
contain files labeled as shadow_t, so we have to remove those when lorax
runs. It uses the flock to prevent removing any directories created by
parallel instances of lorax and only removes ones that are unlocked.
Worst case they will be around until the first run of lorax after a
reboot.

If you want to keep the working directory around for debugging purposes
use --workdir /var/tmp/lorax/my-workdir and it won't be removed by
lorax.

(cherry picked from commit e4fe1aa)
bcl added a commit to weldr/lorax that referenced this pull request Jan 30, 2019
If systemd's tmpfiles.d timer is executed while lorax is running it will
remove any files and directories older than 30 days. This is what has
been causing the occasional error where /proc/ would seem to vanish
during the install.

Upstream has proposed this solution, systemd/systemd#11482
but until that is released we need a work-around to protect the lorax
files.

This commit does several things:

* Move the default tmpdir from /var/tmp/ to /var/tmp/lorax/
* Add a lorax.conf tmpfiles.d file that prevents systemd-tmpfiles from
  removing anything under /var/tmp/lorax/
* Add an exit handler to lorax so that temporary directories are removed on
  exit or on a python traceback.
* Use flock to lock access to the tempdir while lorax is running.
* Remove any unlocked tempdirs named /var/tmp/lorax/lorax.* at startup

Note that the exit handler will not remove the tempdir if lorax is
killed with a signal -- those are being caught by dnf and prevent the
exit handler from running.

systemd-tmpfiles cannot clean up the tempdirs at boot time because they
contain files labeled as shadow_t, so we have to remove those when lorax
runs. It uses the flock to prevent removing any directories created by
parallel instances of lorax and only removes ones that are unlocked.
Worst case they will be around until the first run of lorax after a
reboot.

If you want to keep the working directory around for debugging purposes
use --workdir /var/tmp/lorax/my-workdir and it won't be removed by
lorax.

Resolves: rhbz#1668408
@poettering poettering force-pushed the poettering:tmpfiles-bsd-lock branch from 39ea781 to ebdffe0 Feb 14, 2019
@poettering poettering removed the postponed label Feb 15, 2019
@poettering poettering force-pushed the poettering:tmpfiles-bsd-lock branch from ebdffe0 to a682927 Feb 15, 2019
poettering added 6 commits Jan 18, 2019
Let's minimize file scope, use compund literals and only use LOG_WARN
for errors we ignore.
…ent into

Let's add a fully safe way to exclude certain directories from aging, by
taking a BSD file lock on them before aging them. This is useful for
clients that untar tarballs into /tmp or /var/tmp, which might have
really old timestamps, and to which the aging logic would be very harsh:
they can simply take a BSD file lock on any directory they like and thus
exclude it from automatic aging, and thus need not to be afraid of
untarring stuff below it.

Previously, similar functionality was already available through the
sticky bit on non-directories, but it's problematic, since as soon as
the bit is set no clean-up is done for it at all anymore, forever. Also,
it is not suitable for untarring stuff, since the sticky bit after all
is a concept denoted in the tarball itself.  BSD file locking semantics
are much much nicer there, as they are automatically released when the
application that has them dies, and they are entirely orthogonal to data
encoded in tarballs.

This patch takes BSD file locks only on *directories* while descending
down the tree, not on regular files. Moreover, it will do so in
non-blocking mode only, i.e. if anyone else has a lock the aging for a
dir and everything below it is immediately skipped for the current
clean-up iteration.

Of course applications might take BSD file locks for other reasons than
just prevent aging (i.e for their own reasons), but that should be
entirely OK, as in that case tmpfiles should step away from those files
anyway too: it's a good idea to stay away from any such locked file
anyway since it's apparently curretnly being manipulated.

This allows us to fix bugs like this:

systemd/mkosi#252
We potentially might descent into quite deep directory trees. Let's
hence make sure we can allocate a lot of fds.

(This reflects the fact that glibc nftw() and friends have some logic in
place to reduce fd usage while descending into directory trees. Doing so
is a bit nasty I think, and given that fds are basically free now, if we
ask for them, lte's just protect ourselves and make use of that)

(No, I am not aware of a real-world case where this was necessary, but
let's better be safe than sorry)
@poettering poettering force-pushed the poettering:tmpfiles-bsd-lock branch from a682927 to e535840 Feb 15, 2019
Copy link
Member

keszybz left a comment

LGTM. Let's do any tweaks separately.

@@ -601,6 +602,11 @@ static int dir_cleanup(
continue;
}

if (flock(dirfd(sub_dir), LOCK_EX|LOCK_NB) < 0) {
log_debug_errno(errno, "Couldn't acquire shared BSD lock on directory \"%s\", skipping: %m", p);

This comment has been minimized.

Copy link
@keszybz

keszybz Feb 20, 2019

Member

I think it'd be better to phrase is less like a failure: "Directory "%s" is locked, skipping."

@@ -545,6 +545,14 @@ w- /proc/sys/vm/swappiness - - - - 10</programlisting></para>
(ctime). Any of these three (or two) values will prevent cleanup
if it is more recent than the current time minus the age
field.</para>

<para>Note that while the aging algorithm is run a 'shared' BSD file lock (see <citerefentry

This comment has been minimized.

Copy link
@keszybz

keszybz Feb 20, 2019

Member

Hmm, "aging algorithm"... Maybe call this "age-based cleanup"?

@keszybz keszybz merged commit ac5c138 into systemd:master Feb 20, 2019
8 of 10 checks passed
8 of 10 checks passed
bionic-amd64 autopkgtest finished (failure)
Details
bionic-i386 autopkgtest finished (failure)
Details
CentOS CI Build finished.
Details
CentOS CI (Vagrant) Build finished.
Details
LGTM analysis: C/C++ No new or fixed alerts
Details
LGTM analysis: JavaScript No code changes detected
Details
LGTM analysis: Python No code changes detected
Details
bionic-s390x autopkgtest finished (success)
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
semaphoreci The build passed on Semaphore.
Details
bcl added a commit to weldr/lorax that referenced this pull request Feb 21, 2019
If systemd's tmpfiles.d timer is executed while lorax is running it will
remove any files and directories older than 30 days. This is what has
been causing the occasional error where /proc/ would seem to vanish
during the install.

Upstream has proposed this solution, systemd/systemd#11482
but until that is released we need a work-around to protect the lorax
files.

This commit does several things:

* Move the default tmpdir from /var/tmp/ to /var/tmp/lorax/
* Add a lorax.conf tmpfiles.d file that prevents systemd-tmpfiles from
  removing anything under /var/tmp/lorax/
* Add an exit handler to lorax so that temporary directories are removed on
  exit or on a python traceback.
* Use flock to lock access to the tempdir while lorax is running.
* Remove any unlocked tempdirs named /var/tmp/lorax/lorax.* at startup

Note that the exit handler will not remove the tempdir if lorax is
killed with a signal -- those are being caught by dnf and prevent the
exit handler from running.

systemd-tmpfiles cannot clean up the tempdirs at boot time because they
contain files labeled as shadow_t, so we have to remove those when lorax
runs. It uses the flock to prevent removing any directories created by
parallel instances of lorax and only removes ones that are unlocked.
Worst case they will be around until the first run of lorax after a
reboot.

If you want to keep the working directory around for debugging purposes
use --workdir /var/tmp/lorax/my-workdir and it won't be removed by
lorax.

Resolves: rhbz#1668520
poettering added a commit to poettering/mkosi that referenced this pull request Feb 26, 2019
This makes use of the new logic here:

systemd/systemd#11482

(And on older systems has no effect, but that shouldn't hurt either)

Fixes: systemd#252
lucaswerkmeister added a commit to systemd/mkosi that referenced this pull request Feb 26, 2019
This makes use of the new logic here:

systemd/systemd#11482

(And on older systems has no effect, but that shouldn't hurt either)

Fixes: #252
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
3 participants
You can’t perform that action at this time.