systemd ignores RootDirectory option in .service units #22760

MegaBrutal · 2022-03-15T23:45:00Z

systemd version the issue has been seen with

systemd 249 (249.10-0ubuntu2)

Used distribution

Ubuntu 22.04 Jammy Jellyfish (development branch)

Linux kernel version used (uname -a)

Linux savelog 5.4.0-92-generic #103-Ubuntu SMP Fri Nov 26 16:13:00 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

CPU architecture issue was seen on

x86-64 / amd64

Expected behaviour you didn't see

systemd starts the service in chroot, or fails to start it with an error message telling how the chroot failed.

Unexpected behaviour you saw

systemd just tries to start the service on the host file system without chroot.

Steps to reproduce the problem

Described in detail in the following LaunchPad bug report:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1959047

Additional program output to the terminal or log subsystem illustrating the issue

See LaunchPad bug report for details:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1959047

Important note
Since then I discovered that the issue only affects LXC containers and a workaround is setting the container's AppArmor profile to unconfined. But for once I'd be more comfortable not having to set the container to unconfined for the sake of a chroot (though that's probably Ubuntu's fault); and if the chroot does fail, systemd should not attempt to start the program on the root file system, as this can even be dangerous.

The text was updated successfully, but these errors were encountered:

poettering · 2022-03-16T16:23:56Z

If RootDirectory= is set and we can't set up a mount namespace we should fail, and not attempt any fallbacks.

bluca · 2022-03-17T22:59:21Z

Is this an apparmor specific issue? Can you set log level to debug systemctl log-level debug and try again, and attach the full output of journalctl --unit <foo.service>?

MegaBrutal · 2022-03-17T23:20:00Z

@bluca It seems to be closely related to AppArmor, but I don't really have any level of expertise with that, the only thing I do to it is enable/disable, never fine-tune. That means I blindly use the AppArmor profiles Ubuntu ships for me. The host system that runs the container is Ubuntu 20.04 Focal Fossa.

Here is the output. Interesting that now it mentions "Failed to set up namespace, assuming containerized execution, ignoring: Permission denied". So it seems systemd detects the problem, but then just carries on.

Mar 18 00:05:00 savelog systemd[1]: lsb-release.service: Trying to enqueue job lsb-release.service/start/replace
Mar 18 00:05:00 savelog systemd[1]: lsb-release.service: Installed new job lsb-release.service/start as 1839
Mar 18 00:05:00 savelog systemd[1]: lsb-release.service: Enqueued job lsb-release.service/start as 1839
Mar 18 00:05:00 savelog systemd[1]: lsb-release.service: Failed to reset devices.allow/devices.deny: Operation not permitted
Mar 18 00:05:00 savelog systemd[1]: lsb-release.service: Failed to remove oomd_avoid flag on control group /system.slice/lsb-release.service, ignorin>
Mar 18 00:05:00 savelog systemd[1]: lsb-release.service: Failed to remove oomd_omit flag on control group /system.slice/lsb-release.service, ignoring>
Mar 18 00:05:00 savelog systemd[1]: lsb-release.service: About to execute /bin/pwd
Mar 18 00:05:00 savelog systemd[1]: lsb-release.service: Forked /bin/pwd as 12111
Mar 18 00:05:00 savelog systemd[1]: lsb-release.service: Changed dead -> start-pre
Mar 18 00:05:00 savelog systemd[1]: Starting LSB Release Information...
Mar 18 00:05:00 savelog systemd[12111]: Failed to remount '/' as SLAVE: Permission denied
Mar 18 00:05:00 savelog systemd[12111]: lsb-release.service: Failed to set up namespace, assuming containerized execution, ignoring: Permission denied
Mar 18 00:05:00 savelog systemd[12111]: lsb-release.service: Executing: /bin/pwd
Mar 18 00:05:00 savelog pwd[12111]: /
Mar 18 00:05:00 savelog systemd[1]: lsb-release.service: Failed to read oom_kill field of memory.events cgroup attribute: No such file or directory
Mar 18 00:05:00 savelog systemd[1]: lsb-release.service: Child 12111 belongs to lsb-release.service.
Mar 18 00:05:00 savelog systemd[1]: lsb-release.service: Control process exited, code=exited, status=0/SUCCESS (success)
Mar 18 00:05:00 savelog systemd[1]: lsb-release.service: Got final SIGCHLD for state start-pre.
Mar 18 00:05:00 savelog systemd[1]: lsb-release.service: Passing 0 fds to service
Mar 18 00:05:00 savelog systemd[1]: lsb-release.service: About to execute /usr/bin/lsb_release -a
Mar 18 00:05:00 savelog systemd[1]: lsb-release.service: Forked /usr/bin/lsb_release as 12112
Mar 18 00:05:00 savelog systemd[1]: lsb-release.service: Changed start-pre -> running
Mar 18 00:05:00 savelog systemd[1]: lsb-release.service: Job 1839 lsb-release.service/start finished, result=done
Mar 18 00:05:00 savelog systemd[1]: Started LSB Release Information.
Mar 18 00:05:00 savelog systemd[1]: lsb-release.service: Failed to send unit change signal for lsb-release.service: Connection reset by peer
Mar 18 00:05:00 savelog systemd[1]: lsb-release.service: Control group is empty.
Mar 18 00:05:00 savelog systemd[12112]: Failed to remount '/' as SLAVE: Permission denied
Mar 18 00:05:00 savelog systemd[12112]: lsb-release.service: Failed to set up namespace, assuming containerized execution, ignoring: Permission denied
Mar 18 00:05:00 savelog systemd[12112]: lsb-release.service: Executing: /usr/bin/lsb_release -a
Mar 18 00:05:00 savelog lsb_release[12112]: No LSB modules are available.
Mar 18 00:05:00 savelog lsb_release[12112]: Distributor ID:     Ubuntu
Mar 18 00:05:00 savelog lsb_release[12112]: Description:        Ubuntu Jammy Jellyfish (development branch)
Mar 18 00:05:00 savelog lsb_release[12112]: Release:    22.04
Mar 18 00:05:00 savelog lsb_release[12112]: Codename:   jammy
Mar 18 00:05:00 savelog systemd[1]: lsb-release.service: Failed to read oom_kill field of memory.events cgroup attribute: No such file or directory
Mar 18 00:05:00 savelog systemd[1]: lsb-release.service: Child 12112 belongs to lsb-release.service.
Mar 18 00:05:00 savelog systemd[1]: lsb-release.service: Main process exited, code=exited, status=0/SUCCESS (success)
root@savelog:~# journalctl -a --no-pager --since 0:00 --unit lsb-release
Mar 18 00:05:00 savelog systemd[1]: lsb-release.service: Trying to enqueue job lsb-release.service/start/replace
Mar 18 00:05:00 savelog systemd[1]: lsb-release.service: Installed new job lsb-release.service/start as 1839
Mar 18 00:05:00 savelog systemd[1]: lsb-release.service: Enqueued job lsb-release.service/start as 1839
Mar 18 00:05:00 savelog systemd[1]: lsb-release.service: Failed to reset devices.allow/devices.deny: Operation not permitted
Mar 18 00:05:00 savelog systemd[1]: lsb-release.service: Failed to remove oomd_avoid flag on control group /system.slice/lsb-release.service, ignoring: Operation not supported
Mar 18 00:05:00 savelog systemd[1]: lsb-release.service: Failed to remove oomd_omit flag on control group /system.slice/lsb-release.service, ignoring: Operation not supported
Mar 18 00:05:00 savelog systemd[1]: lsb-release.service: About to execute /bin/pwd
Mar 18 00:05:00 savelog systemd[1]: lsb-release.service: Forked /bin/pwd as 12111
Mar 18 00:05:00 savelog systemd[1]: lsb-release.service: Changed dead -> start-pre
Mar 18 00:05:00 savelog systemd[1]: Starting LSB Release Information...
Mar 18 00:05:00 savelog systemd[12111]: Failed to remount '/' as SLAVE: Permission denied
Mar 18 00:05:00 savelog systemd[12111]: lsb-release.service: Failed to set up namespace, assuming containerized execution, ignoring: Permission denied
Mar 18 00:05:00 savelog systemd[12111]: lsb-release.service: Executing: /bin/pwd
Mar 18 00:05:00 savelog pwd[12111]: /
Mar 18 00:05:00 savelog systemd[1]: lsb-release.service: Failed to read oom_kill field of memory.events cgroup attribute: No such file or directory
Mar 18 00:05:00 savelog systemd[1]: lsb-release.service: Child 12111 belongs to lsb-release.service.
Mar 18 00:05:00 savelog systemd[1]: lsb-release.service: Control process exited, code=exited, status=0/SUCCESS (success)
Mar 18 00:05:00 savelog systemd[1]: lsb-release.service: Got final SIGCHLD for state start-pre.
Mar 18 00:05:00 savelog systemd[1]: lsb-release.service: Passing 0 fds to service
Mar 18 00:05:00 savelog systemd[1]: lsb-release.service: About to execute /usr/bin/lsb_release -a
Mar 18 00:05:00 savelog systemd[1]: lsb-release.service: Forked /usr/bin/lsb_release as 12112
Mar 18 00:05:00 savelog systemd[1]: lsb-release.service: Changed start-pre -> running
Mar 18 00:05:00 savelog systemd[1]: lsb-release.service: Job 1839 lsb-release.service/start finished, result=done
Mar 18 00:05:00 savelog systemd[1]: Started LSB Release Information.
Mar 18 00:05:00 savelog systemd[1]: lsb-release.service: Failed to send unit change signal for lsb-release.service: Connection reset by peer
Mar 18 00:05:00 savelog systemd[1]: lsb-release.service: Control group is empty.
Mar 18 00:05:00 savelog systemd[12112]: Failed to remount '/' as SLAVE: Permission denied
Mar 18 00:05:00 savelog systemd[12112]: lsb-release.service: Failed to set up namespace, assuming containerized execution, ignoring: Permission denied
Mar 18 00:05:00 savelog systemd[12112]: lsb-release.service: Executing: /usr/bin/lsb_release -a
Mar 18 00:05:00 savelog lsb_release[12112]: No LSB modules are available.
Mar 18 00:05:00 savelog lsb_release[12112]: Distributor ID:	Ubuntu
Mar 18 00:05:00 savelog lsb_release[12112]: Description:	Ubuntu Jammy Jellyfish (development branch)
Mar 18 00:05:00 savelog lsb_release[12112]: Release:	22.04
Mar 18 00:05:00 savelog lsb_release[12112]: Codename:	jammy
Mar 18 00:05:00 savelog systemd[1]: lsb-release.service: Failed to read oom_kill field of memory.events cgroup attribute: No such file or directory
Mar 18 00:05:00 savelog systemd[1]: lsb-release.service: Child 12112 belongs to lsb-release.service.
Mar 18 00:05:00 savelog systemd[1]: lsb-release.service: Main process exited, code=exited, status=0/SUCCESS (success)
Mar 18 00:05:00 savelog systemd[1]: lsb-release.service: Deactivated successfully.
Mar 18 00:05:00 savelog systemd[1]: lsb-release.service: Service will not restart (restart setting)
Mar 18 00:05:00 savelog systemd[1]: lsb-release.service: Changed running -> dead
Mar 18 00:05:00 savelog systemd[1]: lsb-release.service: Collecting.

bluca · 2022-03-17T23:29:04Z

What's the exact unit you started? Full configuration

MegaBrutal · 2022-03-18T00:25:15Z

Here it is:

○ lsb-release.service - LSB Release Information
     Loaded: loaded (/etc/systemd/system/lsb-release.service; static)
     Active: inactive (dead)

[Unit]
Description=LSB Release Information

[Service]
Type=simple
RootDirectory=/chroot/postgresql
ExecStartPre=/bin/pwd
ExecStart=/usr/bin/lsb_release -a

Of course it's just a test service for proof of concept. Originally I had a service which would have started a PostgreSQL server within the chroot.

bluca · 2022-03-18T11:01:04Z

This is very strange, I see no messages about setting up RootDirectory in the logs you posted, moreover the check that should make it fail is there in v249.10 (insist_on_sandboxing()), but it's clearly printing the log message that happens when no sandboxing settings are provided: https://github.com/systemd/systemd-stable/blob/v249.10/src/core/execute.c#L3274

Are you 100% sure that's the unit being used? And it's loaded (daemon-reload after adding/editing it, etc)?

bluca · 2022-03-18T11:07:22Z

Actually it's even more bizzarre, it's clearly falling into this path with EPERM: https://github.com/systemd/systemd-stable/blob/v249.10/src/core/namespace.c#L2247 but then it's also clearly taking this path: https://github.com/systemd/systemd-stable/blob/v249.10/src/core/execute.c#L3274 which explicitly checks for ENOANO?? Are you sure it's running under v249.10?

Mar 18 00:05:00 savelog systemd[12112]: Failed to remount '/' as SLAVE: Permission denied
Mar 18 00:05:00 savelog systemd[12112]: lsb-release.service: Failed to set up namespace, assuming containerized execution, ignoring: Permission denied
Mar 18 00:05:00 savelog systemd[12112]: lsb-release.service: Executing: /usr/bin/lsb_release -a

poettering · 2022-03-18T13:24:20Z

Maybe there's confusion about the systemd version within the container, and 249.10 is the one outside the container?

MegaBrutal · 2022-03-18T13:45:30Z

Version inside the container:

systemd 249 (249.10-0ubuntu2)
+PAM +AUDIT +SELINUX +APPARMOR +IMA +SMACK +SECCOMP +GCRYPT +GNUTLS -OPENSSL +ACL +BLKID +CURL +ELFUTILS -FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP -LIBFDISK +PCRE2 -PWQUALITY -P11KIT -QRENCODE +BZIP2 +LZ4 +XZ +ZLIB +ZSTD -XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified

Version outside the container (host system):

systemd 245 (245.4-4ubuntu3.15)
+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=hybrid

Isn't it possible that Ubuntu added their own modifications that broke the logic? Sad that they don't reply on LaunchPad... I mean, I submitted the original report more than a month ago. There's a reason why I report downstream first.

How can I help? Can you reproduce? Shall I build a VM image or perhaps Docker container which reproduces it? Or do you have a build of vanilla systemd to try? Would be best to have a compiled binary, or I hope systemd builds easily...

bluca · 2022-03-18T13:52:24Z

Isn't it possible that Ubuntu added their own modifications that broke the logic? Sad that they don't reply on LaunchPad... I mean, I submitted the original report more than a month ago. There's a reason why I report downstream first.

It indeed is:

https://git.launchpad.net/ubuntu/+source/systemd/commit/src/core/execute.c?h=applied/ubuntu/jammy&id=f933caff8a840ebc5187fa2824133f10ee0cecff

--- a/[src/core/execute.c](https://git.launchpad.net/ubuntu/+source/systemd/tree/src/core/execute.c?h=applied/ubuntu/jammy&id=44a8216ac51cba445adaddabff81464d5fb62e35)
+++ b/[src/core/execute.c](https://git.launchpad.net/ubuntu/+source/systemd/tree/src/core/execute.c?h=applied/ubuntu/jammy&id=f933caff8a840ebc5187fa2824133f10ee0cecff)
@@ -3278,6 +3278,13 @@ static int apply_mount_namespace(
 
 finalize:
         bind_mount_free_many(bind_mounts, n_bind_mounts);
+
+        /* If we couldn't set up the namespace this is probably due to a
+         * missing capability. In this case, silently proceeed. */
+        if (IN_SET(r, -EPERM, -EACCES)) {
+                log_unit_debug_errno(u, r, "Failed to set up namespace, assuming containerized execution, ignoring: %m");
+                return 0;
+        }
         return r;
 }

That is not good - I'll go prod

bluca · 2022-03-18T13:55:36Z

It was added to support old versions of LXC/LXD https://git.launchpad.net/ubuntu/+source/systemd/tree/debian/patches/debian/UBUNTU-Revert-namespace-be-more-careful-when-handling-namespacin.patch?h=ubuntu/jammy

bluca · 2022-03-18T14:04:19Z

Closing here because there's nothing we can do, it's a downstream patch. I've added details do the downstream ticket.

bluca added pid1 needs-reporter-feedback ❓ There's an unanswered question, the reporter needs to answer apparmor labels Mar 17, 2022

bluca added not-our-bug downstream/debian Tracking bugs for Debian/Ubuntu/derivates and removed needs-reporter-feedback ❓ There's an unanswered question, the reporter needs to answer labels Mar 18, 2022

bluca closed this as completed Mar 18, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

systemd ignores RootDirectory option in .service units #22760

systemd ignores RootDirectory option in .service units #22760

MegaBrutal commented Mar 15, 2022

poettering commented Mar 16, 2022

bluca commented Mar 17, 2022

MegaBrutal commented Mar 17, 2022

bluca commented Mar 17, 2022

MegaBrutal commented Mar 18, 2022

bluca commented Mar 18, 2022

bluca commented Mar 18, 2022

poettering commented Mar 18, 2022

MegaBrutal commented Mar 18, 2022

bluca commented Mar 18, 2022

bluca commented Mar 18, 2022

bluca commented Mar 18, 2022

systemd ignores RootDirectory option in .service units #22760

systemd ignores RootDirectory option in .service units #22760

Comments

MegaBrutal commented Mar 15, 2022

poettering commented Mar 16, 2022

bluca commented Mar 17, 2022

MegaBrutal commented Mar 17, 2022

bluca commented Mar 17, 2022

MegaBrutal commented Mar 18, 2022

bluca commented Mar 18, 2022

bluca commented Mar 18, 2022

poettering commented Mar 18, 2022

MegaBrutal commented Mar 18, 2022

bluca commented Mar 18, 2022

bluca commented Mar 18, 2022

bluca commented Mar 18, 2022