lxc: Failed to activate service 'org.freedesktop.systemd1': timed out after update to 233.75-3 from 232-8 #6408
Comments
|
Apologies for the formatting of the journalctl output above. I'm not sure how to improve this here. |
|
Unsure if related, but I experienced similar breakage on an Arch Linux ARM host, where systemd-logind and systemd-networkd fail to start on boot and logins are significantly delayed after recent package updates. I tracked my problem down to the expat package. Booting appears to work properly with expat 2.2.0-1 but not 2.2.1-1. Curious if downgrading would temporarily resolve your issue as well. |
|
Hi hewittc - thanks for responding. I rolled back the container to a good snapshot in which expat was at version 2.2.0-2 and systemd was at 232-8. Everything in the container works just fine at these versions. I added expat to the IgnorePkg list in pacman.conf to prevent it from being updated to 2.2.1-1. systemd was still updated to 233.75-3. Unfortunately this made no difference and systemd still no longer works after the update. I've posted a thread in the arch forum if anybody wants to see more detailed information ... |
|
The same issue seems to have been discussed in lxc/lxc#1678, but I'm not sure that it has been solved. Is there any chance you could try the recipe mentioned in lxc/lxc#1678 and then report the result back to the maintainers of |
poettering
changed the title
|
@evverx Thanks I have read through that thread but I can't see a solution there. The thread is marked as closed but the original poster's last few comments seem to show the issue is not resolved. Maybe I've missed something. |
I think this is a separate issue that happens to cause a similar symptom. I opened #6418. |
The original poster seems to have written that
bash-4.3# echo $$
1
bash-4.3# /lib/systemd/systemd --version
systemd 234
+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD -IDN2 +IDN default-hierarchy=hybrid
bash-4.3# mount -t sysfs sysfs /sys
bash-4.3# mount -t tmpfs tmpfs /sys/fs/cgroup/
bash-4.3# mkdir /sys/fs/cgroup/unified
bash-4.3# mount -t cgroup2 cgroup2 -o ro /sys/fs/cgroup/unified
bash-4.3# exec /lib/systemd/systemd
...
[!!!!!!] Failed to allocate manager object, freezing.Do you know if something similar happens inside or outside your container? I've already run I'd suggest running |
evverx
added
the
needs-reporter-feedback ❓
|
I'm working / travelling at the moment. I'll try and get back on this over the weekend - thanks. |
|
Hi - here is some more detailed information: cgroups listed on host machine: attach to the running lxc container (before systemd upgrade from 232-8 to 233.75-3), show systemd version + cgroups ... run During the systemd update there was a delay and the message displayed above. At this point the container won't shutdown either from within the container with
listing cgroups inside container now shows ... "unified" now does appear but owned by "nobody:nobody" while others are all owned by the root group. Perhaps this is the issue? But why is unified owned by nobody group and not root like everything else? This is an unprivileged container which is started by root on the host but all processes in the container effectively run as nobody from the host point of view. users/groups 0 - 65536 inside the container are mapped to 100000 - 165536 outside the container. On the host machine I have this in the and this is the container configuration file .... This configuration has been working perfectly well until systemd was updated from 232-8 to 233.75-3 inside the container. So this leads me to reporting it as a systemd issue, but I accept I may be wrong and it could be an issue with this uid:gid mapping mechanism in lxc. I'm hoping some more knowledgeable folk can tell me the answer! |
|
Thank you for the feedback. I think I understand what is going on. First of all, the issue can be worked around by passing lxc-start -n arch /sbin/init systemd.legacy_systemd_cgroup_controller=yes
systemctl -ff poweroffcan be run inside a container to shut it down. Also, it's possible to "kill" a container by running the following commands outside it: PID=$(lxc-info -n <name-of-the-container> -p -H)
kill -9 $PID
Yes, I can, but I've already reproduced the issue, so I think that is not needed any more. I think this is a bug. The hybrid hierarchy doesn't seem to be supposed to break tools which work well with the legacy hierarchy, and |
evverx
added
bug 🐛
brauner
pushed a commit
to brauner/lxc
that referenced
this issue
Jul 26, 2017
Closes lxc#1669. Closes lxc#1678. Relates to systemd/systemd#6408. Signed-off-by: Christian Brauner <christian.brauner@ubuntu.com>
|
@evverx, thanks for commenting on the https://github.com/lxc/lxc issue. :) I just sent a branch to handle the empty v2 hierarchy! |
|
@brauner , thank you for fixing lxc/lxc#1678. Now I'm wondering what should be done to see the following :) For the record, I've just seen it in lxc/lxc#1669. |
|
Yeah, I'm not completely sure either. What does systemd >= 233 do if the underlying kernel doesn't support cgroup v2? |
|
If this requires a patch in systemd I'm happy to send one. Did it before. :) |
|
That would be great! |
stgraber
pushed a commit
to lxc/lxc
that referenced
this issue
Aug 14, 2017
Closes #1669. Closes #1678. Relates to systemd/systemd#6408. Signed-off-by: Christian Brauner <christian.brauner@ubuntu.com>
stgraber
pushed a commit
to lxc/lxc-pkg-ubuntu
that referenced
this issue
Aug 22, 2017
Closes #1669. Closes #1678. Relates to systemd/systemd#6408. Signed-off-by: Christian Brauner <christian.brauner@ubuntu.com>
|
The fix is waiting in #7401. |
evverx
added a commit
to evverx/systemd
that referenced
this issue
Nov 21, 2017
…roup/unified` It's possible for `systemd` inside an unprivileged user namespace container to be able to mount `cgroup2` on `/sys/fs/cgroup/unified` without being able to create directories there. When this happens, `systemd` fails to boot, making it impossible to reexecute itself without restarting the container runtime. In this patch the issue is avoided by trying creating a temporary directory after mounting `cgroup2` and falling back to `v1` if `mkdir` fails. Closes systemd#6408 and lxc/lxc#1678.
evverx
added a commit
to evverx/systemd
that referenced
this issue
Nov 21, 2017
…roup/unified` It's possible for `systemd` inside an unprivileged user namespace container to be able to mount `cgroup2` on `/sys/fs/cgroup/unified` without being able to create directories there. When this happens, `systemd` fails to boot, making it impossible to reexecute itself without restarting the container runtime. In this patch the issue is avoided by trying creating a temporary directory after mounting `cgroup2` and falling back to `v1` if `mkdir` fails. Closes systemd#6408 and lxc/lxc#1678.
brauner
pushed a commit
to brauner/systemd
that referenced
this issue
Nov 22, 2017
When systemd is running inside a container employing user namespaces it currently mounts the unified cgroup hierarchy without being able to write to it. This causes systemd to freeze during boot. This patch checks whether the unified cgroup hierarchy is writable. If it is not it will not mount it. This solution is based on a patch by Evgeny Vereshchagin. Closes systemd#6408. Closes lxc/lxc#1678 .
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Submission type
systemd version the issue has been seen with
Used distribution
In case of bug report: Expected behaviour you didn't see
In case of bug report: Unexpected behaviour you saw
In case of bug report: Steps to reproduce the problem
The text was updated successfully, but these errors were encountered: