Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ubuntu 23.10 (mantic) containers fail to start #788

Closed
dimitry-unified-streaming opened this issue Oct 14, 2023 · 16 comments
Closed

Ubuntu 23.10 (mantic) containers fail to start #788

dimitry-unified-streaming opened this issue Oct 14, 2023 · 16 comments
Labels
Incomplete Waiting on more information from reporter

Comments

@dimitry-unified-streaming
Copy link
Contributor

Starting a freshly created mantic (20231014_07:42) container results in:

# lxc-create -n foobar -t download -- -d ubuntu -r mantic -a amd64
Downloading the image index
Downloading the rootfs
Downloading the metadata
The image cache is now ready
Unpacking the rootfs

---
You just created an Ubuntu mantic amd64 (20231014_07:42) container.

To enable SSH, run: apt install openssh-server
No default root or user password are set by LXC.

# lxc-start -n foobar -F
systemd 253.5-1ubuntu6 running in system mode (+PAM +AUDIT +SELINUX +APPARMOR +IMA +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 -PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD -BPF_FRAMEWORK -XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified)
Detected virtualization lxc.
Detected architecture x86-64.

Welcome to Ubuntu 23.10!

Initializing machine ID from random generator.
Failed to fork off sandboxing environment for executing generators: Protocol error
[!!!!!!] Failed to start up manager.
Exiting PID 1...

This looks very much like systemd/systemd#27436, so it might indeed be due to some issue with /tmp not being available in the initial file system. I will check if I can work around it with a custom config file.

@dimitry-unified-streaming
Copy link
Contributor Author

Added a line:

lxc.mount.entry = tmpfs /tmp tmpfs rw,nosuid,nodev,create=dir 0 0

to the config file, but it still gives the same error. Next up is attempting to convince systemd to print more information...

@unxed
Copy link

unxed commented Oct 16, 2023

Also suffer from this issue.

@stgraber
Copy link
Member

I'm not having any problem here, what kernel are you using?

root@dakara:~# lxc-create -n foobar -t download -- -d ubuntu -r mantic -a amd64
Using image from local cache
Unpacking the rootfs

---
You just created an Ubuntu mantic amd64 (20231016_07:42) container.

To enable SSH, run: apt install openssh-server
No default root or user password are set by LXC.
root@dakara:~# lxc-start -n foobar -F
systemd 253.5-1ubuntu6 running in system mode (+PAM +AUDIT +SELINUX +APPARMOR +IMA +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 -PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD -BPF_FRAMEWORK -XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified)
Detected virtualization lxc.
Detected architecture x86-64.

Welcome to Ubuntu 23.10!

Initializing machine ID from random generator.
Queued start job for default target graphical.target.
[  OK  ] Created slice system-modprobe.slice - Slice /system/modprobe.
[  OK  ] Created slice user.slice - User and Session Slice.
[  OK  ] Started systemd-ask-password-console.path - Dispatch Password Requests to Console Directory Watch.
[  OK  ] Started systemd-ask-password-wall.path - Forward Password Requests to Wall Directory Watch.
[  OK  ] Reached target cryptsetup.target - Local Encrypted Volumes.
[  OK  ] Reached target integritysetup.target - Local Integrity Protected Volumes.
[  OK  ] Reached target paths.target - Path Units.
[  OK  ] Reached target remote-fs.target - Remote File Systems.
[  OK  ] Reached target slices.target - Slice Units.
[  OK  ] Reached target swap.target - Swaps.
[  OK  ] Reached target veritysetup.target - Local Verity Protected Volumes.
[  OK  ] Listening on syslog.socket - Syslog Socket.
[  OK  ] Listening on systemd-initctl.socket - initctl Compatibility Named Pipe.
[  OK  ] Listening on systemd-journald-dev-log.socket - Journal Socket (/dev/log).
[  OK  ] Listening on systemd-journald.socket - Journal Socket.
[  OK  ] Listening on systemd-networkd.socket - Network Service Netlink Socket.
[  OK  ] Reached target sockets.target - Socket Units.
         Mounting dev-mqueue.mount - POSIX Message Queue File System...
         Starting systemd-journald.service - Journal Service...
         Starting keyboard-setup.service - Set the console keyboard layout...
         Starting systemd-network-generator.service - Generate network units from Kernel command line...
         Starting systemd-remount-fs.service - Remount Root and Kernel File Systems...
         Starting systemd-sysctl.service - Apply Kernel Variables...
[  OK  ] Mounted dev-mqueue.mount - POSIX Message Queue File System.
[  OK  ] Finished systemd-network-generator.service - Generate network units from Kernel command line.
[  OK  ] Started systemd-journald.service - Journal Service.
[  OK  ] Finished keyboard-setup.service - Set the console keyboard layout.
[  OK  ] Finished systemd-remount-fs.service - Remount Root and Kernel File Systems.
[  OK  ] Finished systemd-sysctl.service - Apply Kernel Variables.
[  OK  ] Reached target network-pre.target - Preparation for Network.
         Starting systemd-journal-flush.service - Flush Journal to Persistent Storage...
         Starting systemd-sysusers.service - Create System Users...
[  OK  ] Finished systemd-sysusers.service - Create System Users.
         Starting systemd-tmpfiles-setup-dev.service - Create Static Device Nodes in /dev...
[  OK  ] Finished systemd-tmpfiles-setup-dev.service - Create Static Device Nodes in /dev.
[  OK  ] Reached target local-fs-pre.target - Preparation for Local File Systems.
[  OK  ] Reached target local-fs.target - Local File Systems.
         Starting console-setup.service - Set console font and keymap...
         Starting systemd-networkd.service - Network Configuration...
[  OK  ] Finished console-setup.service - Set console font and keymap.
[  OK  ] Finished systemd-journal-flush.service - Flush Journal to Persistent Storage.
         Starting systemd-tmpfiles-setup.service - Create Volatile Files and Directories...
[  OK  ] Finished systemd-tmpfiles-setup.service - Create Volatile Files and Directories.
         Starting systemd-resolved.service - Network Name Resolution...
[  OK  ] Reached target time-set.target - System Time Set.
         Starting systemd-update-utmp.service - Record System Boot/Shutdown in UTMP...
[  OK  ] Finished systemd-update-utmp.service - Record System Boot/Shutdown in UTMP.
[  OK  ] Started systemd-networkd.service - Network Configuration.
[  OK  ] Started systemd-resolved.service - Network Name Resolution.
[  OK  ] Reached target network.target - Network.
[  OK  ] Reached target nss-lookup.target - Host and Network Name Lookups.
[  OK  ] Reached target sysinit.target - System Initialization.
[  OK  ] Started apt-daily.timer - Daily apt download activities.
[  OK  ] Started apt-daily-upgrade.timer - Daily apt upgrade and clean activities.
[  OK  ] Started dpkg-db-backup.timer - Daily dpkg database backup timer.
[  OK  ] Started e2scrub_all.timer - Periodic ext4 Online Metadata Check for All Filesystems.
[  OK  ] Started logrotate.timer - Daily rotation of log files.
[  OK  ] Started motd-news.timer - Message of the Day.
[  OK  ] Started systemd-tmpfiles-clean.timer - Daily Cleanup of Temporary Directories.
[  OK  ] Reached target timers.target - Timer Units.
[  OK  ] Listening on dbus.socket - D-Bus System Message Bus Socket.
[  OK  ] Reached target basic.target - Basic System.
[  OK  ] Started cron.service - Regular background program processing daemon.
         Starting dbus.service - D-Bus System Message Bus...
[  OK  ] Started dmesg.service - Save initial kernel messages after boot.
         Starting rsyslog.service - System Logging Service...
         Starting systemd-logind.service - User Login Management...
         Starting systemd-user-sessions.service - Permit User Sessions...
[  OK  ] Started dbus.service - D-Bus System Message Bus.
[  OK  ] Finished systemd-user-sessions.service - Permit User Sessions.
[  OK  ] Started console-getty.service - Console Getty.
[  OK  ] Created slice system-getty.slice - Slice /system/getty.
[  OK  ] Reached target getty.target - Login Prompts.
[  OK  ] Started rsyslog.service - System Logging Service.
[  OK  ] Started systemd-logind.service - User Login Management.
[  OK  ] Reached target multi-user.target - Multi-User System.
[  OK  ] Reached target graphical.target - Graphical Interface.
         Starting systemd-update-utmp-runlevel.service - Record Runlevel Change in UTMP...
[  OK  ] Finished systemd-update-utmp-runlevel.service - Record Runlevel Change in UTMP.

Ubuntu 23.10 foobar console

foobar login: 

@stgraber stgraber added the Incomplete Waiting on more information from reporter label Oct 16, 2023
@dimitry-unified-streaming
Copy link
Contributor Author

In my case, it is with an Ubuntu 22.04 LTS host, using the linux-generic-hwe-22.04 kernel:

Linux containerhost 6.2.0-34-generic #34~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Thu Sep  7 13:12:03 UTC 2 x86_64 x86_64 x86_64 GNU/Linux

Another note is that I am using systemd.unified_cgroup_hierarchy=0 in the kernel boot parameters, to work around problems starting containers with certain OSes in them (typically "Failed to mount cgroup at /sys/fs/cgroup/systemd: Operation not permitted", see many tickets in lxc itself). However, this does not prevent most other images from starting.

@stgraber
Copy link
Member

Please try on a system without unified_cgroup_hierarchy=0 as I suspect very recent systemd such as the one in mantic has now done away with cgroup1 support.

And yes, that effectively means that you'll have to chose between having your system being able to run older distros (Ubuntu 18.04, CentOS 7, ...) and being able to run modern distros as older ones don't understand cgroup2 and newer ones no longer understand cgroup1.

@unxed
Copy link

unxed commented Oct 16, 2023

No special boot options in my case. Host os is

$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=20.04
DISTRIB_CODENAME=focal
DISTRIB_DESCRIPTION="Ubuntu 20.04.6 LTS"
$ sudo lxc-create -n foobar -t download -- -d ubuntu -r mantic -a amd64
Using image from local cache
Unpacking the rootfs

---
You just created an Ubuntu mantic amd64 (20231016_07:42) container.

To enable SSH, run: apt install openssh-server
No default root or user password are set by LXC.
----------------------------------------------------------------------------------------------------------------------------
$ sudo lxc-attach -n foobar
lxc-attach: foobar: attach.c: get_attach_context: 405 Connection refused - Failed to get init pid
lxc-attach: foobar: attach.c: lxc_attach: 1469 Connection refused - Failed to get attach context
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
$ sudo lxc-start -F foobar
systemd 253.5-1ubuntu6 running in system mode (+PAM +AUDIT +SELINUX +APPARMOR +IMA +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL
+ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 -PWQUALITY +P11KIT +QRENCODE +TPM2
 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD -BPF_FRAMEWORK -XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified)
Detected virtualization lxc.
Detected architecture x86-64.

Welcome to Ubuntu 23.10!

Initializing machine ID from random generator.
Failed to fork off sandboxing environment for executing generators: Protocol error
[!!!!!!] Failed to start up manager.
Exiting PID 1...
----------------------------------------------------------------------------------------------------------------------------
$ uname -a
Linux main-2018 5.4.0-152-generic #169-Ubuntu SMP Tue Jun 6 22:23:09 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

@stgraber
Copy link
Member

Ubuntu 20.04 defaults to cgroup1 so you'd need to pass a boot parameter to force cgroup2 or upgrade your system to 22.04

@unxed
Copy link

unxed commented Oct 17, 2023

Same with 22.04 machine:

systemd 253.5-1ubuntu6 running in system mode (+PAM +AUDIT +SELINUX +APPARMOR +IMA +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL
+ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 -PWQUALITY +P11KIT +QRENCODE +TPM2
 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD -BPF_FRAMEWORK -XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified)
Detected virtualization lxc.
Detected architecture x86-64.

Welcome to Ubuntu 23.10!

Initializing machine ID from random generator.
Failed to fork off sandboxing environment for executing generators: Protocol error
[!!!!!!] Failed to start up manager.
Exiting PID 1...
$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=22.04
DISTRIB_CODENAME=jammy
DISTRIB_DESCRIPTION="Ubuntu 22.04.3 LTS"
$ uname -a
Linux shuttle 5.15.0-79-generic #86-Ubuntu SMP Mon Jul 10 16:07:21 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
$ grep cgroup /proc/filesystems
nodev   cgroup
nodev   cgroup2

@stgraber
Copy link
Member

@unxed can you show:

  • cat /proc/self/cgroup
  • cat /proc/cmdline

@unxed
Copy link

unxed commented Oct 17, 2023

For sure!

$ cat /proc/self/cgroup
0::/user.slice/user-1000.slice/session-4881.scope
$ cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-5.15.0-79-generic root=UUID=51630eb5-0dcb-4dfc-ae3c-b59d08372d35 ro maybe-ubiquity elevator=deadline

@stgraber
Copy link
Member

stgraber commented Oct 17, 2023

Can you show the output of sudo dmesg | tail -n 50 following an lxc-start failure?

@unxed
Copy link

unxed commented Oct 17, 2023

$ sudo lxc-create -n foobar -t download -- -d ubuntu -r mantic -a amd64
[sudo] password for unxed:
Using image from local cache
Unpacking the rootfs

---
You just created an Ubuntu mantic amd64 (20231017_07:42) container.

To enable SSH, run: apt install openssh-server
No default root or user password are set by LXC.

$ sudo lxc-start -F foobar
[sudo] password for unxed:
systemd 253.5-1ubuntu6 running in system mode (+PAM +AUDIT +SELINUX +APPARMOR +IMA +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL
+ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 -PWQUALITY +P11KIT +QRENCODE +TPM2
 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD -BPF_FRAMEWORK -XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified)
Detected virtualization lxc.
Detected architecture x86-64.

Welcome to Ubuntu 23.10!

Initializing machine ID from random generator.
Failed to fork off sandboxing environment for executing generators: Protocol error
[!!!!!!] Failed to start up manager.
Exiting PID 1...

$ sudo dmesg | tail -n 50
[sudo] password for unxed:
[5261497.030619] systemd[1]: Started OpenBSD Secure Shell server.
[5261497.400204] systemd[1]: Started Journal Service.
[5261620.063052] lxcbr0: port 1(vethfBK6cH) entered blocking state
[5261620.063077] lxcbr0: port 1(vethfBK6cH) entered disabled state
[5261620.063401] device vethfBK6cH entered promiscuous mode
[5261620.067054] eth0: renamed from vethrXzDK4
[5261620.094156] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[5261620.094365] IPv6: ADDRCONF(NETDEV_CHANGE): vethfBK6cH: link becomes ready
[5261620.094611] lxcbr0: port 1(vethfBK6cH) entered blocking state
[5261620.094633] lxcbr0: port 1(vethfBK6cH) entered forwarding state
[5261620.094894] IPv6: ADDRCONF(NETDEV_CHANGE): lxcbr0: link becomes ready
[5261620.348185] kauditd_printk_skb: 10 callbacks suppressed
[5261620.348197] audit: type=1400 audit(1697552034.547:158): apparmor="DENIED" operation="mount" info="failed flags match" e
rror=-13 profile="lxc-container-default-cgns" name="/" pid=1140387 comm="(sd-gens)" flags="rw, rslave"
[5261620.374238] lxcbr0: port 1(vethfBK6cH) entered disabled state
[5261620.375680] device vethfBK6cH left promiscuous mode
[5261620.375712] lxcbr0: port 1(vethfBK6cH) entered disabled state
[5262838.970097] systemd[1]: Stopping Journal Service...
[5262838.982882] systemd-journald[1140108]: Received SIGTERM from PID 1 (systemd).
[5262839.022649] systemd[1]: systemd-journald.service: Deactivated successfully.
[5262839.025281] systemd[1]: Stopped Journal Service.
[5262839.025636] systemd[1]: systemd-journald.service: Consumed 1.072s CPU time.
[5262839.064033] systemd[1]: Starting Journal Service...
[5262839.300625] systemd[1]: Started Journal Service.
[5263556.597339] systemd[1]: systemd 249.11-0ubuntu3.10 running in system mode (+PAM +AUDIT +SELINUX +APPARMOR +IMA +SMACK +
SECCOMP +GCRYPT +GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 -
PWQUALITY -P11KIT -QRENCODE +BZIP2 +LZ4 +XZ +ZLIB +ZSTD -XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified)
[5263556.618231] systemd[1]: Detected architecture x86-64.
[5263918.812172] lxcbr0: port 1(vethPhGmOd) entered blocking state
[5263918.812195] lxcbr0: port 1(vethPhGmOd) entered disabled state
[5263918.812549] device vethPhGmOd entered promiscuous mode
[5263918.816057] eth0: renamed from vethYdYb0r
[5263918.847010] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[5263918.847213] IPv6: ADDRCONF(NETDEV_CHANGE): vethPhGmOd: link becomes ready
[5263918.847430] lxcbr0: port 1(vethPhGmOd) entered blocking state
[5263918.847448] lxcbr0: port 1(vethPhGmOd) entered forwarding state
[5263918.847680] IPv6: ADDRCONF(NETDEV_CHANGE): lxcbr0: link becomes ready
[5263919.227166] audit: type=1400 audit(1697554333.430:159): apparmor="DENIED" operation="mount" info="failed flags match" e
rror=-13 profile="lxc-container-default-cgns" name="/" pid=1174782 comm="(sd-gens)" flags="rw, rslave"
[5263919.266164] lxcbr0: port 1(vethPhGmOd) entered disabled state
[5263919.270083] device vethPhGmOd left promiscuous mode
[5263919.270118] lxcbr0: port 1(vethPhGmOd) entered disabled state
[5271036.294317] lxcbr0: port 1(vethpZ20hp) entered blocking state
[5271036.294341] lxcbr0: port 1(vethpZ20hp) entered disabled state
[5271036.295398] device vethpZ20hp entered promiscuous mode
[5271036.305961] eth0: renamed from veth2AbF9S
[5271036.333495] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[5271036.333724] IPv6: ADDRCONF(NETDEV_CHANGE): vethpZ20hp: link becomes ready
[5271036.333991] lxcbr0: port 1(vethpZ20hp) entered blocking state
[5271036.334012] lxcbr0: port 1(vethpZ20hp) entered forwarding state
[5271036.661234] audit: type=1400 audit(1697561450.860:160): apparmor="DENIED" operation="mount" info="failed flags match" e
rror=-13 profile="lxc-container-default-cgns" name="/" pid=1176445 comm="(sd-gens)" flags="rw, rslave"
[5271036.703675] lxcbr0: port 1(vethpZ20hp) entered disabled state
[5271036.707953] device vethpZ20hp left promiscuous mode
[5271036.707985] lxcbr0: port 1(vethpZ20hp) entered disabled state

@stgraber
Copy link
Member

Ah, right so you're hitting:

[5271036.661234] audit: type=1400 audit(1697561450.860:160): apparmor="DENIED" operation="mount" info="failed flags match" e
rror=-13 profile="lxc-container-default-cgns" name="/" pid=1176445 comm="(sd-gens)" flags="rw, rslave"

This is something that's been fixed in lxc/lxc#4295 but hasn't been part of an LXC release yet as we're waiting on the various distros to catch the newer apparmor first.

If your system has apparmor 2.13.8 or 3.0.10 (and newer), then you could try applying the changes directly to your files in /etc/apparmor.d, reboot the system and try again:
https://github.com/lxc/lxc/pull/4295/files

That should take care of the DENIED that you're seeing in dmesg and hopefully let systemd boot properly.

This whole mess came from a bad security bug in apparmor which apparmor upstream pretty much ignored for a decade until @mihalicyn decided to just go and fix it for them earlier this year.
We've been waiting for the fix to trickle into all distros before we get to relax our policies without causing major security issues (allowing the container to take over the whole system).

@unxed
Copy link

unxed commented Oct 17, 2023

This helped, thank you! Works both on 20.04 and 22.04

Had to update

/etc/apparmor.d/abstractions/lxc/container-base

and then

sudo apparmor_parser -r /etc/apparmor.d/lxc-containers
sudo systemctl restart apparmor

@stgraber
Copy link
Member

Closing as there's nothing we can really do with the image, things will improve as distros update their apparmor and liblxc.

@dimitry-unified-streaming
Copy link
Contributor Author

Here on Ubuntu 22.04 this also worked around the problem, but as the apparmor version seems to be 3.0.4-2ubuntu2.3, I guess the original "FIXME: This currently doesn't work due to the apparmor parser treating those as allowing all mounts" comment might apply?

Especially since @stgraber mentions:

If your system has apparmor 2.13.8 or 3.0.10 (and newer)

which is unfortunately not the case on Ubuntu 22.04. But I don't get any complaints from apparmor_parser.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Incomplete Waiting on more information from reporter
Development

No branches or pull requests

3 participants