New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unable to start a container due to error: No medium found #1669

Closed
graysky2 opened this Issue Jul 3, 2017 · 76 comments

Comments

5 participants
@graysky2

graysky2 commented Jul 3, 2017

Required information

  • Distribution: Arch ARM (aarch64)
  • Distribution version: rolling release
  • The output of
lxc-start --version
2.0.8
# lxc-checkconfig
--- Namespaces ---
Namespaces: enabled
Utsname namespace: enabled
Ipc namespace: enabled
Pid namespace: enabled
User namespace: enabled
Warning: newuidmap is not setuid-root
Warning: newgidmap is not setuid-root
Network namespace: enabled
Multiple /dev/pts instances: enabled

--- Control groups ---
Cgroup: enabled
Cgroup clone_children flag: enabled
Cgroup device: enabled
Cgroup sched: enabled
Cgroup cpu account: enabled
Cgroup memory controller: enabled
Cgroup cpuset: enabled

--- Misc ---
Veth pair device: enabled
Macvlan: enabled
Vlan: enabled
Bridges: enabled
Advanced netfilter: enabled
CONFIG_NF_NAT_IPV4: enabled
CONFIG_NF_NAT_IPV6: enabled
CONFIG_IP_NF_TARGET_MASQUERADE: enabled
CONFIG_IP6_NF_TARGET_MASQUERADE: enabled
CONFIG_NETFILTER_XT_TARGET_CHECKSUM: enabled
FUSE (for use with lxcfs): enabled

--- Checkpoint/Restore ---
checkpoint restore: missing
CONFIG_FHANDLE: enabled
CONFIG_EVENTFD: enabled
CONFIG_EPOLL: enabled
CONFIG_UNIX_DIAG: enabled
CONFIG_INET_DIAG: enabled
CONFIG_PACKET_DIAG: enabled
CONFIG_NETLINK_DIAG: missing
File capabilities: enabled
uname -a
Linux odroid64 3.14.79-25-ARCH #1 SMP PREEMPT Fri Jun 2 20:10:11 MDT 2017 aarch64 GNU/Linux
# cat /proc/self/cgroup
2:devices:/user.slice
11:perf_event:/
10:hugetlb:/
9:blkio:/
8:cpuset:/
7:net_cls:/
6:freezer:/
5:memory:/
4:debug:/
3:cpu,cpuacct:/
2:name=systemd:/user.slice/user-1000.slice/session-c1.scope
rootfs / rootfs rw 0 0
proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
sys /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
dev /dev devtmpfs rw,nosuid,relatime,size=733692k,nr_inodes=183423,mode=755 0 0
run /run tmpfs rw,nosuid,nodev,relatime,mode=755 0 0
/dev/mmcblk0p1 / ext4 rw,relatime,data=ordered 0 0
securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0
tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0
devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0
tmpfs /sys/fs/cgroup tmpfs ro,nosuid,nodev,noexec,mode=755 0 0
cgroup /sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0
cgroup /sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0
cgroup /sys/fs/cgroup/debug cgroup rw,nosuid,nodev,noexec,relatime,debug 0 0
cgroup /sys/fs/cgroup/memory cgroup rw,nosuid,nodev,noexec,relatime,memory 0 0
cgroup /sys/fs/cgroup/devices cgroup rw,nosuid,nodev,noexec,relatime,devices 0 0
cgroup /sys/fs/cgroup/blkio cgroup rw,nosuid,nodev,noexec,relatime,blkio 0 0
cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0
cgroup /sys/fs/cgroup/perf_event cgroup rw,nosuid,nodev,noexec,relatime,perf_event 0 0
cgroup /sys/fs/cgroup/freezer cgroup rw,nosuid,nodev,noexec,relatime,freezer 0 0
cgroup /sys/fs/cgroup/hugetlb cgroup rw,nosuid,nodev,noexec,relatime,hugetlb 0 0
cgroup /sys/fs/cgroup/net_cls cgroup rw,nosuid,nodev,noexec,relatime,net_cls 0 0
systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=34,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0
mqueue /dev/mqueue mqueue rw,relatime 0 0
debugfs /sys/kernel/debug debugfs rw,relatime 0 0
configfs /sys/kernel/config configfs rw,relatime 0 0
hugetlbfs /dev/hugepages hugetlbfs rw,relatime 0 0
tmpfs /tmp tmpfs rw,nosuid,nodev 0 0
tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=175848k,mode=700,uid=1000,gid=100 0 0

Issue description

I have been running lxcs for over a year without issues on an Odroid-C2 running Arch ARM (aarch64).  Recently, I am unable to start any of the containers due to some errors around cgroups.  This happened today after an update where 4 packages got updated both on the host and in the container.

libsystemd (232-8 -> 233-6)
systemd (232-8 -> 233-6)
device-mapper (2.02.171-1 -> 2.02.172-1) 
python2 (2.7.13-2 -> 2.7.13-3)
systemd-sysvcompat (232-8 -> 233-6)

Below is an example of the error when starting in foreground mode:

# lxc-start -n base-odroid64 -F
systemd 233 running in system mode. (+PAM -AUDIT -SELINUX -IMA -APPARMOR +SMACK -SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN default-hierarchy=hybrid)
Detected virtualization lxc.
Detected architecture arm64.

Welcome to Arch Linux ARM!

Set hostname to <base-odroid64>.
Cannot determine cgroup we are running in: No medium found
Failed to allocate manager object: No medium found
[!!!!!!] Failed to allocate manager object, freezing.
Freezing execution.

EDIT - Restoring my container from a backup running the older versions of these packages fixes the problem. The host can run the updated packages but the container CANNOT run them or rebooting it causes what I am showing above.

Steps to reproduce

  1. Try to start the container using the command given.

Information to attach

  • any relevant kernel output (dmesg)
  • container log (The file from running lxc-start -n <c> -l DEBUG -o <log>)
  • the containers configuration file

config.txt
debug.log.txt
dmesg.txt

@brauner

This comment has been minimized.

Show comment
Hide comment
@brauner

brauner Jul 3, 2017

Member

The problem is (as always) the cgroup v2 hierarchy (/cc @hallyn, @stgraber). If you look at the container's boot output:

systemd 233 running in system mode. (+PAM -AUDIT -SELINUX -IMA -APPARMOR +SMACK -SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN default-hierarchy=hybrid)
Detected virtualization lxc.
Detected architecture arm64.

you see default-hierarchy=hybrid which means systemd is trying to mount a mixed hierarchy of cgroup v1 + cgroup v2 which is currently unsupported or let's call it an experimental preview. This options is iirc a compile-time option. Which means your distro (ArchLinux) uses the hybrid cgroup hierarchy as default. Arch shouldn't do this yet since regardless of what upstream says cgroup v2 is experimental to downstream as long as it hasn't reached feature parity with cgroup v1.

To workaround this problem you can set:

lxc.init_cmd = /sbin/init verbose systemd.unified_cgroup_hierarchy=false

Can you please try and report back?

Member

brauner commented Jul 3, 2017

The problem is (as always) the cgroup v2 hierarchy (/cc @hallyn, @stgraber). If you look at the container's boot output:

systemd 233 running in system mode. (+PAM -AUDIT -SELINUX -IMA -APPARMOR +SMACK -SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN default-hierarchy=hybrid)
Detected virtualization lxc.
Detected architecture arm64.

you see default-hierarchy=hybrid which means systemd is trying to mount a mixed hierarchy of cgroup v1 + cgroup v2 which is currently unsupported or let's call it an experimental preview. This options is iirc a compile-time option. Which means your distro (ArchLinux) uses the hybrid cgroup hierarchy as default. Arch shouldn't do this yet since regardless of what upstream says cgroup v2 is experimental to downstream as long as it hasn't reached feature parity with cgroup v1.

To workaround this problem you can set:

lxc.init_cmd = /sbin/init verbose systemd.unified_cgroup_hierarchy=false

Can you please try and report back?

@brauner brauner added the Incomplete label Jul 3, 2017

@graysky2

This comment has been minimized.

Show comment
Hide comment
@graysky2

graysky2 Jul 3, 2017

Thank you for the quick reply. I added that line to the container config but it is still unable to boot with this new version of systemd.

graysky2 commented Jul 3, 2017

Thank you for the quick reply. I added that line to the container config but it is still unable to boot with this new version of systemd.

@brauner

This comment has been minimized.

Show comment
Hide comment
@brauner

brauner Jul 3, 2017

Member
  • Is it still freezing during boot?
  • Is the cgroup output from the host before or after the upgrade? If before, can you show the ouput after the upgrade please?
Member

brauner commented Jul 3, 2017

  • Is it still freezing during boot?
  • Is the cgroup output from the host before or after the upgrade? If before, can you show the ouput after the upgrade please?
@graysky2

This comment has been minimized.

Show comment
Hide comment
@graysky2

graysky2 Jul 3, 2017

Is it still freezing during boot?

Yes, if I simply modify the config adding that line, the boot still freezes. Only restoring an older backup of my container (using the old systemd version) gives a bootable container. The problem definitely seems related to the updated version of systemd. For example, within that functional backup, I simply update the system, journalctl throws errors similar to what I see starting the updated container:

:: Starting full system upgrade...
resolving dependencies...
looking for conflicting packages...
warning: dependency cycle detected:
warning: cryptsetup will be installed before its systemd dependency

Packages (4) cryptsetup-1.7.5-1  libsystemd-233-6  systemd-233-6  systemd-sysvcompat-233-6

Total Installed Size:  20.36 MiB
Net Upgrade Size:       1.93 MiB

:: Proceed with installation? [Y/n] 
(4/4) checking keys in keyring                                               [############################################] 100%
(4/4) checking package integrity                                             [############################################] 100%
(4/4) loading package files                                                  [############################################] 100%
(4/4) checking for file conflicts                                            [############################################] 100%
:: Processing package changes...
(1/4) upgrading libsystemd                                                   [############################################] 100%
(2/4) installing cryptsetup                                                  [############################################] 100%
(3/4) upgrading systemd                                                      [############################################] 100%

Broadcast message from systemd-journald@base-odroid64 (Mon 2017-07-03 06:31:45 EDT):

systemd[1]: Failed to allocate manager object: No medium found


Broadcast message from systemd-journald@base-odroid64 (Mon 2017-07-03 06:31:45 EDT):

systemd[1]: Freezing execution.

(4/4) upgrading systemd-sysvcompat                                           [############################################] 100%
:: Running post-transaction hooks...
(1/4) Updating udev hardware database...
(2/4) Updating system user accounts...
(3/4) Creating temporary files...
(4/4) Arming ConditionNeedsUpdate...

After that happens, I cannot boot the container with or without the lxc.init_cmd = /sbin/init verbose systemd.unified_cgroup_hierarchy=false line in the container config.

Is the cgroup output from the host before or after the upgrade? If before, can you show the ouput after the upgrade please?

To be unambiguous:
On the host under systemd version 233-6:

# cat /proc/self/cgroup
12:cpuset:/
11:hugetlb:/
10:blkio:/
9:perf_event:/
8:debug:/
7:devices:/user.slice
6:net_cls:/
5:memory:/
4:cpu,cpuacct:/
3:freezer:/
2:name=systemd:/user.slice/user-1000.slice/session-c1.scope

On the host under systemd version 232-8:

# cat /proc/self/cgroup
12:freezer:/
11:cpuset:/
10:memory:/
9:net_cls:/
8:debug:/
7:hugetlb:/
6:perf_event:/
5:devices:/user.slice
4:blkio:/
3:cpu,cpuacct:/
2:name=systemd:/user.slice/user-1000.slice/session-c1.scope

graysky2 commented Jul 3, 2017

Is it still freezing during boot?

Yes, if I simply modify the config adding that line, the boot still freezes. Only restoring an older backup of my container (using the old systemd version) gives a bootable container. The problem definitely seems related to the updated version of systemd. For example, within that functional backup, I simply update the system, journalctl throws errors similar to what I see starting the updated container:

:: Starting full system upgrade...
resolving dependencies...
looking for conflicting packages...
warning: dependency cycle detected:
warning: cryptsetup will be installed before its systemd dependency

Packages (4) cryptsetup-1.7.5-1  libsystemd-233-6  systemd-233-6  systemd-sysvcompat-233-6

Total Installed Size:  20.36 MiB
Net Upgrade Size:       1.93 MiB

:: Proceed with installation? [Y/n] 
(4/4) checking keys in keyring                                               [############################################] 100%
(4/4) checking package integrity                                             [############################################] 100%
(4/4) loading package files                                                  [############################################] 100%
(4/4) checking for file conflicts                                            [############################################] 100%
:: Processing package changes...
(1/4) upgrading libsystemd                                                   [############################################] 100%
(2/4) installing cryptsetup                                                  [############################################] 100%
(3/4) upgrading systemd                                                      [############################################] 100%

Broadcast message from systemd-journald@base-odroid64 (Mon 2017-07-03 06:31:45 EDT):

systemd[1]: Failed to allocate manager object: No medium found


Broadcast message from systemd-journald@base-odroid64 (Mon 2017-07-03 06:31:45 EDT):

systemd[1]: Freezing execution.

(4/4) upgrading systemd-sysvcompat                                           [############################################] 100%
:: Running post-transaction hooks...
(1/4) Updating udev hardware database...
(2/4) Updating system user accounts...
(3/4) Creating temporary files...
(4/4) Arming ConditionNeedsUpdate...

After that happens, I cannot boot the container with or without the lxc.init_cmd = /sbin/init verbose systemd.unified_cgroup_hierarchy=false line in the container config.

Is the cgroup output from the host before or after the upgrade? If before, can you show the ouput after the upgrade please?

To be unambiguous:
On the host under systemd version 233-6:

# cat /proc/self/cgroup
12:cpuset:/
11:hugetlb:/
10:blkio:/
9:perf_event:/
8:debug:/
7:devices:/user.slice
6:net_cls:/
5:memory:/
4:cpu,cpuacct:/
3:freezer:/
2:name=systemd:/user.slice/user-1000.slice/session-c1.scope

On the host under systemd version 232-8:

# cat /proc/self/cgroup
12:freezer:/
11:cpuset:/
10:memory:/
9:net_cls:/
8:debug:/
7:hugetlb:/
6:perf_event:/
5:devices:/user.slice
4:blkio:/
3:cpu,cpuacct:/
2:name=systemd:/user.slice/user-1000.slice/session-c1.scope
@brauner

This comment has been minimized.

Show comment
Hide comment
@brauner

brauner Jul 3, 2017

Member

Can you please show me the output of the container in the foreground again with lxc.init_cmd set?

Member

brauner commented Jul 3, 2017

Can you please show me the output of the container in the foreground again with lxc.init_cmd set?

@graysky2

This comment has been minimized.

Show comment
Hide comment
@graysky2

graysky2 Jul 3, 2017

Can you please show me the output of the container in the foreground again with lxc.init_cmd set?

You bet, do you want my host running systemd version 232-8 or 233-6?

graysky2 commented Jul 3, 2017

Can you please show me the output of the container in the foreground again with lxc.init_cmd set?

You bet, do you want my host running systemd version 232-8 or 233-6?

@brauner

This comment has been minimized.

Show comment
Hide comment
@brauner

brauner Jul 3, 2017

Member

233-6 :)

Member

brauner commented Jul 3, 2017

233-6 :)

@graysky2

This comment has been minimized.

Show comment
Hide comment
@graysky2

graysky2 Jul 3, 2017

OK, with the host on system 233-6 here are two conditions with both having the lxc.init_cmd = /sbin/init verbose systemd.unified_cgroup_hierarchy=false set in the container's config:

  1. container is on systemd 232-8 (ie from the backup version)
  2. container is on systemd 233-6 (I updated the backup)

Number 1

# lxc-start -n base-odroid64 -F
systemd 232 running in system mode. (+PAM -AUDIT -SELINUX -IMA -APPARMOR +SMACK -SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN)
Detected virtualization lxc.
Detected architecture arm64.

Welcome to Arch Linux ARM!

Set hostname to <base-odroid64>.
Failed to install release agent, ignoring: No such file or directory
[  OK  ] Listening on /dev/initctl Compatibility Named Pipe.
[  OK  ] Created slice User and Session Slice.
[  OK  ] Listening on Network Service Netlink Socket.
[  OK  ] Started Forward Password Requests to Wall Directory Watch.
[  OK  ] Created slice System Slice.
         Mounting Huge Pages File System...
[  OK  ] Started Dispatch Password Requests to Console Directory Watch.
[  OK  ] Reached target Paths.
[  OK  ] Created slice system-container\x2dgetty.slice.
[  OK  ] Listening on Journal Socket (/dev/log).
[  OK  ] Created slice system-getty.slice.
[  OK  ] Listening on Journal Socket.
[  OK  ] Reached target Slices.
[  OK  ] Reached target Swap.
         Mounting Temporary Directory...
         Starting Remount Root and Kernel File Systems...
         Starting Journal Service...
         Mounting POSIX Message Queue File System...
[  OK  ] Reached target Encrypted Volumes.
[  OK  ] Reached target Remote File Systems.
[  OK  ] Listening on Process Core Dump Socket.
         Starting Apply Kernel Variables...
[  OK  ] Listening on Device-mapper event daemon FIFOs.
[  OK  ] Mounted POSIX Message Queue File System.
[  OK  ] Mounted Huge Pages File System.
[  OK  ] Mounted Temporary Directory.
[  OK  ] Started Remount Root and Kernel File Systems.
[  OK  ] Reached target Local File Systems (Pre).
[  OK  ] Reached target Local File Systems.
[  OK  ] Started Journal Service.
[  OK  ] Started Apply Kernel Variables.
         Starting Flush Journal to Persistent Storage...
[  OK  ] Started Flush Journal to Persistent Storage.
         Starting Create Volatile Files and Directories...
[  OK  ] Started Create Volatile Files and Directories.
         Starting Update UTMP about System Boot/Shutdown...
[  OK  ] Started Update UTMP about System Boot/Shutdown.
[  OK  ] Reached target System Initialization.
[  OK  ] Started Daily Cleanup of Temporary Directories.
[  OK  ] Started Weekly ad-serving domains gathering.
[  OK  ] Started Daily verification of password and group files.
[  OK  ] Listening on D-Bus System Message Bus Socket.
[  OK  ] Reached target Sockets.
[  OK  ] Started Daily man-db cache update.
[  OK  ] Reached target Basic System.
[  OK  ] Started Pi-hole FTL engine.
[  OK  ] Started D-Bus System Message Bus.
[  OK  ] Started Daily rotation of log files.
         Starting Network Service...
[  OK  ] Started Daily reset of dnsmasq/pi-hole query log.
[  OK  ] Reached target Timers.
         Starting Login Service...
[  OK  ] Started Login Service.
[  OK  ] Started Network Service.
[  OK  ] Reached target Network.
         Starting A lightweight DHCP and caching DNS server...
         Starting The PHP FastCGI Process Manager...
[  OK  ] Started OpenSSH Daemon.
         Starting Permit User Sessions...
         Starting A high performance web server and a reverse proxy server...
         Starting Network Name Resolution...
         Starting Hostname Service...
[  OK  ] Started Permit User Sessions.
[  OK  ] Started Getty on lxc/tty2.
[  OK  ] Started Container Getty on /dev/pts/3.
[  OK  ] Started Getty on lxc/tty3.
[  OK  ] Started Container Getty on /dev/pts/0.
[  OK  ] Started Container Getty on /dev/pts/5.
[  OK  ] Started Getty on lxc/tty4.
[  OK  ] Started Container Getty on /dev/pts/2.
[  OK  ] Started Getty on lxc/tty6.
[  OK  ] Started Getty on lxc/tty1.
[  OK  ] Started Container Getty on /dev/pts/1.
[  OK  ] Started Getty on lxc/tty5.
[  OK  ] Started Console Getty.
[  OK  ] Started Container Getty on /dev/pts/4.
[  OK  ] Reached target Login Prompts.
[  OK  ] Started A lightweight DHCP and caching DNS server.
[  OK  ] Started The PHP FastCGI Process Manager.
[  OK  ] Started Network Name Resolution.
[  OK  ] Started A high performance web server and a reverse proxy server.
[  OK  ] Reached target Multi-User System.
[  OK  ] Started Hostname Service.

Arch Linux 3.14.79-25-ARCH (console)

base-odroid64 login:

Number 2:

# lxc-start -n base-odroid64 -F
systemd 233 running in system mode. (+PAM -AUDIT -SELINUX -IMA -APPARMOR +SMACK -SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN default-hierarchy=hybrid)
Detected virtualization lxc.
Detected architecture arm64.

Welcome to Arch Linux ARM!

Set hostname to <base-odroid64>.
Cannot determine cgroup we are running in: No medium found
Failed to allocate manager object: No medium found
[!!!!!!] Failed to allocate manager object, freezing.
Freezing execution.

It's clear that the updated version ignores the directive but why?

systemd 232 running in system mode. (+PAM -AUDIT -SELINUX -IMA -APPARMOR +SMACK -SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN)

vs

systemd 233 running in system mode. (+PAM -AUDIT -SELINUX -IMA -APPARMOR +SMACK -SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN default-hierarchy=hybrid)

graysky2 commented Jul 3, 2017

OK, with the host on system 233-6 here are two conditions with both having the lxc.init_cmd = /sbin/init verbose systemd.unified_cgroup_hierarchy=false set in the container's config:

  1. container is on systemd 232-8 (ie from the backup version)
  2. container is on systemd 233-6 (I updated the backup)

Number 1

# lxc-start -n base-odroid64 -F
systemd 232 running in system mode. (+PAM -AUDIT -SELINUX -IMA -APPARMOR +SMACK -SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN)
Detected virtualization lxc.
Detected architecture arm64.

Welcome to Arch Linux ARM!

Set hostname to <base-odroid64>.
Failed to install release agent, ignoring: No such file or directory
[  OK  ] Listening on /dev/initctl Compatibility Named Pipe.
[  OK  ] Created slice User and Session Slice.
[  OK  ] Listening on Network Service Netlink Socket.
[  OK  ] Started Forward Password Requests to Wall Directory Watch.
[  OK  ] Created slice System Slice.
         Mounting Huge Pages File System...
[  OK  ] Started Dispatch Password Requests to Console Directory Watch.
[  OK  ] Reached target Paths.
[  OK  ] Created slice system-container\x2dgetty.slice.
[  OK  ] Listening on Journal Socket (/dev/log).
[  OK  ] Created slice system-getty.slice.
[  OK  ] Listening on Journal Socket.
[  OK  ] Reached target Slices.
[  OK  ] Reached target Swap.
         Mounting Temporary Directory...
         Starting Remount Root and Kernel File Systems...
         Starting Journal Service...
         Mounting POSIX Message Queue File System...
[  OK  ] Reached target Encrypted Volumes.
[  OK  ] Reached target Remote File Systems.
[  OK  ] Listening on Process Core Dump Socket.
         Starting Apply Kernel Variables...
[  OK  ] Listening on Device-mapper event daemon FIFOs.
[  OK  ] Mounted POSIX Message Queue File System.
[  OK  ] Mounted Huge Pages File System.
[  OK  ] Mounted Temporary Directory.
[  OK  ] Started Remount Root and Kernel File Systems.
[  OK  ] Reached target Local File Systems (Pre).
[  OK  ] Reached target Local File Systems.
[  OK  ] Started Journal Service.
[  OK  ] Started Apply Kernel Variables.
         Starting Flush Journal to Persistent Storage...
[  OK  ] Started Flush Journal to Persistent Storage.
         Starting Create Volatile Files and Directories...
[  OK  ] Started Create Volatile Files and Directories.
         Starting Update UTMP about System Boot/Shutdown...
[  OK  ] Started Update UTMP about System Boot/Shutdown.
[  OK  ] Reached target System Initialization.
[  OK  ] Started Daily Cleanup of Temporary Directories.
[  OK  ] Started Weekly ad-serving domains gathering.
[  OK  ] Started Daily verification of password and group files.
[  OK  ] Listening on D-Bus System Message Bus Socket.
[  OK  ] Reached target Sockets.
[  OK  ] Started Daily man-db cache update.
[  OK  ] Reached target Basic System.
[  OK  ] Started Pi-hole FTL engine.
[  OK  ] Started D-Bus System Message Bus.
[  OK  ] Started Daily rotation of log files.
         Starting Network Service...
[  OK  ] Started Daily reset of dnsmasq/pi-hole query log.
[  OK  ] Reached target Timers.
         Starting Login Service...
[  OK  ] Started Login Service.
[  OK  ] Started Network Service.
[  OK  ] Reached target Network.
         Starting A lightweight DHCP and caching DNS server...
         Starting The PHP FastCGI Process Manager...
[  OK  ] Started OpenSSH Daemon.
         Starting Permit User Sessions...
         Starting A high performance web server and a reverse proxy server...
         Starting Network Name Resolution...
         Starting Hostname Service...
[  OK  ] Started Permit User Sessions.
[  OK  ] Started Getty on lxc/tty2.
[  OK  ] Started Container Getty on /dev/pts/3.
[  OK  ] Started Getty on lxc/tty3.
[  OK  ] Started Container Getty on /dev/pts/0.
[  OK  ] Started Container Getty on /dev/pts/5.
[  OK  ] Started Getty on lxc/tty4.
[  OK  ] Started Container Getty on /dev/pts/2.
[  OK  ] Started Getty on lxc/tty6.
[  OK  ] Started Getty on lxc/tty1.
[  OK  ] Started Container Getty on /dev/pts/1.
[  OK  ] Started Getty on lxc/tty5.
[  OK  ] Started Console Getty.
[  OK  ] Started Container Getty on /dev/pts/4.
[  OK  ] Reached target Login Prompts.
[  OK  ] Started A lightweight DHCP and caching DNS server.
[  OK  ] Started The PHP FastCGI Process Manager.
[  OK  ] Started Network Name Resolution.
[  OK  ] Started A high performance web server and a reverse proxy server.
[  OK  ] Reached target Multi-User System.
[  OK  ] Started Hostname Service.

Arch Linux 3.14.79-25-ARCH (console)

base-odroid64 login:

Number 2:

# lxc-start -n base-odroid64 -F
systemd 233 running in system mode. (+PAM -AUDIT -SELINUX -IMA -APPARMOR +SMACK -SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN default-hierarchy=hybrid)
Detected virtualization lxc.
Detected architecture arm64.

Welcome to Arch Linux ARM!

Set hostname to <base-odroid64>.
Cannot determine cgroup we are running in: No medium found
Failed to allocate manager object: No medium found
[!!!!!!] Failed to allocate manager object, freezing.
Freezing execution.

It's clear that the updated version ignores the directive but why?

systemd 232 running in system mode. (+PAM -AUDIT -SELINUX -IMA -APPARMOR +SMACK -SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN)

vs

systemd 233 running in system mode. (+PAM -AUDIT -SELINUX -IMA -APPARMOR +SMACK -SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN default-hierarchy=hybrid)

@brauner

This comment has been minimized.

Show comment
Hide comment
@brauner

brauner Jul 3, 2017

Member

Interesting, I've installed a new Archlinux container and performed an update and my fixes to the LXC codebase for the cgroup v2 handling at least work for me here. On the host, please do:

sudo mkdir -p /sys/fs/cgroup/unified
sudo mount -t cgroup2 none /sys/fs/cgroup/unified

and try to run the container again without lxc.init_cmd? Also have you rebooted your host after the systemd upgrade?

Member

brauner commented Jul 3, 2017

Interesting, I've installed a new Archlinux container and performed an update and my fixes to the LXC codebase for the cgroup v2 handling at least work for me here. On the host, please do:

sudo mkdir -p /sys/fs/cgroup/unified
sudo mount -t cgroup2 none /sys/fs/cgroup/unified

and try to run the container again without lxc.init_cmd? Also have you rebooted your host after the systemd upgrade?

@graysky2

This comment has been minimized.

Show comment
Hide comment
@graysky2

graysky2 Jul 3, 2017

Interesting, I've installed a new Archlinux container and performed an update and my fixes to the LXC codebase for the cgroup v2 handling at least work for me here.

/sys/fs/cgroup/unified was present on the filesystem by default but I seem unable to perform the mount command:

odroid-c2 # mount -t cgroup2 none /sys/fs/cgroup/unified
mount: unknown filesystem type 'cgroup2'

Note that the odroid-c2 is not a raspberry pi... the kernel version is it stuck on is rather old (3.14.79-25). I don't know if that is to blame for the failure to mount... or not. On my x86_64 box, (kernel 4.11.8), this is mounted by default:

x86_64 % mount | grep group2
cgroup on /sys/fs/cgroup/unified type cgroup2 (rw,nosuid,nodev,noexec,relatime)

Also have you rebooted your host after the systemd upgrade?

Yes, I rebooted it after the update.

New question: on the target box (the odroid-c2), I have lxc 2.0.8. Are the changes you mentioned in your reply rolled up in that version or do I need something from git?

graysky2 commented Jul 3, 2017

Interesting, I've installed a new Archlinux container and performed an update and my fixes to the LXC codebase for the cgroup v2 handling at least work for me here.

/sys/fs/cgroup/unified was present on the filesystem by default but I seem unable to perform the mount command:

odroid-c2 # mount -t cgroup2 none /sys/fs/cgroup/unified
mount: unknown filesystem type 'cgroup2'

Note that the odroid-c2 is not a raspberry pi... the kernel version is it stuck on is rather old (3.14.79-25). I don't know if that is to blame for the failure to mount... or not. On my x86_64 box, (kernel 4.11.8), this is mounted by default:

x86_64 % mount | grep group2
cgroup on /sys/fs/cgroup/unified type cgroup2 (rw,nosuid,nodev,noexec,relatime)

Also have you rebooted your host after the systemd upgrade?

Yes, I rebooted it after the update.

New question: on the target box (the odroid-c2), I have lxc 2.0.8. Are the changes you mentioned in your reply rolled up in that version or do I need something from git?

@brauner

This comment has been minimized.

Show comment
Hide comment
@brauner

brauner Jul 3, 2017

Member

Hahaha, that's the whole issue: there's no cgroup v2 in your kernel and apparently systemd is not able to boot in this case. This is very likely a systemd bug.

Member

brauner commented Jul 3, 2017

Hahaha, that's the whole issue: there's no cgroup v2 in your kernel and apparently systemd is not able to boot in this case. This is very likely a systemd bug.

@graysky2

This comment has been minimized.

Show comment
Hide comment
@graysky2

graysky2 Jul 3, 2017

Grrrr... thank you for the help. Is there anything I can do from an lxc perspective to work-around this?

graysky2 commented Jul 3, 2017

Grrrr... thank you for the help. Is there anything I can do from an lxc perspective to work-around this?

@brauner

This comment has been minimized.

Show comment
Hide comment
@brauner

brauner Jul 3, 2017

Member

Can you show systemctl --version on the host, please?

Member

brauner commented Jul 3, 2017

Can you show systemctl --version on the host, please?

@graysky2

This comment has been minimized.

Show comment
Hide comment
@graysky2

graysky2 Jul 3, 2017

Yes, on the odroid-c2:

# systemctl --version
systemd 233
+PAM -AUDIT -SELINUX -IMA -APPARMOR +SMACK -SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN default-hierarchy=hybrid

Perhaps compile systemd with --with-default-hierarchy=unified or --with-default-hierarchy=legacy?

graysky2 commented Jul 3, 2017

Yes, on the odroid-c2:

# systemctl --version
systemd 233
+PAM -AUDIT -SELINUX -IMA -APPARMOR +SMACK -SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN default-hierarchy=hybrid

Perhaps compile systemd with --with-default-hierarchy=unified or --with-default-hierarchy=legacy?

@brauner

This comment has been minimized.

Show comment
Hide comment
@brauner

brauner Jul 3, 2017

Member

@graysky2, can you try to delete the /sys/fs/cgroup/unified directory on the host and try again, please?

Member

brauner commented Jul 3, 2017

@graysky2, can you try to delete the /sys/fs/cgroup/unified directory on the host and try again, please?

@graysky2

This comment has been minimized.

Show comment
Hide comment
@graysky2

graysky2 Jul 3, 2017

Doesn't seem to be able to do it:

# rmdir /sys/fs/cgroup/unified
rmdir: failed to remove '/sys/fs/cgroup/unified': Read-only file system

graysky2 commented Jul 3, 2017

Doesn't seem to be able to do it:

# rmdir /sys/fs/cgroup/unified
rmdir: failed to remove '/sys/fs/cgroup/unified': Read-only file system
@brauner

This comment has been minimized.

Show comment
Hide comment
@brauner

brauner Jul 3, 2017

Member

Right,

mount -o remount,ro /sys/fs/cgroup
rm -r /sys/fs/cgroup/unified
mount -o remount,rw /sys/fs/cgroup
Member

brauner commented Jul 3, 2017

Right,

mount -o remount,ro /sys/fs/cgroup
rm -r /sys/fs/cgroup/unified
mount -o remount,rw /sys/fs/cgroup
@graysky2

This comment has been minimized.

Show comment
Hide comment
@graysky2

graysky2 Jul 3, 2017

Ah! OK...

# mount -o remount,rw /sys/fs/cgroup          
# rm -r /sys/fs/cgroup/unified
# mount -o remount,ro /sys/fs/cgroup

graysky2 commented Jul 3, 2017

Ah! OK...

# mount -o remount,rw /sys/fs/cgroup          
# rm -r /sys/fs/cgroup/unified
# mount -o remount,ro /sys/fs/cgroup
@brauner

This comment has been minimized.

Show comment
Hide comment
@brauner

brauner Jul 3, 2017

Member

Sorry, my bad. The other way around:

mount -o remount,rw /sys/fs/cgroup
rm -r /sys/fs/cgroup/unified
mount -o remount,ro /sys/fs/cgroup
Member

brauner commented Jul 3, 2017

Sorry, my bad. The other way around:

mount -o remount,rw /sys/fs/cgroup
rm -r /sys/fs/cgroup/unified
mount -o remount,ro /sys/fs/cgroup
@graysky2

This comment has been minimized.

Show comment
Hide comment
@graysky2

graysky2 Jul 3, 2017

Yes, I caught the other-way-around.... so I did that, started the container, updated it to systemd 233 and while updating:

(3/4) upgrading systemd                                                      [############################################] 100%

Broadcast message from systemd-journald@base-odroid64 (Mon 2017-07-03 09:46:11 EDT):

systemd[1]: Failed to allocate manager object: No medium found


Broadcast message from systemd-journald@base-odroid64 (Mon 2017-07-03 09:46:11 EDT):

systemd[1]: Freezing execution.

Unable to boot again.

graysky2 commented Jul 3, 2017

Yes, I caught the other-way-around.... so I did that, started the container, updated it to systemd 233 and while updating:

(3/4) upgrading systemd                                                      [############################################] 100%

Broadcast message from systemd-journald@base-odroid64 (Mon 2017-07-03 09:46:11 EDT):

systemd[1]: Failed to allocate manager object: No medium found


Broadcast message from systemd-journald@base-odroid64 (Mon 2017-07-03 09:46:11 EDT):

systemd[1]: Freezing execution.

Unable to boot again.

@brauner

This comment has been minimized.

Show comment
Hide comment
@brauner

brauner Jul 3, 2017

Member

Are both systems Archlinux (host and container)?

Member

brauner commented Jul 3, 2017

Are both systems Archlinux (host and container)?

@graysky2

This comment has been minimized.

Show comment
Hide comment
@graysky2

graysky2 Jul 3, 2017

Yes, both are Arch ARM (aarch64).

graysky2 commented Jul 3, 2017

Yes, both are Arch ARM (aarch64).

@graysky2

This comment has been minimized.

Show comment
Hide comment
@graysky2

graysky2 Jul 4, 2017

Thanks again for your help. The Odroid C2 is cool but its antiquated kernel makes it a pain in the ass to run. I just bought a RPi3 and will retired this hardware at least until Hardkernel has a modern kernel in their mainline support.

graysky2 commented Jul 4, 2017

Thanks again for your help. The Odroid C2 is cool but its antiquated kernel makes it a pain in the ass to run. I just bought a RPi3 and will retired this hardware at least until Hardkernel has a modern kernel in their mainline support.

@graysky2 graysky2 closed this Jul 4, 2017

@milouse

This comment has been minimized.

Show comment
Hide comment
@milouse

milouse Jul 4, 2017

Just for information, I run in the exact same problem today. Thanks to your long thread, I understood the problem was with the last systemd update. But as I've no backup (yes, I know, it's bad) I was sort of stuck outside my containers, until I try to arch-chroot in it, downgrade systemd package, and it boots.

I think I'll blacklist systemd package to avoid another crash.

milouse commented Jul 4, 2017

Just for information, I run in the exact same problem today. Thanks to your long thread, I understood the problem was with the last systemd update. But as I've no backup (yes, I know, it's bad) I was sort of stuck outside my containers, until I try to arch-chroot in it, downgrade systemd package, and it boots.

I think I'll blacklist systemd package to avoid another crash.

@graysky2

This comment has been minimized.

Show comment
Hide comment
@graysky2

graysky2 Jul 4, 2017

@milouse - Glad you found this of value. Odroid C2 for you as well?

graysky2 commented Jul 4, 2017

@milouse - Glad you found this of value. Odroid C2 for you as well?

@milouse

This comment has been minimized.

Show comment
Hide comment
@milouse

milouse Jul 4, 2017

Not at all, a regular server rent to OVH (a Kimsufi unit). We have no access to the kernel package, stuck in 3.14.

uname -a:

Linux ns3256470 3.14.32-xxxx-grs-ipv6-64 #7 SMP Wed Jan 27 18:05:09 CET 2016 x86_64 GNU/Linux

lxc-checkconfig:

--- Namespaces ---
Namespaces: enabled
Utsname namespace: enabled
Ipc namespace: enabled
Pid namespace: enabled
User namespace: enabled
Warning: newuidmap is not setuid-root
Warning: newgidmap is not setuid-root
Network namespace: enabled
Multiple /dev/pts instances: enabled

--- Control groups ---
Cgroup: enabled
Cgroup clone_children flag: enabled
Cgroup device: enabled
Cgroup sched: enabled
Cgroup cpu account: enabled
Cgroup memory controller: enabled
Cgroup cpuset: enabled

--- Misc ---
Veth pair device: enabled
Macvlan: enabled
Vlan: enabled
Bridges: enabled
Advanced netfilter: enabled
CONFIG_NF_NAT_IPV4: enabled
CONFIG_NF_NAT_IPV6: missing
CONFIG_IP_NF_TARGET_MASQUERADE: enabled
CONFIG_IP6_NF_TARGET_MASQUERADE: missing
CONFIG_NETFILTER_XT_TARGET_CHECKSUM: enabled
FUSE (for use with lxcfs): enabled

--- Checkpoint/Restore ---
checkpoint restore: missing
CONFIG_FHANDLE: enabled
CONFIG_EVENTFD: enabled
CONFIG_EPOLL: enabled
CONFIG_UNIX_DIAG: missing
CONFIG_INET_DIAG: enabled
CONFIG_PACKET_DIAG: enabled
CONFIG_NETLINK_DIAG: missing
File capabilities: enabled

Note : Before booting a new kernel, you can check its configuration
usage : CONFIG=/path/to/config /usr/bin/lxc-checkconfig

milouse commented Jul 4, 2017

Not at all, a regular server rent to OVH (a Kimsufi unit). We have no access to the kernel package, stuck in 3.14.

uname -a:

Linux ns3256470 3.14.32-xxxx-grs-ipv6-64 #7 SMP Wed Jan 27 18:05:09 CET 2016 x86_64 GNU/Linux

lxc-checkconfig:

--- Namespaces ---
Namespaces: enabled
Utsname namespace: enabled
Ipc namespace: enabled
Pid namespace: enabled
User namespace: enabled
Warning: newuidmap is not setuid-root
Warning: newgidmap is not setuid-root
Network namespace: enabled
Multiple /dev/pts instances: enabled

--- Control groups ---
Cgroup: enabled
Cgroup clone_children flag: enabled
Cgroup device: enabled
Cgroup sched: enabled
Cgroup cpu account: enabled
Cgroup memory controller: enabled
Cgroup cpuset: enabled

--- Misc ---
Veth pair device: enabled
Macvlan: enabled
Vlan: enabled
Bridges: enabled
Advanced netfilter: enabled
CONFIG_NF_NAT_IPV4: enabled
CONFIG_NF_NAT_IPV6: missing
CONFIG_IP_NF_TARGET_MASQUERADE: enabled
CONFIG_IP6_NF_TARGET_MASQUERADE: missing
CONFIG_NETFILTER_XT_TARGET_CHECKSUM: enabled
FUSE (for use with lxcfs): enabled

--- Checkpoint/Restore ---
checkpoint restore: missing
CONFIG_FHANDLE: enabled
CONFIG_EVENTFD: enabled
CONFIG_EPOLL: enabled
CONFIG_UNIX_DIAG: missing
CONFIG_INET_DIAG: enabled
CONFIG_PACKET_DIAG: enabled
CONFIG_NETLINK_DIAG: missing
File capabilities: enabled

Note : Before booting a new kernel, you can check its configuration
usage : CONFIG=/path/to/config /usr/bin/lxc-checkconfig
@graysky2

This comment has been minimized.

Show comment
Hide comment
@graysky2

graysky2 Jul 4, 2017

Lame! Glad you have a work-around but keeping packages stale (ie IgnorePkg = systemd libsystemd systemd-sysvcompat) is bad long-term and will eventually cause breakage.

graysky2 commented Jul 4, 2017

Lame! Glad you have a work-around but keeping packages stale (ie IgnorePkg = systemd libsystemd systemd-sysvcompat) is bad long-term and will eventually cause breakage.

@milouse

This comment has been minimized.

Show comment
Hide comment
@milouse

milouse Jul 5, 2017

Yeah, I know it. I have to take time to force-compile a recent-decent kernel, but didn't have it yet.

milouse commented Jul 5, 2017

Yeah, I know it. I have to take time to force-compile a recent-decent kernel, but didn't have it yet.

@graysky2

This comment has been minimized.

Show comment
Hide comment
@graysky2

graysky2 Jul 9, 2017

@brauner -

can you try to delete the /sys/fs/cgroup/unified directory on the host and try again, please?

I tried booting the ODROID-C2 appending the following to my kernel line params per the release notes, but found that the same error with containers persisted:
systemd.legacy_systemd_cgroup_controller=1 systemd.unified_cgroup_hierarchy=0

Note that upon booting into legacy mode, /sys/fs/cgroup/unified did not exist.

# dmesg [ 0.000000] Kernel command line: root=/dev/mmcblk0p1 rootwait rw console=ttyS0,115200n8 console=tty0 no_console_suspend hdmimode=1080p60hz m_bpp=32 vout=dvi fsck.repair=yes net.ifnames=0 elevator=noop disablehpd=true max_freq=1536 maxcpus=4 monitor_onoff=false disableuhs=false mmc_removable=true usbmulticam=false systemd.legacy_systemd_cgroup_controller=1 systemd.unified_cgroup_hierarchy=0 ...

graysky2 commented Jul 9, 2017

@brauner -

can you try to delete the /sys/fs/cgroup/unified directory on the host and try again, please?

I tried booting the ODROID-C2 appending the following to my kernel line params per the release notes, but found that the same error with containers persisted:
systemd.legacy_systemd_cgroup_controller=1 systemd.unified_cgroup_hierarchy=0

Note that upon booting into legacy mode, /sys/fs/cgroup/unified did not exist.

# dmesg [ 0.000000] Kernel command line: root=/dev/mmcblk0p1 rootwait rw console=ttyS0,115200n8 console=tty0 no_console_suspend hdmimode=1080p60hz m_bpp=32 vout=dvi fsck.repair=yes net.ifnames=0 elevator=noop disablehpd=true max_freq=1536 maxcpus=4 monitor_onoff=false disableuhs=false mmc_removable=true usbmulticam=false systemd.legacy_systemd_cgroup_controller=1 systemd.unified_cgroup_hierarchy=0 ...

@graysky2 graysky2 reopened this Jul 9, 2017

@brauner

This comment has been minimized.

Show comment
Hide comment
@brauner

brauner Jul 9, 2017

Member

@graysky, can you start a container again and show the -l strace output of it, please?

Member

brauner commented Jul 9, 2017

@graysky, can you start a container again and show the -l strace output of it, please?

@brauner

This comment has been minimized.

Show comment
Hide comment
@brauner

brauner Jul 9, 2017

Member

What kernel version is the ODROID-C2 on? I thought 3.14. That would cause the same problem.

Member

brauner commented Jul 9, 2017

What kernel version is the ODROID-C2 on? I thought 3.14. That would cause the same problem.

@graysky2

This comment has been minimized.

Show comment
Hide comment
@graysky2

graysky2 Jul 9, 2017

@brauner - Correct: the ODROID-C2 is currently limited to kernel version 3.14.79, but interestingly, using systemd v232 did not cause the problem. Only updating to v233 does the problem arise.

I am currently compiling systemd with the --with-default-hierarchy=legacy option for the ODROID-C2. The boot parameter should have disabled it but I wanted to try the compile-time option as well. Once that finishes, I will start the container with the -l strace option as you asked.

graysky2 commented Jul 9, 2017

@brauner - Correct: the ODROID-C2 is currently limited to kernel version 3.14.79, but interestingly, using systemd v232 did not cause the problem. Only updating to v233 does the problem arise.

I am currently compiling systemd with the --with-default-hierarchy=legacy option for the ODROID-C2. The boot parameter should have disabled it but I wanted to try the compile-time option as well. Once that finishes, I will start the container with the -l strace option as you asked.

@graysky2

This comment has been minimized.

Show comment
Hide comment
@graysky2

graysky2 Jul 9, 2017

OK, I installed systemd compiled with the --with-default-hierarchy=legacy option on both the host and within the lxc (starting from an older lxc with systemd v232 in it so it booted). I got the same problem when I started the container.

On the host: systemctl --version systemd 233 +PAM -AUDIT -SELINUX -IMA -APPARMOR +SMACK -SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN default-hierarchy=legacy

What is the command you want me execute? I didn't see an option -l strace for lxc-start so I am confused.

graysky2 commented Jul 9, 2017

OK, I installed systemd compiled with the --with-default-hierarchy=legacy option on both the host and within the lxc (starting from an older lxc with systemd v232 in it so it booted). I got the same problem when I started the container.

On the host: systemctl --version systemd 233 +PAM -AUDIT -SELINUX -IMA -APPARMOR +SMACK -SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN default-hierarchy=legacy

What is the command you want me execute? I didn't see an option -l strace for lxc-start so I am confused.

@brauner

This comment has been minimized.

Show comment
Hide comment
@brauner

brauner Jul 9, 2017

Member
lxc-start -n <container-name> -l trace -o <container-name>.log

and then append the full log. :)

Member

brauner commented Jul 9, 2017

lxc-start -n <container-name> -l trace -o <container-name>.log

and then append the full log. :)

@graysky2

This comment has been minimized.

Show comment
Hide comment
@graysky2

graysky2 Jul 9, 2017

You bet, attached.
base-odroid64.txt

Note that this is both the host and container using the modified systemd I compiled with the --with-default-hierarchy=legacy switch. I can revert to the Arch standard for both both and lxc if you wish.

graysky2 commented Jul 9, 2017

You bet, attached.
base-odroid64.txt

Note that this is both the host and container using the modified systemd I compiled with the --with-default-hierarchy=legacy switch. I can revert to the Arch standard for both both and lxc if you wish.

@brauner

This comment has been minimized.

Show comment
Hide comment
@brauner

brauner Jul 9, 2017

Member

Thanks!

  • Is there anything interesting in dmesg when you start the container?
  • What is the output to the terminal when you start the container in foreground mode?
Member

brauner commented Jul 9, 2017

Thanks!

  • Is there anything interesting in dmesg when you start the container?
  • What is the output to the terminal when you start the container in foreground mode?
@graysky2

This comment has been minimized.

Show comment
Hide comment
@graysky2

graysky2 Jul 9, 2017

First, here is everything in dmesg on the host after I issue lxc-start -n base-odroid64 -F

[   64.671159] device vethWH7CRV entered promiscuous mode
[   64.671695] IPv6: ADDRCONF(NETDEV_UP): vethWH7CRV: link is not ready
[   64.677875] br0: port 2(vethWH7CRV) entered forwarding state
[   64.682705] br0: port 2(vethWH7CRV) entered forwarding state
[   64.688385] br0: port 2(vethWH7CRV) entered disabled state
[   64.783612] lxc-start[418]: syscall 279
[   64.783722] Code: aa0503e4 aa0603e5 aa0703e6 d4000001 (b13ffc1f) 
[   64.788045] CPU: 0 PID: 418 Comm: lxc-start Not tainted 3.14.79-26-ARCH #1
[   64.794829] IPv6: ADDRCONF(NETDEV_CHANGE): vethWH7CRV: link becomes ready
[   64.801512] br0: port 2(vethWH7CRV) entered forwarding state
[   64.807135] br0: port 2(vethWH7CRV) entered forwarding state
[   64.811832] task: ffffffc05939cc80 ti: ffffffc058d14000 task.ti: ffffffc058d14000
[   64.811865] PC is at 0x7f9af73cf4
[   64.811867] LR is at 0x7f9b07fde0
[   64.811870] pc : [<0000007f9af73cf4>] lr : [<0000007f9b07fde0>] pstate: 80000000
[   64.811871] sp : 0000007feeac85f0
[   64.811877] x29: 0000007feeac85f0 x28: 0000000000437900 
[   64.811880] x27: 0000000000432e80 x26: 0000007feeac8788 
[   64.811884] x25: 0000007f9b0d5000 x24: 0000000000432ee0 
[   64.811887] x23: 0000000000432f08 x22: 0000000000000000 
[   64.811891] x21: 0000007feeac8708 x20: 0000000000434fb0 
[   64.811894] x19: 0000000000000004 x18: 000000000000072b 
[   64.811897] x17: 0000007f9b0cfa98 x16: 0000007f9af73cd0 
[   64.811900] x15: 0000007f9aeabcc8 x14: 0000007f9aeb8b08 
[   64.811903] x13: 0000000000000000 x12: 0000000000000020 
[   64.811907] x11: 0000000000000000 x10: 0101010101010101 
[   64.811910] x9 : 0000000000000000 x8 : 0000000000000117 
[   64.811913] x7 : 0000000000002130 x6 : 0000000000002130 
[   64.811916] x5 : 0000000000000064 x4 : 0000007f9affa500 
[   64.811919] x3 : 0000000000000000 x2 : 00000000ffffffff 
[   64.811922] x1 : 0000000000000001 x0 : 0000007f9b0b3800 

And here is the output of the start command in foreground mode:

# lxc-start -n base-odroid64 -F
systemd 233 running in system mode. (+PAM -AUDIT -SELINUX -IMA -APPARMOR +SMACK -SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN default-hierarchy=legacy)
Detected virtualization lxc.
Detected architecture arm64.

Welcome to Arch Linux ARM!

Set hostname to <base-odroid64>.
Cannot determine cgroup we are running in: No medium found
Failed to allocate manager object: No medium found
[!!!!!!] Failed to allocate manager object, freezing.
Freezing execution.

graysky2 commented Jul 9, 2017

First, here is everything in dmesg on the host after I issue lxc-start -n base-odroid64 -F

[   64.671159] device vethWH7CRV entered promiscuous mode
[   64.671695] IPv6: ADDRCONF(NETDEV_UP): vethWH7CRV: link is not ready
[   64.677875] br0: port 2(vethWH7CRV) entered forwarding state
[   64.682705] br0: port 2(vethWH7CRV) entered forwarding state
[   64.688385] br0: port 2(vethWH7CRV) entered disabled state
[   64.783612] lxc-start[418]: syscall 279
[   64.783722] Code: aa0503e4 aa0603e5 aa0703e6 d4000001 (b13ffc1f) 
[   64.788045] CPU: 0 PID: 418 Comm: lxc-start Not tainted 3.14.79-26-ARCH #1
[   64.794829] IPv6: ADDRCONF(NETDEV_CHANGE): vethWH7CRV: link becomes ready
[   64.801512] br0: port 2(vethWH7CRV) entered forwarding state
[   64.807135] br0: port 2(vethWH7CRV) entered forwarding state
[   64.811832] task: ffffffc05939cc80 ti: ffffffc058d14000 task.ti: ffffffc058d14000
[   64.811865] PC is at 0x7f9af73cf4
[   64.811867] LR is at 0x7f9b07fde0
[   64.811870] pc : [<0000007f9af73cf4>] lr : [<0000007f9b07fde0>] pstate: 80000000
[   64.811871] sp : 0000007feeac85f0
[   64.811877] x29: 0000007feeac85f0 x28: 0000000000437900 
[   64.811880] x27: 0000000000432e80 x26: 0000007feeac8788 
[   64.811884] x25: 0000007f9b0d5000 x24: 0000000000432ee0 
[   64.811887] x23: 0000000000432f08 x22: 0000000000000000 
[   64.811891] x21: 0000007feeac8708 x20: 0000000000434fb0 
[   64.811894] x19: 0000000000000004 x18: 000000000000072b 
[   64.811897] x17: 0000007f9b0cfa98 x16: 0000007f9af73cd0 
[   64.811900] x15: 0000007f9aeabcc8 x14: 0000007f9aeb8b08 
[   64.811903] x13: 0000000000000000 x12: 0000000000000020 
[   64.811907] x11: 0000000000000000 x10: 0101010101010101 
[   64.811910] x9 : 0000000000000000 x8 : 0000000000000117 
[   64.811913] x7 : 0000000000002130 x6 : 0000000000002130 
[   64.811916] x5 : 0000000000000064 x4 : 0000007f9affa500 
[   64.811919] x3 : 0000000000000000 x2 : 00000000ffffffff 
[   64.811922] x1 : 0000000000000001 x0 : 0000007f9b0b3800 

And here is the output of the start command in foreground mode:

# lxc-start -n base-odroid64 -F
systemd 233 running in system mode. (+PAM -AUDIT -SELINUX -IMA -APPARMOR +SMACK -SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN default-hierarchy=legacy)
Detected virtualization lxc.
Detected architecture arm64.

Welcome to Arch Linux ARM!

Set hostname to <base-odroid64>.
Cannot determine cgroup we are running in: No medium found
Failed to allocate manager object: No medium found
[!!!!!!] Failed to allocate manager object, freezing.
Freezing execution.
@brauner

This comment has been minimized.

Show comment
Hide comment
@brauner

brauner Jul 27, 2017

Member

I'll reopen this so we can track this easier.

Member

brauner commented Jul 27, 2017

I'll reopen this so we can track this easier.

@brauner brauner reopened this Jul 27, 2017

@graysky2

This comment has been minimized.

Show comment
Hide comment
@graysky2

graysky2 Jul 27, 2017

@evverx - I updated the ODROID-C2 kernel from the old series to the 4.12.x series to allow for lxc to run on it. In order to test this, I need to purchase a new micro SD and put the old version on it... I am confused though... doesn't #1713 fix this under the old kernel version?

@evverx - I updated the ODROID-C2 kernel from the old series to the 4.12.x series to allow for lxc to run on it. In order to test this, I need to purchase a new micro SD and put the old version on it... I am confused though... doesn't #1713 fix this under the old kernel version?

@brauner

This comment has been minimized.

Show comment
Hide comment
@brauner

brauner Jul 27, 2017

Member

@graysky2, not likely. @evverx and I suspect that this might be caused by systemd in a container not handling kernels that do not implement cgroup v2 correctly. So this would be a separate systemd patch we're talking about.

Member

brauner commented Jul 27, 2017

@graysky2, not likely. @evverx and I suspect that this might be caused by systemd in a container not handling kernels that do not implement cgroup v2 correctly. So this would be a separate systemd patch we're talking about.

@graysky2

This comment has been minimized.

Show comment
Hide comment
@graysky2

graysky2 Jul 27, 2017

OK. I just need to get a card then I can test this and report the output for you guys.

OK. I just need to get a card then I can test this and report the output for you guys.

@brauner

This comment has been minimized.

Show comment
Hide comment
@brauner

brauner Jul 27, 2017

Member

@graysky2, I really appreciate you taking the time to debug this with us but please don't start spending money just to debug this. :) We'll figure this one out even if you don't buy this card @evverx is helping out as well here so we've got combined systemd + lxc knowledge going. :)

Member

brauner commented Jul 27, 2017

@graysky2, I really appreciate you taking the time to debug this with us but please don't start spending money just to debug this. :) We'll figure this one out even if you don't buy this card @evverx is helping out as well here so we've got combined systemd + lxc knowledge going. :)

@graysky2

This comment has been minimized.

Show comment
Hide comment
@graysky2

graysky2 Jul 27, 2017

@graysky2, I really appreciate you taking the time to debug this with us but please don't start spending money just to debug this. :) We'll figure this one out even if you don't buy this card @evverx is helping out as well here so we've got combined systemd + lxc knowledge going here. :)

No worries. I enjoy collaborating with you guys to make lxc as robust as it can be... THANK you both for the time you've invested here!

OK... I found an older card and am copying over the image now. Below is the config I plan to you for the tests. Does it contain anything odd or will it be fine:

lxc.rootfs = /var/lib/lxc/obase/rootfs
lxc.rootfs.backend = dir
lxc.utsname = obase
lxc.arch = aarch64
lxc.include = /usr/share/lxc/config/archlinux.common.conf

## network
lxc.network.type = veth
lxc.network.flags = up
lxc.network.link = br0
lxc.network.name = eth0
lxc.network.hwaddr = 00:1e:06:33:59:e6

## systemd within the lxc
lxc.autodev = 1
lxc.hook.autodev = /var/lib/lxc/obase/autodev
lxc.pts = 1024
lxc.kmsg = 0

## mounts
lxc.mount.entry = /var/cache/pacman/pkg var/cache/pacman/pkg none bind 0 0

graysky2 commented Jul 27, 2017

@graysky2, I really appreciate you taking the time to debug this with us but please don't start spending money just to debug this. :) We'll figure this one out even if you don't buy this card @evverx is helping out as well here so we've got combined systemd + lxc knowledge going here. :)

No worries. I enjoy collaborating with you guys to make lxc as robust as it can be... THANK you both for the time you've invested here!

OK... I found an older card and am copying over the image now. Below is the config I plan to you for the tests. Does it contain anything odd or will it be fine:

lxc.rootfs = /var/lib/lxc/obase/rootfs
lxc.rootfs.backend = dir
lxc.utsname = obase
lxc.arch = aarch64
lxc.include = /usr/share/lxc/config/archlinux.common.conf

## network
lxc.network.type = veth
lxc.network.flags = up
lxc.network.link = br0
lxc.network.name = eth0
lxc.network.hwaddr = 00:1e:06:33:59:e6

## systemd within the lxc
lxc.autodev = 1
lxc.hook.autodev = /var/lib/lxc/obase/autodev
lxc.pts = 1024
lxc.kmsg = 0

## mounts
lxc.mount.entry = /var/cache/pacman/pkg var/cache/pacman/pkg none bind 0 0
@brauner

This comment has been minimized.

Show comment
Hide comment
@brauner

brauner Jul 27, 2017

Member

Nope, seems fine. :)

Member

brauner commented Jul 27, 2017

Nope, seems fine. :)

@graysky2

This comment has been minimized.

Show comment
Hide comment
@graysky2

graysky2 Jul 27, 2017

OK, under 3.14.79-27-ARCH (systemd on the host is 234.11... there is an older version in the container due to it not being up-to-date since it cannot start):

# lxc-start -n obase -- /usr/bin/strace -D -s500 -v -yy -o OUT /sbin/init
lxc-start: tools/lxc_start.c: main: 366 The container failed to start.
lxc-start: tools/lxc_start.c: main: 368 To get more details, run the container in foreground mode.
lxc-start: tools/lxc_start.c: main: 370 Additional information can be obtained by setting the --logfile and --logpriority option

# lxc-attach -n obase -- cat /OUT
lxc-attach: attach.c: lxc_attach: 742 Failed to get init pid.

graysky2 commented Jul 27, 2017

OK, under 3.14.79-27-ARCH (systemd on the host is 234.11... there is an older version in the container due to it not being up-to-date since it cannot start):

# lxc-start -n obase -- /usr/bin/strace -D -s500 -v -yy -o OUT /sbin/init
lxc-start: tools/lxc_start.c: main: 366 The container failed to start.
lxc-start: tools/lxc_start.c: main: 368 To get more details, run the container in foreground mode.
lxc-start: tools/lxc_start.c: main: 370 Additional information can be obtained by setting the --logfile and --logpriority option

# lxc-attach -n obase -- cat /OUT
lxc-attach: attach.c: lxc_attach: 742 Failed to get init pid.
@evverx

This comment has been minimized.

Show comment
Hide comment
@evverx

evverx Jul 27, 2017

It seems that strace has not been installed in the container.

evverx commented Jul 27, 2017

It seems that strace has not been installed in the container.

@graysky2

This comment has been minimized.

Show comment
Hide comment
@graysky2

graysky2 Jul 27, 2017

I will downgrade the host, start the container, update the container and install strace therein, then update the host again, reboot and repeat.

I will downgrade the host, start the container, update the container and install strace therein, then update the host again, reboot and repeat.

@graysky2

This comment has been minimized.

Show comment
Hide comment
@graysky2

graysky2 Jul 27, 2017

OK... with systemd-232-8 in the container and with systemd-234.11 on the host:
# lxc-start -n obase -- /usr/bin/strace -D -s500 -v -yy -o OUT /sbin/init returned no output, but the next command lxc-attach -n obase -- cat /OUT gave >10,000 lines which I linked here.

graysky2 commented Jul 27, 2017

OK... with systemd-232-8 in the container and with systemd-234.11 on the host:
# lxc-start -n obase -- /usr/bin/strace -D -s500 -v -yy -o OUT /sbin/init returned no output, but the next command lxc-attach -n obase -- cat /OUT gave >10,000 lines which I linked here.

@evverx

This comment has been minimized.

Show comment
Hide comment
@evverx

evverx Jul 27, 2017

Thank you. systemd-232-8 was not "frozen". Could you update systemd in the container and run the same commands?

evverx commented Jul 27, 2017

Thank you. systemd-232-8 was not "frozen". Could you update systemd in the container and run the same commands?

@brauner

This comment has been minimized.

Show comment
Hide comment
@brauner

brauner Jul 27, 2017

Member

Yeah, systems 232 does not try to mount any cgroup2 fs.

Member

brauner commented Jul 27, 2017

Yeah, systems 232 does not try to mount any cgroup2 fs.

@evverx

This comment has been minimized.

Show comment
Hide comment
@evverx

evverx Jul 27, 2017

I've just reread https://gist.github.com/graysky2/e5fcedf54df62ad7e6abb0d2bbf1345d and I think I found a way to hang systemd with No medium found:

bash-4.3# mount -t sysfs sysfs /sys
bash-4.3# mount -t proc proc /proc/
bash-4.3# mount -t tmpfs tmpfs /sys/fs/cgroup/
bash-4.3# mkdir /sys/fs/cgroup/systemd
bash-4.3# mount -t tmpfs tmpfs /sys/fs/cgroup/systemd/
bash-4.3# exec /usr/lib/systemd/systemd --unit=rescue.target

Welcome to Ubuntu 16.04!

[   60.275683] systemd[1]: Failed to allocate manager object: No medium found
[!!!!!!] Failed to allocate manager object, freezing.
[   60.289669] systemd[1]: Freezing execution.

The same commands work fine on Fedora 25, which supports cgroup2.

I'm not sure I understand what premounts /sys/fs/cgroup/systemd. systemd doesn't do it:

$ grep '^mount' gistfile1.txt
mount("tmpfs", "/dev/shm", "tmpfs", MS_NOSUID|MS_NODEV|MS_STRICTATIME, "mode=1777") = 0
mount("tmpfs", "/run", "tmpfs", MS_NOSUID|MS_NODEV|MS_STRICTATIME, "mode=755") = 0
mount("cgroup", "/sys/fs/cgroup/cpu,cpuacct", "cgroup", MS_NOSUID|MS_NODEV|MS_NOEXEC, "cpu,cpuacct") = 0
mount("tmpfs", "/sys/fs/cgroup", 0x55676fab58, MS_RDONLY|MS_NOSUID|MS_NODEV|MS_NOEXEC|MS_REMOUNT|MS_STRICTATIME, "mode=755") = 0

@brauner, do you know if lxc premounts /sys/fs/cgroup/systemd?

@graysky2, I think the root cause has been found, but it would be great to see what strace prints for systemd-233-* to be sure.

evverx commented Jul 27, 2017

I've just reread https://gist.github.com/graysky2/e5fcedf54df62ad7e6abb0d2bbf1345d and I think I found a way to hang systemd with No medium found:

bash-4.3# mount -t sysfs sysfs /sys
bash-4.3# mount -t proc proc /proc/
bash-4.3# mount -t tmpfs tmpfs /sys/fs/cgroup/
bash-4.3# mkdir /sys/fs/cgroup/systemd
bash-4.3# mount -t tmpfs tmpfs /sys/fs/cgroup/systemd/
bash-4.3# exec /usr/lib/systemd/systemd --unit=rescue.target

Welcome to Ubuntu 16.04!

[   60.275683] systemd[1]: Failed to allocate manager object: No medium found
[!!!!!!] Failed to allocate manager object, freezing.
[   60.289669] systemd[1]: Freezing execution.

The same commands work fine on Fedora 25, which supports cgroup2.

I'm not sure I understand what premounts /sys/fs/cgroup/systemd. systemd doesn't do it:

$ grep '^mount' gistfile1.txt
mount("tmpfs", "/dev/shm", "tmpfs", MS_NOSUID|MS_NODEV|MS_STRICTATIME, "mode=1777") = 0
mount("tmpfs", "/run", "tmpfs", MS_NOSUID|MS_NODEV|MS_STRICTATIME, "mode=755") = 0
mount("cgroup", "/sys/fs/cgroup/cpu,cpuacct", "cgroup", MS_NOSUID|MS_NODEV|MS_NOEXEC, "cpu,cpuacct") = 0
mount("tmpfs", "/sys/fs/cgroup", 0x55676fab58, MS_RDONLY|MS_NOSUID|MS_NODEV|MS_NOEXEC|MS_REMOUNT|MS_STRICTATIME, "mode=755") = 0

@brauner, do you know if lxc premounts /sys/fs/cgroup/systemd?

@graysky2, I think the root cause has been found, but it would be great to see what strace prints for systemd-233-* to be sure.

@brauner

This comment has been minimized.

Show comment
Hide comment
@brauner

brauner Jul 27, 2017

Member

@evverx, I may have a suspicion. @graysky2, I suspect your kernel doesn't support CLONE_NEWCGROUP aka cgroup namespaces? So, when no cgroup namespaces are enabled then LXC will pre-mount the cgroup controllers. Users have a variety of options so that they can make the cgroup tree read-writeable or just read-only or a mixture of this. This can be fine-tuned. But how would this affect systemd booting correctly? Especially the difference between 232 and 233?

Member

brauner commented Jul 27, 2017

@evverx, I may have a suspicion. @graysky2, I suspect your kernel doesn't support CLONE_NEWCGROUP aka cgroup namespaces? So, when no cgroup namespaces are enabled then LXC will pre-mount the cgroup controllers. Users have a variety of options so that they can make the cgroup tree read-writeable or just read-only or a mixture of this. This can be fine-tuned. But how would this affect systemd booting correctly? Especially the difference between 232 and 233?

@evverx

This comment has been minimized.

Show comment
Hide comment
@evverx

evverx Jul 27, 2017

systemd/systemd@f08e928#diff-0d48d4d5663167a0bb6786a4abaddc35R2322 was added to prevent systemd from causing the kernel panic during reexecuting from v232 to v233. And that code assumes that /sys/fs/cgroup/systemd is either cgroup or cgroup2. Unfortunately, nobody seems to have assumed that /sys/fs/cgroup/systemd might be premounted as tmpfs. That looks like a regression to me.

evverx commented Jul 27, 2017

systemd/systemd@f08e928#diff-0d48d4d5663167a0bb6786a4abaddc35R2322 was added to prevent systemd from causing the kernel panic during reexecuting from v232 to v233. And that code assumes that /sys/fs/cgroup/systemd is either cgroup or cgroup2. Unfortunately, nobody seems to have assumed that /sys/fs/cgroup/systemd might be premounted as tmpfs. That looks like a regression to me.

@evverx

This comment has been minimized.

Show comment
Hide comment
@evverx

evverx Jul 27, 2017

I'm sorry, It seems that I gave the wrong link. systemd/systemd@2977724#diff-0d48d4d5663167a0bb6786a4abaddc35L2257 seems to be the right one.

evverx commented Jul 27, 2017

I'm sorry, It seems that I gave the wrong link. systemd/systemd@2977724#diff-0d48d4d5663167a0bb6786a4abaddc35L2257 seems to be the right one.

@evverx

This comment has been minimized.

Show comment
Hide comment
@evverx

evverx Jul 28, 2017

But that, of course, doesn't explain why systemd doesn't fail to boot when cgroup2 is supported and /sys/fs/cgroup/systemd is mounted as tmpfs. I'm a bit puzzled.

evverx commented Jul 28, 2017

But that, of course, doesn't explain why systemd doesn't fail to boot when cgroup2 is supported and /sys/fs/cgroup/systemd is mounted as tmpfs. I'm a bit puzzled.

@brauner

This comment has been minimized.

Show comment
Hide comment
@brauner

brauner Jul 28, 2017

Member

@evverx, isn't systemd using /sys/fs/cgroup/unified to track processes and not /sys/fs/cgroup/systemd when cgroup2 and hybrid mode is supported?

Member

brauner commented Jul 28, 2017

@evverx, isn't systemd using /sys/fs/cgroup/unified to track processes and not /sys/fs/cgroup/systemd when cgroup2 and hybrid mode is supported?

@evverx

This comment has been minimized.

Show comment
Hide comment
@evverx

evverx Jul 28, 2017

Yes, you are right. I should have drunk more coffee after reading the output of strace :-)

evverx commented Jul 28, 2017

Yes, you are right. I should have drunk more coffee after reading the output of strace :-)

@brauner

This comment has been minimized.

Show comment
Hide comment
@brauner

brauner Jul 28, 2017

Member

@evverx, ah, totally fine without your help this would've taken way longer. :)

Member

brauner commented Jul 28, 2017

@evverx, ah, totally fine without your help this would've taken way longer. :)

@evverx

This comment has been minimized.

Show comment
Hide comment
@evverx

evverx Jul 28, 2017

I've just compiled systemd-232 and it turned out that the commands I had mentioned hung systemd-v232 as well:

bash-4.3# /lib/systemd/systemd --version
systemd 232
+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ -LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN
bash-4.3# mount -t proc proc /proc/
bash-4.3# mount -t sysfs sysfs /sys/
bash-4.3# mount -t tmpfs tmpfs /sys/fs/cgroup/
bash-4.3# mkdir /sys/fs/cgroup/systemd
bash-4.3# mount -t tmpfs tmpfs /sys/fs/cgroup/systemd
bash-4.3# exec /lib/systemd/systemd --unit=rescue.target

Welcome to Ubuntu 16.04!

[  112.822083] systemd[1]: Failed to allocate manager object: No data available
[!!!!!!] Failed to allocate manager object, freezing.
[  112.852054] systemd[1]: Freezing execution.

I think it would be better to wait for @graysky2's reply.

evverx commented Jul 28, 2017

I've just compiled systemd-232 and it turned out that the commands I had mentioned hung systemd-v232 as well:

bash-4.3# /lib/systemd/systemd --version
systemd 232
+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ -LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN
bash-4.3# mount -t proc proc /proc/
bash-4.3# mount -t sysfs sysfs /sys/
bash-4.3# mount -t tmpfs tmpfs /sys/fs/cgroup/
bash-4.3# mkdir /sys/fs/cgroup/systemd
bash-4.3# mount -t tmpfs tmpfs /sys/fs/cgroup/systemd
bash-4.3# exec /lib/systemd/systemd --unit=rescue.target

Welcome to Ubuntu 16.04!

[  112.822083] systemd[1]: Failed to allocate manager object: No data available
[!!!!!!] Failed to allocate manager object, freezing.
[  112.852054] systemd[1]: Freezing execution.

I think it would be better to wait for @graysky2's reply.

@graysky2

This comment has been minimized.

Show comment
Hide comment
@graysky2

graysky2 Jul 28, 2017

I am happy to try whatever. You want me to update the lxc to the current version of systemd and repeat the two above commands?

I am happy to try whatever. You want me to update the lxc to the current version of systemd and repeat the two above commands?

@evverx

This comment has been minimized.

Show comment
Hide comment
@evverx

evverx Jul 28, 2017

@graysky2 , yes, it would be great if you could upgrade systemd to a version >= 233 in the container and run the two commands above.

evverx commented Jul 28, 2017

@graysky2 , yes, it would be great if you could upgrade systemd to a version >= 233 in the container and run the two commands above.

@graysky2

This comment has been minimized.

Show comment
Hide comment
@graysky2

graysky2 Jul 28, 2017

So in the past, when I update systemd in the container it triggers the bug while the updated packages are deployed if I remember correctly. I will downgrade the host and try. One sec.

EDIT: yes:

<<< from within the container >>>
# pacman -Syyu
:: Synchronizing package databases...
 core is up to date
 extra is up to date
 community is up to date
 alarm is up to date
 aur is up to date
:: Starting full system upgrade...
resolving dependencies...
looking for conflicting packages...

Packages (3) libsystemd-234.11-1  systemd-234.11-1  systemd-sysvcompat-234.11-1

Total Installed Size:  22.95 MiB
Net Upgrade Size:       4.52 MiB

:: Proceed with installation? [Y/n] 
(3/3) checking keys in keyring                                               [############################################] 100%
(3/3) checking package integrity                                             [############################################] 100%
(3/3) loading package files                                                  [############################################] 100%
(3/3) checking for file conflicts                                            [############################################] 100%
:: Processing package changes...
(1/3) upgrading libsystemd                                                   [############################################] 100%
(2/3) upgrading systemd                                                      [############################################] 100%

Broadcast message from systemd-journald@base (Thu 2017-07-27 21:44:01 EDT):

systemd[1]: Failed to allocate manager object: No medium found


Broadcast message from systemd-journald@base (Thu 2017-07-27 21:44:01 EDT):

systemd[1]: Freezing execution.

<<< pause for approx 30 sec >>>

Failed to get unit file state for systemd-resolved.service: Failed to activate service 'org.freedesktop.systemd1': timed out
(3/3) upgrading systemd-sysvcompat                                           [############################################] 100%
:: Running post-transaction hooks...
(1/4) Updating udev hardware database...
(2/4) Updating system user accounts...
(3/4) Creating temporary files...
(4/4) Arming ConditionNeedsUpdate...

Now the container is updated as you see. I next updated the host then ran the experiment you asked for. Note that I had to kill -9 the pid of lxc-start in order to stop the container first. For good measure the following was run after a reboot (again both host and container are running systemd 234.11-1:
# lxc-start -n obase -- /usr/bin/strace -D -s500 -v -yy -o OUT /sbin/init gave no output but lxc-attach -n obase -- cat /OUT gave approx. 1,000 lines of info.

graysky2 commented Jul 28, 2017

So in the past, when I update systemd in the container it triggers the bug while the updated packages are deployed if I remember correctly. I will downgrade the host and try. One sec.

EDIT: yes:

<<< from within the container >>>
# pacman -Syyu
:: Synchronizing package databases...
 core is up to date
 extra is up to date
 community is up to date
 alarm is up to date
 aur is up to date
:: Starting full system upgrade...
resolving dependencies...
looking for conflicting packages...

Packages (3) libsystemd-234.11-1  systemd-234.11-1  systemd-sysvcompat-234.11-1

Total Installed Size:  22.95 MiB
Net Upgrade Size:       4.52 MiB

:: Proceed with installation? [Y/n] 
(3/3) checking keys in keyring                                               [############################################] 100%
(3/3) checking package integrity                                             [############################################] 100%
(3/3) loading package files                                                  [############################################] 100%
(3/3) checking for file conflicts                                            [############################################] 100%
:: Processing package changes...
(1/3) upgrading libsystemd                                                   [############################################] 100%
(2/3) upgrading systemd                                                      [############################################] 100%

Broadcast message from systemd-journald@base (Thu 2017-07-27 21:44:01 EDT):

systemd[1]: Failed to allocate manager object: No medium found


Broadcast message from systemd-journald@base (Thu 2017-07-27 21:44:01 EDT):

systemd[1]: Freezing execution.

<<< pause for approx 30 sec >>>

Failed to get unit file state for systemd-resolved.service: Failed to activate service 'org.freedesktop.systemd1': timed out
(3/3) upgrading systemd-sysvcompat                                           [############################################] 100%
:: Running post-transaction hooks...
(1/4) Updating udev hardware database...
(2/4) Updating system user accounts...
(3/4) Creating temporary files...
(4/4) Arming ConditionNeedsUpdate...

Now the container is updated as you see. I next updated the host then ran the experiment you asked for. Note that I had to kill -9 the pid of lxc-start in order to stop the container first. For good measure the following was run after a reboot (again both host and container are running systemd 234.11-1:
# lxc-start -n obase -- /usr/bin/strace -D -s500 -v -yy -o OUT /sbin/init gave no output but lxc-attach -n obase -- cat /OUT gave approx. 1,000 lines of info.

@graysky2

This comment has been minimized.

Show comment
Hide comment
@graysky2

graysky2 Jul 28, 2017

I suspect your kernel doesn't support CLONE_NEWCGROUP aka cgroup namespaces?

@brauner

% uname -a
Linux odroid64 3.14.79-27-ARCH #1 SMP PREEMPT Mon Jul 10 19:29:37 MDT 2017 aarch64 GNU/Linux

% zgrep -i clone /proc/config.gz 
CONFIG_CLONE_BACKWARDS=y

Doesn't seem to be in there.... here is the kernel config used when Arch ARM compiles the linux-odroid-c2 kernel package.

EDIT: To be fair, the official Arch kernel (x86_64) does not either.

graysky2 commented Jul 28, 2017

I suspect your kernel doesn't support CLONE_NEWCGROUP aka cgroup namespaces?

@brauner

% uname -a
Linux odroid64 3.14.79-27-ARCH #1 SMP PREEMPT Mon Jul 10 19:29:37 MDT 2017 aarch64 GNU/Linux

% zgrep -i clone /proc/config.gz 
CONFIG_CLONE_BACKWARDS=y

Doesn't seem to be in there.... here is the kernel config used when Arch ARM compiles the linux-odroid-c2 kernel package.

EDIT: To be fair, the official Arch kernel (x86_64) does not either.

@brauner

This comment has been minimized.

Show comment
Hide comment
@brauner

brauner Jul 28, 2017

Member

Ok, I know that ArchLinux has a strong stance on the whole user namespace thing, fine. But cgroup namespaces? Really?

Member

brauner commented Jul 28, 2017

Ok, I know that ArchLinux has a strong stance on the whole user namespace thing, fine. But cgroup namespaces? Really?

@evverx

This comment has been minimized.

Show comment
Hide comment
@evverx

evverx Jul 28, 2017

@graysky2 , thank you. Could you also attach the output of the following command:

lxc-start -F -n <container> -- /bin/bash -c 'cat /proc/self/cgroup /proc/self/mountinfo'

?

evverx commented Jul 28, 2017

@graysky2 , thank you. Could you also attach the output of the following command:

lxc-start -F -n <container> -- /bin/bash -c 'cat /proc/self/cgroup /proc/self/mountinfo'

?

@evverx

This comment has been minimized.

Show comment
Hide comment
@evverx

evverx Jul 28, 2017

I extracted /proc/self/cgroup and /proc/self/mountinfo from https://gist.github.com/graysky2/e5fcedf54df62ad7e6abb0d2bbf1345d. Here is another way to hang systemd-233 that doesn't hang systemd-232:

bash-4.3# mount -t sysfs sysfs /sys
bash-4.3# mount -t proc proc /proc/
bash-4.3# mount -t tmpfs tmpfs /sys/fs/cgroup/
bash-4.3# mkdir /sys/fs/cgroup/systemd
bash-4.3# mount -t tmpfs tmpfs /sys/fs/cgroup/systemd
bash-4.3# mkdir /sys/fs/cgroup/systemd/lxc
bash-4.3# mount -o remount,rw /
bash-4.3# mkdir /SUB
bash-4.3# mount -t cgroup cgroup -o none,name=systemd /SUB
bash-4.3# mkdir /SUB/lxc
bash-4.3# echo 1 >/SUB/lxc/cgroup.procs
bash-4.3# mount --bind /SUB/lxc /sys/fs/cgroup/systemd/lxc
bash-4.3# cat /proc/self/cgroup
1:name=systemd:/lxc
bash-4.3# exec /usr/lib/systemd/system

Welcome to Ubuntu 16.04!

[  161.003113] systemd[1]: Failed to allocate manager object: No medium found
[!!!!!!] Failed to allocate manager object, freezing.
[  161.006041] systemd[1]: Freezing execution.

It seems that the code in systemd/systemd@2977724#diff-0d48d4d5663167a0bb6786a4abaddc35L2257 should be changed to deal with /sys/fs/cgroup/systemd + /lxc instead of /sys/fs/cgroup/systemd.

evverx commented Jul 28, 2017

I extracted /proc/self/cgroup and /proc/self/mountinfo from https://gist.github.com/graysky2/e5fcedf54df62ad7e6abb0d2bbf1345d. Here is another way to hang systemd-233 that doesn't hang systemd-232:

bash-4.3# mount -t sysfs sysfs /sys
bash-4.3# mount -t proc proc /proc/
bash-4.3# mount -t tmpfs tmpfs /sys/fs/cgroup/
bash-4.3# mkdir /sys/fs/cgroup/systemd
bash-4.3# mount -t tmpfs tmpfs /sys/fs/cgroup/systemd
bash-4.3# mkdir /sys/fs/cgroup/systemd/lxc
bash-4.3# mount -o remount,rw /
bash-4.3# mkdir /SUB
bash-4.3# mount -t cgroup cgroup -o none,name=systemd /SUB
bash-4.3# mkdir /SUB/lxc
bash-4.3# echo 1 >/SUB/lxc/cgroup.procs
bash-4.3# mount --bind /SUB/lxc /sys/fs/cgroup/systemd/lxc
bash-4.3# cat /proc/self/cgroup
1:name=systemd:/lxc
bash-4.3# exec /usr/lib/systemd/system

Welcome to Ubuntu 16.04!

[  161.003113] systemd[1]: Failed to allocate manager object: No medium found
[!!!!!!] Failed to allocate manager object, freezing.
[  161.006041] systemd[1]: Freezing execution.

It seems that the code in systemd/systemd@2977724#diff-0d48d4d5663167a0bb6786a4abaddc35L2257 should be changed to deal with /sys/fs/cgroup/systemd + /lxc instead of /sys/fs/cgroup/systemd.

@graysky2

This comment has been minimized.

Show comment
Hide comment
@graysky2

graysky2 Jul 28, 2017

@evverx:

# lxc-start -F -n obase -- /bin/bash -c 'cat /proc/self/cgroup /proc/self/mountinfo'           
12:memory:/lxc/obase
11:debug:/lxc/obase
10:net_cls:/lxc/obase
9:devices:/lxc/obase
8:blkio:/lxc/obase
7:hugetlb:/lxc/obase
6:perf_event:/lxc/obase
5:cpu,cpuacct:/lxc/obase
4:freezer:/lxc/obase
3:cpuset:/lxc/obase
2:name=systemd:/lxc/obase
188 189 179:1 /var/lib/lxc/obase/rootfs / rw,relatime master:1 - ext4 /dev/mmcblk0p1 rw,data=ordered
219 188 0:37 / /dev rw,relatime - tmpfs none rw,size=492k,mode=755
220 188 0:39 / /proc rw,nosuid,nodev,noexec,relatime - proc proc rw
221 222 0:39 /sys/net /proc/sys/net rw,nosuid,nodev,noexec,relatime - proc proc rw
222 220 0:39 /sys /proc/sys ro,nosuid,nodev,noexec,relatime - proc proc rw
223 220 0:39 /sysrq-trigger /proc/sysrq-trigger ro,nosuid,nodev,noexec,relatime - proc proc rw
224 188 0:40 / /sys rw,nosuid,nodev,noexec,relatime - sysfs sysfs rw
225 224 0:40 / /sys ro,nosuid,nodev,noexec,relatime - sysfs sysfs rw
226 225 0:40 / /sys/devices/virtual/net rw,relatime - sysfs sysfs rw
227 226 0:40 /devices/virtual/net /sys/devices/virtual/net rw,nosuid,nodev,noexec,relatime - sysfs sysfs rw
228 188 179:1 /var/cache/pacman/pkg /var/cache/pacman/pkg rw,relatime master:1 - ext4 /dev/mmcblk0p1 rw,data=ordered
229 225 0:41 / /sys/fs/cgroup rw,nosuid,nodev,noexec,relatime - tmpfs cgroup_root rw,size=10240k,mode=755
230 229 0:41 /systemd /sys/fs/cgroup/systemd ro,relatime - tmpfs cgroup_root rw,size=10240k,mode=755
231 230 0:19 /lxc/obase /sys/fs/cgroup/systemd/lxc/obase rw,nosuid,nodev,noexec,relatime master:9 - cgroup cgroup rw,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd
232 229 0:41 /cpuset /sys/fs/cgroup/cpuset ro,relatime - tmpfs cgroup_root rw,size=10240k,mode=755
233 232 0:20 /lxc/obase /sys/fs/cgroup/cpuset/lxc/obase rw,nosuid,nodev,noexec,relatime master:11 - cgroup cgroup rw,cpuset
234 229 0:41 /freezer /sys/fs/cgroup/freezer ro,relatime - tmpfs cgroup_root rw,size=10240k,mode=755
235 234 0:21 /lxc/obase /sys/fs/cgroup/freezer/lxc/obase rw,nosuid,nodev,noexec,relatime master:12 - cgroup cgroup rw,freezer
236 229 0:41 /cpu /sys/fs/cgroup/cpu ro,relatime - tmpfs cgroup_root rw,size=10240k,mode=755
237 236 0:22 /lxc/obase /sys/fs/cgroup/cpu/lxc/obase rw,nosuid,nodev,noexec,relatime master:13 - cgroup cgroup rw,cpu,cpuacct
238 229 0:41 /perf_event /sys/fs/cgroup/perf_event ro,relatime - tmpfs cgroup_root rw,size=10240k,mode=755
239 238 0:23 /lxc/obase /sys/fs/cgroup/perf_event/lxc/obase rw,nosuid,nodev,noexec,relatime master:14 - cgroup cgroup rw,perf_event
240 229 0:41 /hugetlb /sys/fs/cgroup/hugetlb ro,relatime - tmpfs cgroup_root rw,size=10240k,mode=755
241 240 0:24 /lxc/obase /sys/fs/cgroup/hugetlb/lxc/obase rw,nosuid,nodev,noexec,relatime master:15 - cgroup cgroup rw,hugetlb
242 229 0:41 /blkio /sys/fs/cgroup/blkio ro,relatime - tmpfs cgroup_root rw,size=10240k,mode=755
243 242 0:25 /lxc/obase /sys/fs/cgroup/blkio/lxc/obase rw,nosuid,nodev,noexec,relatime master:16 - cgroup cgroup rw,blkio
244 229 0:41 /devices /sys/fs/cgroup/devices ro,relatime - tmpfs cgroup_root rw,size=10240k,mode=755
245 244 0:26 /lxc/obase /sys/fs/cgroup/devices/lxc/obase rw,nosuid,nodev,noexec,relatime master:17 - cgroup cgroup rw,devices
246 229 0:41 /net_cls /sys/fs/cgroup/net_cls ro,relatime - tmpfs cgroup_root rw,size=10240k,mode=755
247 246 0:27 /lxc/obase /sys/fs/cgroup/net_cls/lxc/obase rw,nosuid,nodev,noexec,relatime master:18 - cgroup cgroup rw,net_cls
248 229 0:41 /debug /sys/fs/cgroup/debug ro,relatime - tmpfs cgroup_root rw,size=10240k,mode=755
249 248 0:28 /lxc/obase /sys/fs/cgroup/debug/lxc/obase rw,nosuid,nodev,noexec,relatime master:19 - cgroup cgroup rw,debug
250 229 0:41 /memory /sys/fs/cgroup/memory ro,relatime - tmpfs cgroup_root rw,size=10240k,mode=755
251 250 0:29 /lxc/obase /sys/fs/cgroup/memory/lxc/obase rw,nosuid,nodev,noexec,relatime master:20 - cgroup cgroup rw,memory
252 219 0:11 /1 /dev/lxc/console rw,nosuid,noexec,relatime master:4 - devpts devpts rw,gid=5,mode=620,ptmxmode=000
190 219 0:42 / /dev/pts rw,relatime - devpts devpts rw,gid=5,mode=620,ptmxmode=666
191 219 0:42 /ptmx /dev/ptmx rw,relatime - devpts devpts rw,gid=5,mode=620,ptmxmode=666
192 219 0:42 /0 /dev/lxc/tty1 rw,relatime - devpts devpts rw,gid=5,mode=620,ptmxmode=666
193 219 0:42 /1 /dev/lxc/tty2 rw,relatime - devpts devpts rw,gid=5,mode=620,ptmxmode=666
194 219 0:42 /2 /dev/lxc/tty3 rw,relatime - devpts devpts rw,gid=5,mode=620,ptmxmode=666
195 219 0:42 /3 /dev/lxc/tty4 rw,relatime - devpts devpts rw,gid=5,mode=620,ptmxmode=666
196 219 0:42 /4 /dev/lxc/tty5 rw,relatime - devpts devpts rw,gid=5,mode=620,ptmxmode=666
197 219 0:42 /5 /dev/lxc/tty6 rw,relatime - devpts devpts rw,gid=5,mode=620,ptmxmode=666

@evverx:

# lxc-start -F -n obase -- /bin/bash -c 'cat /proc/self/cgroup /proc/self/mountinfo'           
12:memory:/lxc/obase
11:debug:/lxc/obase
10:net_cls:/lxc/obase
9:devices:/lxc/obase
8:blkio:/lxc/obase
7:hugetlb:/lxc/obase
6:perf_event:/lxc/obase
5:cpu,cpuacct:/lxc/obase
4:freezer:/lxc/obase
3:cpuset:/lxc/obase
2:name=systemd:/lxc/obase
188 189 179:1 /var/lib/lxc/obase/rootfs / rw,relatime master:1 - ext4 /dev/mmcblk0p1 rw,data=ordered
219 188 0:37 / /dev rw,relatime - tmpfs none rw,size=492k,mode=755
220 188 0:39 / /proc rw,nosuid,nodev,noexec,relatime - proc proc rw
221 222 0:39 /sys/net /proc/sys/net rw,nosuid,nodev,noexec,relatime - proc proc rw
222 220 0:39 /sys /proc/sys ro,nosuid,nodev,noexec,relatime - proc proc rw
223 220 0:39 /sysrq-trigger /proc/sysrq-trigger ro,nosuid,nodev,noexec,relatime - proc proc rw
224 188 0:40 / /sys rw,nosuid,nodev,noexec,relatime - sysfs sysfs rw
225 224 0:40 / /sys ro,nosuid,nodev,noexec,relatime - sysfs sysfs rw
226 225 0:40 / /sys/devices/virtual/net rw,relatime - sysfs sysfs rw
227 226 0:40 /devices/virtual/net /sys/devices/virtual/net rw,nosuid,nodev,noexec,relatime - sysfs sysfs rw
228 188 179:1 /var/cache/pacman/pkg /var/cache/pacman/pkg rw,relatime master:1 - ext4 /dev/mmcblk0p1 rw,data=ordered
229 225 0:41 / /sys/fs/cgroup rw,nosuid,nodev,noexec,relatime - tmpfs cgroup_root rw,size=10240k,mode=755
230 229 0:41 /systemd /sys/fs/cgroup/systemd ro,relatime - tmpfs cgroup_root rw,size=10240k,mode=755
231 230 0:19 /lxc/obase /sys/fs/cgroup/systemd/lxc/obase rw,nosuid,nodev,noexec,relatime master:9 - cgroup cgroup rw,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd
232 229 0:41 /cpuset /sys/fs/cgroup/cpuset ro,relatime - tmpfs cgroup_root rw,size=10240k,mode=755
233 232 0:20 /lxc/obase /sys/fs/cgroup/cpuset/lxc/obase rw,nosuid,nodev,noexec,relatime master:11 - cgroup cgroup rw,cpuset
234 229 0:41 /freezer /sys/fs/cgroup/freezer ro,relatime - tmpfs cgroup_root rw,size=10240k,mode=755
235 234 0:21 /lxc/obase /sys/fs/cgroup/freezer/lxc/obase rw,nosuid,nodev,noexec,relatime master:12 - cgroup cgroup rw,freezer
236 229 0:41 /cpu /sys/fs/cgroup/cpu ro,relatime - tmpfs cgroup_root rw,size=10240k,mode=755
237 236 0:22 /lxc/obase /sys/fs/cgroup/cpu/lxc/obase rw,nosuid,nodev,noexec,relatime master:13 - cgroup cgroup rw,cpu,cpuacct
238 229 0:41 /perf_event /sys/fs/cgroup/perf_event ro,relatime - tmpfs cgroup_root rw,size=10240k,mode=755
239 238 0:23 /lxc/obase /sys/fs/cgroup/perf_event/lxc/obase rw,nosuid,nodev,noexec,relatime master:14 - cgroup cgroup rw,perf_event
240 229 0:41 /hugetlb /sys/fs/cgroup/hugetlb ro,relatime - tmpfs cgroup_root rw,size=10240k,mode=755
241 240 0:24 /lxc/obase /sys/fs/cgroup/hugetlb/lxc/obase rw,nosuid,nodev,noexec,relatime master:15 - cgroup cgroup rw,hugetlb
242 229 0:41 /blkio /sys/fs/cgroup/blkio ro,relatime - tmpfs cgroup_root rw,size=10240k,mode=755
243 242 0:25 /lxc/obase /sys/fs/cgroup/blkio/lxc/obase rw,nosuid,nodev,noexec,relatime master:16 - cgroup cgroup rw,blkio
244 229 0:41 /devices /sys/fs/cgroup/devices ro,relatime - tmpfs cgroup_root rw,size=10240k,mode=755
245 244 0:26 /lxc/obase /sys/fs/cgroup/devices/lxc/obase rw,nosuid,nodev,noexec,relatime master:17 - cgroup cgroup rw,devices
246 229 0:41 /net_cls /sys/fs/cgroup/net_cls ro,relatime - tmpfs cgroup_root rw,size=10240k,mode=755
247 246 0:27 /lxc/obase /sys/fs/cgroup/net_cls/lxc/obase rw,nosuid,nodev,noexec,relatime master:18 - cgroup cgroup rw,net_cls
248 229 0:41 /debug /sys/fs/cgroup/debug ro,relatime - tmpfs cgroup_root rw,size=10240k,mode=755
249 248 0:28 /lxc/obase /sys/fs/cgroup/debug/lxc/obase rw,nosuid,nodev,noexec,relatime master:19 - cgroup cgroup rw,debug
250 229 0:41 /memory /sys/fs/cgroup/memory ro,relatime - tmpfs cgroup_root rw,size=10240k,mode=755
251 250 0:29 /lxc/obase /sys/fs/cgroup/memory/lxc/obase rw,nosuid,nodev,noexec,relatime master:20 - cgroup cgroup rw,memory
252 219 0:11 /1 /dev/lxc/console rw,nosuid,noexec,relatime master:4 - devpts devpts rw,gid=5,mode=620,ptmxmode=000
190 219 0:42 / /dev/pts rw,relatime - devpts devpts rw,gid=5,mode=620,ptmxmode=666
191 219 0:42 /ptmx /dev/ptmx rw,relatime - devpts devpts rw,gid=5,mode=620,ptmxmode=666
192 219 0:42 /0 /dev/lxc/tty1 rw,relatime - devpts devpts rw,gid=5,mode=620,ptmxmode=666
193 219 0:42 /1 /dev/lxc/tty2 rw,relatime - devpts devpts rw,gid=5,mode=620,ptmxmode=666
194 219 0:42 /2 /dev/lxc/tty3 rw,relatime - devpts devpts rw,gid=5,mode=620,ptmxmode=666
195 219 0:42 /3 /dev/lxc/tty4 rw,relatime - devpts devpts rw,gid=5,mode=620,ptmxmode=666
196 219 0:42 /4 /dev/lxc/tty5 rw,relatime - devpts devpts rw,gid=5,mode=620,ptmxmode=666
197 219 0:42 /5 /dev/lxc/tty6 rw,relatime - devpts devpts rw,gid=5,mode=620,ptmxmode=666
@evverx

This comment has been minimized.

Show comment
Hide comment
@evverx

evverx Jul 28, 2017

@graysky2, thank you. I've just filed systemd/systemd#6477. It seems that the issue can be worked around by using cgroup namespaces or by mounting cgroup onto /sys/fs/cgroup/systemd.

evverx commented Jul 28, 2017

@graysky2, thank you. I've just filed systemd/systemd#6477. It seems that the issue can be worked around by using cgroup namespaces or by mounting cgroup onto /sys/fs/cgroup/systemd.

@graysky2

This comment has been minimized.

Show comment
Hide comment
@graysky2

graysky2 Jul 28, 2017

Let me know if there is anything test you want me to with it... ie edit an lxc file to do that manual mount step you mentioned or the like.

Let me know if there is anything test you want me to with it... ie edit an lxc file to do that manual mount step you mentioned or the like.

@graysky2

This comment has been minimized.

Show comment
Hide comment
@graysky2

graysky2 Jul 31, 2017

I built systemd with the patch you linked above on my ODROID-C2. After updating both the host and lxc running the current stable kernel package (3.14.79-27-ARCH), everything again works as expected.

@evverx and @brauner - thank you both for all the time and thought you put into this admittedly niche issue 👍

graysky2 commented Jul 31, 2017

I built systemd with the patch you linked above on my ODROID-C2. After updating both the host and lxc running the current stable kernel package (3.14.79-27-ARCH), everything again works as expected.

@evverx and @brauner - thank you both for all the time and thought you put into this admittedly niche issue 👍

stgraber added a commit that referenced this issue Aug 14, 2017

cgroups: handle hybrid cgroup layouts
Closes #1669.
Closes #1678.
Relates to systemd/systemd#6408.

Signed-off-by: Christian Brauner <christian.brauner@ubuntu.com>

@brauner brauner closed this Aug 23, 2017

poettering added a commit to systemd/systemd that referenced this issue Nov 17, 2017

cgroup: assume the use of v1 when all the preceding checks fail (#7366)
This patch restores the default that was changed in 2977724,
making the tools depending on it work again.

Closes: #6477 and lxc/lxc#1669
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment