LXCFS, Alpine Linux and Systemd containers

[bocajspear1]

I'm trying to run Systemd containers on Alpine Linux using lxcfs. I managed to get things working once upon a time, and I documented my steps here: https://j2h2.com/entry/alpine-linux-systemd-containers.

However, I came back to update cgmanager and lxcfs, and now it doesn't work anymore.

I'm using lxc 1.1.5, and the latest lxcfs and cgmanager.
Kernel is 4.4.30.

I've been trying to troubleshoot this for 10+ hours, and I'm at my wits end.

This is the output of the container:

lxc-start: utils.c: safe_mount: 1692 Invalid argument - Failed to mount /sys/kernel/debug onto /usr/lib/lxc/rootfs/sys/kernel/debug
lxc-start: utils.c: open_without_symlink: 1626 No such file or directory - Error examining efi in /usr/lib/lxc/rootfs/sys/firmware/efi/efivars
umount: can't unmount /usr/lib/lxc/rootfs/sys/fs/cgroup/blkio/lxc/u16: Invalid argument
umount: can't unmount /usr/lib/lxc/rootfs/sys/fs/cgroup/cpuacct/lxc/u16: Invalid argument
umount: can't unmount /usr/lib/lxc/rootfs/sys/fs/cgroup/cpu/lxc/u16: Invalid argument
umount: can't unmount /usr/lib/lxc/rootfs/sys/fs/cgroup/cpuset/lxc/u16: Invalid argument
umount: can't unmount /usr/lib/lxc/rootfs/sys/fs/cgroup/devices/lxc/u16: Invalid argument
umount: can't unmount /usr/lib/lxc/rootfs/sys/fs/cgroup/freezer/lxc/u16: Invalid argument
umount: can't unmount /usr/lib/lxc/rootfs/sys/fs/cgroup/memory/lxc/u16: Invalid argument
umount: can't unmount /usr/lib/lxc/rootfs/sys/fs/cgroup/net_cls/lxc/u16: Invalid argument
umount: can't unmount /usr/lib/lxc/rootfs/sys/fs/cgroup/net_prio/lxc/u16: Invalid argument
umount: can't unmount /usr/lib/lxc/rootfs/sys/fs/cgroup/pids/lxc/u16: Invalid argument
systemd 229 running in system mode. (+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ -LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD -IDN)
Detected virtualization lxc.
Detected architecture x86-64.

Welcome to Ubuntu 16.04.1 LTS!

Set hostname to <u16>.
Failed to read AF_UNIX datagram queue length, ignoring: No such file or directory
Failed to install release agent, ignoring: File exists
Failed to create /init.scope control group: Permission denied
Failed to allocate manager object: Permission denied
[!!!!!!] Failed to allocate manager object, freezing.
Freezing execution.

Debug output:

lxc-start 1478667036.945 INFO     lxc_start_ui - lxc_start.c:main:264 - using rcfile /var/lib/lxc/u16/config
lxc-start 1478667036.945 WARN     lxc_confile - confile.c:config_pivotdir:1801 - lxc.pivotdir is ignored.  It will soon become an error.
lxc-start 1478667036.945 INFO     lxc_confile - confile.c:config_idmap:1437 - read uid map: type u nsid 0 hostid 100000 range 65536
lxc-start 1478667036.945 INFO     lxc_confile - confile.c:config_idmap:1437 - read uid map: type g nsid 0 hostid 100000 range 65536
lxc-start 1478667036.945 WARN     lxc_cgfs - cgfs.c:lxc_cgroup_get_container_info:1100 - Not attaching to cgroup cpuset unknown to /var/lib/lxc u16
lxc-start 1478667036.945 WARN     lxc_cgfs - cgfs.c:lxc_cgroup_get_container_info:1100 - Not attaching to cgroup cpu unknown to /var/lib/lxc u16
lxc-start 1478667036.945 WARN     lxc_cgfs - cgfs.c:lxc_cgroup_get_container_info:1100 - Not attaching to cgroup cpuacct unknown to /var/lib/lxc u16
lxc-start 1478667036.945 WARN     lxc_cgfs - cgfs.c:lxc_cgroup_get_container_info:1100 - Not attaching to cgroup blkio unknown to /var/lib/lxc u16
lxc-start 1478667036.945 WARN     lxc_cgfs - cgfs.c:lxc_cgroup_get_container_info:1100 - Not attaching to cgroup memory unknown to /var/lib/lxc u16
lxc-start 1478667036.945 WARN     lxc_cgfs - cgfs.c:lxc_cgroup_get_container_info:1100 - Not attaching to cgroup devices unknown to /var/lib/lxc u16
lxc-start 1478667036.945 WARN     lxc_cgfs - cgfs.c:lxc_cgroup_get_container_info:1100 - Not attaching to cgroup freezer unknown to /var/lib/lxc u16
lxc-start 1478667036.945 WARN     lxc_cgfs - cgfs.c:lxc_cgroup_get_container_info:1100 - Not attaching to cgroup net_cls unknown to /var/lib/lxc u16
lxc-start 1478667036.945 WARN     lxc_cgfs - cgfs.c:lxc_cgroup_get_container_info:1100 - Not attaching to cgroup net_prio unknown to /var/lib/lxc u16
lxc-start 1478667036.945 WARN     lxc_cgfs - cgfs.c:lxc_cgroup_get_container_info:1100 - Not attaching to cgroup pids unknown to /var/lib/lxc u16
lxc-start 1478667036.945 DEBUG    lxc_start - start.c:setup_signal_fd:264 - sigchild handler set
lxc-start 1478667036.946 DEBUG    lxc_console - console.c:lxc_console_peer_default:500 - opening /dev/tty for console peer
lxc-start 1478667036.946 DEBUG    lxc_console - console.c:lxc_console_peer_default:506 - using '/dev/tty' as console
lxc-start 1478667036.946 DEBUG    lxc_console - console.c:lxc_console_sigwinch_init:179 - 3022 got SIGWINCH fd 9
lxc-start 1478667036.946 DEBUG    lxc_console - console.c:lxc_console_winsz:88 - set winsz dstfd:6 cols:190 rows:48
lxc-start 1478667036.946 INFO     lxc_start - start.c:lxc_init:460 - 'u16' is initialized
lxc-start 1478667036.946 DEBUG    lxc_start - start.c:__lxc_start:1165 - Not dropping cap_sys_boot or watching utmp
lxc-start 1478667036.946 INFO     lxc_start - start.c:resolve_clone_flags:869 - Cloning a new user namespace
lxc-start 1478667036.946 DEBUG    lxc_conf - conf.c:instantiate_veth:2827 - instantiated veth 'veth5WTNN3/veth5F67JO', index is '9'
lxc-start 1478667036.946 INFO     lxc_cgroup - cgroup.c:cgroup_init:65 - cgroup driver cgroupfs initing for u16
lxc-start 1478667036.953 DEBUG    lxc_conf - conf.c:lxc_assign_network:3244 - move '(null)' to '3034'
lxc-start 1478667036.953 NOTICE   lxc_start - start.c:do_start:685 - switching to gid/uid 0 in new user namespace
lxc-start 1478667036.953 DEBUG    lxc_conf - conf.c:setup_rootfs:1295 - mounted '/var/lib/lxc/u16/rootfs' on '/usr/lib/lxc/rootfs'
lxc-start 1478667036.953 INFO     lxc_conf - conf.c:setup_utsname:928 - 'u16' hostname has been setup
lxc-start 1478667036.954 DEBUG    lxc_conf - conf.c:setup_netdev:2595 - 'eth0' has been setup
lxc-start 1478667036.954 INFO     lxc_conf - conf.c:setup_network:2616 - network has been setup
lxc-start 1478667036.954 INFO     lxc_conf - conf.c:mount_autodev:1157 - Mounting container /dev
lxc-start 1478667036.954 INFO     lxc_conf - conf.c:mount_autodev:1179 - Mounted tmpfs onto /usr/lib/lxc/rootfs/dev
lxc-start 1478667036.954 INFO     lxc_conf - conf.c:mount_autodev:1197 - Mounted container /dev
lxc-start 1478667036.954 DEBUG    lxc_conf - conf.c:mount_entry:1738 - remounting /sys/fs/fuse/connections on /usr/lib/lxc/rootfs/sys/fs/fuse/connections to respect bind or remount options
lxc-start 1478667036.954 DEBUG    lxc_conf - conf.c:mount_entry:1753 - (at remount) flags for /sys/fs/fuse/connections was 4142, required extra flags are 14
lxc-start 1478667036.954 DEBUG    lxc_conf - conf.c:mount_entry:1788 - mounted '/sys/fs/fuse/connections' on '/usr/lib/lxc/rootfs/sys/fs/fuse/connections', type 'none'
lxc-start 1478667036.954 ERROR    lxc_utils - utils.c:safe_mount:1692 - Invalid argument - Failed to mount /sys/kernel/debug onto /usr/lib/lxc/rootfs/sys/kernel/debug
lxc-start 1478667036.954 INFO     lxc_conf - conf.c:mount_entry:1727 - failed to mount '/sys/kernel/debug' on '/usr/lib/lxc/rootfs/sys/kernel/debug' (optional): Invalid argument
lxc-start 1478667036.954 DEBUG    lxc_conf - conf.c:mount_entry:1738 - remounting /sys/kernel/security on /usr/lib/lxc/rootfs/sys/kernel/security to respect bind or remount options
lxc-start 1478667036.954 DEBUG    lxc_conf - conf.c:mount_entry:1753 - (at remount) flags for /sys/kernel/security was 4142, required extra flags are 14
lxc-start 1478667036.954 DEBUG    lxc_conf - conf.c:mount_entry:1788 - mounted '/sys/kernel/security' on '/usr/lib/lxc/rootfs/sys/kernel/security', type 'none'
lxc-start 1478667036.954 DEBUG    lxc_conf - conf.c:mount_entry:1738 - remounting /sys/fs/pstore on /usr/lib/lxc/rootfs/sys/fs/pstore to respect bind or remount options
lxc-start 1478667036.954 DEBUG    lxc_conf - conf.c:mount_entry:1753 - (at remount) flags for /sys/fs/pstore was 4142, required extra flags are 14
lxc-start 1478667036.954 DEBUG    lxc_conf - conf.c:mount_entry:1788 - mounted '/sys/fs/pstore' on '/usr/lib/lxc/rootfs/sys/fs/pstore', type 'none'
lxc-start 1478667036.954 DEBUG    lxc_conf - conf.c:mount_entry:1788 - mounted 'mqueue' on '/usr/lib/lxc/rootfs/dev/mqueue', type 'mqueue'
lxc-start 1478667036.954 DEBUG    lxc_conf - conf.c:mount_entry:1738 - remounting /dev/console on /usr/lib/lxc/rootfs/dev/console to respect bind or remount options
lxc-start 1478667036.954 DEBUG    lxc_conf - conf.c:mount_entry:1753 - (at remount) flags for /dev/console was 4130, required extra flags are 2
lxc-start 1478667036.954 DEBUG    lxc_conf - conf.c:mount_entry:1788 - mounted '/dev/console' on '/usr/lib/lxc/rootfs/dev/console', type 'none'
lxc-start 1478667036.954 DEBUG    lxc_conf - conf.c:mount_entry:1738 - remounting /dev/full on /usr/lib/lxc/rootfs/dev/full to respect bind or remount options
lxc-start 1478667036.954 DEBUG    lxc_conf - conf.c:mount_entry:1753 - (at remount) flags for /dev/full was 4130, required extra flags are 2
lxc-start 1478667036.954 DEBUG    lxc_conf - conf.c:mount_entry:1788 - mounted '/dev/full' on '/usr/lib/lxc/rootfs/dev/full', type 'none'
lxc-start 1478667036.954 DEBUG    lxc_conf - conf.c:mount_entry:1738 - remounting /dev/null on /usr/lib/lxc/rootfs/dev/null to respect bind or remount options
lxc-start 1478667036.954 DEBUG    lxc_conf - conf.c:mount_entry:1753 - (at remount) flags for /dev/null was 4130, required extra flags are 2
lxc-start 1478667036.954 DEBUG    lxc_conf - conf.c:mount_entry:1788 - mounted '/dev/null' on '/usr/lib/lxc/rootfs/dev/null', type 'none'
lxc-start 1478667036.954 DEBUG    lxc_conf - conf.c:mount_entry:1738 - remounting /dev/random on /usr/lib/lxc/rootfs/dev/random to respect bind or remount options
lxc-start 1478667036.954 DEBUG    lxc_conf - conf.c:mount_entry:1753 - (at remount) flags for /dev/random was 4130, required extra flags are 2
lxc-start 1478667036.954 DEBUG    lxc_conf - conf.c:mount_entry:1788 - mounted '/dev/random' on '/usr/lib/lxc/rootfs/dev/random', type 'none'
lxc-start 1478667036.954 DEBUG    lxc_conf - conf.c:mount_entry:1738 - remounting /dev/tty on /usr/lib/lxc/rootfs/dev/tty to respect bind or remount options
lxc-start 1478667036.954 DEBUG    lxc_conf - conf.c:mount_entry:1753 - (at remount) flags for /dev/tty was 4130, required extra flags are 2
lxc-start 1478667036.954 DEBUG    lxc_conf - conf.c:mount_entry:1788 - mounted '/dev/tty' on '/usr/lib/lxc/rootfs/dev/tty', type 'none'
lxc-start 1478667036.954 DEBUG    lxc_conf - conf.c:mount_entry:1738 - remounting /dev/urandom on /usr/lib/lxc/rootfs/dev/urandom to respect bind or remount options
lxc-start 1478667036.954 DEBUG    lxc_conf - conf.c:mount_entry:1753 - (at remount) flags for /dev/urandom was 4130, required extra flags are 2
lxc-start 1478667036.954 DEBUG    lxc_conf - conf.c:mount_entry:1788 - mounted '/dev/urandom' on '/usr/lib/lxc/rootfs/dev/urandom', type 'none'
lxc-start 1478667036.954 DEBUG    lxc_conf - conf.c:mount_entry:1738 - remounting /dev/zero on /usr/lib/lxc/rootfs/dev/zero to respect bind or remount options
lxc-start 1478667036.954 DEBUG    lxc_conf - conf.c:mount_entry:1753 - (at remount) flags for /dev/zero was 4130, required extra flags are 2
lxc-start 1478667036.954 DEBUG    lxc_conf - conf.c:mount_entry:1788 - mounted '/dev/zero' on '/usr/lib/lxc/rootfs/dev/zero', type 'none'
lxc-start 1478667036.954 ERROR    lxc_utils - utils.c:open_without_symlink:1626 - No such file or directory - Error examining efi in /usr/lib/lxc/rootfs/sys/firmware/efi/efivars
lxc-start 1478667036.954 INFO     lxc_conf - conf.c:mount_entry:1727 - failed to mount '/sys/firmware/efi/efivars' on '/usr/lib/lxc/rootfs/sys/firmware/efi/efivars' (optional): No such file or directory
lxc-start 1478667036.954 DEBUG    lxc_conf - conf.c:mount_entry:1738 - remounting /proc/sys/fs/binfmt_misc on /usr/lib/lxc/rootfs/proc/sys/fs/binfmt_misc to respect bind or remount options
lxc-start 1478667036.954 DEBUG    lxc_conf - conf.c:mount_entry:1753 - (at remount) flags for /proc/sys/fs/binfmt_misc was 4142, required extra flags are 14
lxc-start 1478667036.954 DEBUG    lxc_conf - conf.c:mount_entry:1788 - mounted '/proc/sys/fs/binfmt_misc' on '/usr/lib/lxc/rootfs/proc/sys/fs/binfmt_misc', type 'none'
lxc-start 1478667036.954 INFO     lxc_conf - conf.c:mount_file_entries:2150 - mount points have been setup
lxc-start 1478667036.954 INFO     lxc_conf - conf.c:run_script_argv:362 - Executing script '/usr/share/lxcfs/lxc.mount.hook' for container 'u16', config section 'lxc'
lxc-start 1478667037.014 INFO     lxc_conf - conf.c:fill_autodev:1225 - Creating initial consoles under container /dev
lxc-start 1478667037.014 INFO     lxc_conf - conf.c:fill_autodev:1236 - Populating container /dev
lxc-start 1478667037.014 INFO     lxc_conf - conf.c:fill_autodev:1269 - Populated container /dev
lxc-start 1478667037.014 INFO     lxc_conf - conf.c:setup_dev_console:1518 - console has been setup
lxc-start 1478667037.014 INFO     lxc_utils - utils.c:mount_proc_if_needed:1724 - I am 1, /proc/self points to '1'
lxc-start 1478667037.014 DEBUG    lxc_conf - conf.c:setup_rootfs_pivot_root:1135 - pivot_root syscall to '/usr/lib/lxc/rootfs' successful
lxc-start 1478667037.014 DEBUG    lxc_conf - conf.c:lxc_create_tty:3488 - allocated pty '/dev/pts/0' (11/14)
lxc-start 1478667037.014 DEBUG    lxc_conf - conf.c:lxc_create_tty:3488 - allocated pty '/dev/pts/1' (15/16)
lxc-start 1478667037.014 DEBUG    lxc_conf - conf.c:lxc_create_tty:3488 - allocated pty '/dev/pts/2' (17/18)
lxc-start 1478667037.014 DEBUG    lxc_conf - conf.c:lxc_create_tty:3488 - allocated pty '/dev/pts/3' (19/20)
lxc-start 1478667037.014 INFO     lxc_conf - conf.c:lxc_create_tty:3499 - tty's configured
lxc-start 1478667037.014 INFO     lxc_conf - conf.c:setup_tty:1080 - 4 tty(s) has been setup
lxc-start 1478667037.014 INFO     lxc_conf - conf.c:setup_personality:1473 - set personality to '0x0'
lxc-start 1478667037.014 DEBUG    lxc_conf - conf.c:setup_caps:2279 - drop capability 'mac_admin' (33)
lxc-start 1478667037.014 DEBUG    lxc_conf - conf.c:setup_caps:2279 - drop capability 'mac_override' (32)
lxc-start 1478667037.014 DEBUG    lxc_conf - conf.c:setup_caps:2279 - drop capability 'sys_time' (25)
lxc-start 1478667037.014 DEBUG    lxc_conf - conf.c:setup_caps:2279 - drop capability 'sys_module' (16)
lxc-start 1478667037.014 DEBUG    lxc_conf - conf.c:setup_caps:2288 - capabilities have been setup
lxc-start 1478667037.014 NOTICE   lxc_conf - conf.c:lxc_setup:4032 - 'u16' is setup.
lxc-start 1478667037.014 NOTICE   lxc_start - start.c:start:1274 - exec'ing '/sbin/init'
lxc-start 1478667037.014 NOTICE   lxc_start - start.c:post_start:1285 - '/sbin/init' started with pid '3034'
lxc-start 1478667037.014 WARN     lxc_start - start.c:signal_handler:312 - invalid pid for SIGCHLD

[hallyn]

Hi could you please show the results of cat /proc/cgroups cat /proc/self/cgroup grep cgroup /proc/self/mountinfo cat /etc/lxc/lxc.conf and the full container configuration? note that since cgfs is being used, either cgmanager is broken or the lxc version isn't built with cgmanager support. Are you running these containers as non-root? thanks.

[bocajspear1]

test:~# cat /proc/cgroups
#subsys_name    hierarchy       num_cgroups     enabled
cpuset  2       1       1
cpu     3       1       1
cpuacct 4       1       1
blkio   5       1       1
memory  6       1       1
devices 7       1       1
freezer 8       1       1
net_cls 9       1       1
net_prio        10      1       1
pids    11      1       1

test:~# cat /proc/self/cgroup
12:name=systemd:/
11:pids:/
10:net_prio:/
9:net_cls:/
8:freezer:/
7:devices:/
6:memory:/
5:blkio:/
4:cpuacct:/
3:cpu:/
2:cpuset:/
1:name=openrc:/sshd

test:~# grep cgroup /proc/self/mountinfo
26 16 0:20 / /sys/fs/cgroup rw,nosuid,nodev,noexec,relatime - tmpfs cgroup_root rw,size=10240k,mode=755
27 26 0:21 / /sys/fs/cgroup/openrc rw,nosuid,nodev,noexec,relatime - cgroup openrc rw,release_agent=/lib/rc/sh/cgroup-release-agent.sh,name=openrc
28 26 0:22 / /sys/fs/cgroup/cpuset rw,nosuid,nodev,noexec,relatime - cgroup cpuset rw,cpuset,clone_children
29 26 0:23 / /sys/fs/cgroup/cpu rw,nosuid,nodev,noexec,relatime - cgroup cpu rw,cpu
30 26 0:24 / /sys/fs/cgroup/cpuacct rw,nosuid,nodev,noexec,relatime - cgroup cpuacct rw,cpuacct
31 26 0:25 / /sys/fs/cgroup/blkio rw,nosuid,nodev,noexec,relatime - cgroup blkio rw,blkio
32 26 0:26 / /sys/fs/cgroup/memory rw,nosuid,nodev,noexec,relatime - cgroup memory rw,memory
33 26 0:27 / /sys/fs/cgroup/devices rw,nosuid,nodev,noexec,relatime - cgroup devices rw,devices
34 26 0:28 / /sys/fs/cgroup/freezer rw,nosuid,nodev,noexec,relatime - cgroup freezer rw,freezer
35 26 0:29 / /sys/fs/cgroup/net_cls rw,nosuid,nodev,noexec,relatime - cgroup net_cls rw,net_cls
36 26 0:30 / /sys/fs/cgroup/net_prio rw,nosuid,nodev,noexec,relatime - cgroup net_prio rw,net_prio
37 26 0:31 / /sys/fs/cgroup/pids rw,nosuid,nodev,noexec,relatime - cgroup pids rw,pids,release_agent=/run/cgmanager/agents/cgm-release-agent.pids

For good measure:

test:~# cat /etc/lxc/default.conf 
lxc.network.type = empty
lxc.network.type = veth
lxc.id_map = u 0 100000 65536
lxc.id_map = g 0 100000 65536

I don't have lxc.conf

These are root non-privileged domains.

Thanks.

[graysky2]

@bocajspear1 - Have you solved this? I am experiencing something very similar with unprivileged containers on Arch.

[brauner]

@graysky2, the thread you linked is unrelated to this issue and has been sufficiently answered by Fajar, I think (https://lists.linuxcontainers.org/pipermail/lxc-users/2017-January/012734.html). :)

[bocajspear1]

Finally figured out a fix for this. I found this issue: debops/ansible-lxc#15, where it mentions creating and mounting the systemd cgroup manually. So I did this:

mkdir -p /sys/fs/cgroup/systemd
mount -t cgroup -o none,name=systemd systemd /sys/fs/cgroup/systemd
chown 100000:100000 -R /sys/fs/cgroup/systemd/
lxcfs /usr/var/lib/lxcfs/ &

It seems to work, except I do get a lot of

Failed to create cgroup /user.slice: Permission denied

when the container starts in the foreground. Is this normal?

[hallyn]

Can you show more complete logs? (lxc-start -n container -l trace -o debug.out, post debug.out)

Then after startup, show /proc/1/cgroup inside the container, and the result of 'tree -d /sys/fs/cgroup/systemd' on the host.

[brauner]

Closing since there was no response in over a year.