cannot start containers on Debian with apparmor enabled in the kernel. #1895

terceiro · 2017-11-02T13:33:05Z

Required information

Distribution: Debian
Distribution version: unstable/sid
The output of
- lxc-start --version: 2.0.9
- lxc-checkconfig: lxc-checkconfig.txt
- uname -a: Linux $HOST 4.13.0-1-amd64 Prefix tests with lxc-test- #1 SMP Debian 4.13.10-1 (2017-10-30) x86_64 GNU/Linux
- cat /proc/self/cgroup: cgroup.txt
- cat /proc/1/mounts: mounts.txt

Issue description

After apparmor was enabled in the Debian kernel, I cannot start containers anymore. Log:

lxc_apparmor - lsm/apparmor.c:apparmor_process_label_set:220 - If you really want to start this container, set
lxc_apparmor - lsm/apparmor.c:apparmor_process_label_set:221 - lxc.aa_allow_incomplete = 1
lxc_apparmor - lsm/apparmor.c:apparmor_process_label_set:222 - in your container configuration file
lxc_sync - sync.c:__sync_wait:57 - An error occurred in another process (expected sequence number 5)
lxc_container - lxccontainer.c:wait_on_daemonized_start:754 - Received container state "ABORTING" instead of "RUNNING"
lxc_start - start.c:__lxc_start:1530 - Failed to spawn container "autopkgtest-sid-amd64".
lxc_start_ui - tools/lxc_start.c:main:368 - The container failed to start.
lxc_start_ui - tools/lxc_start.c:main:370 - To get more details, run the container in foreground mode.
lxc_start_ui - tools/lxc_start.c:main:372 - Additional information can be obtained by setting the --logfile and --logpriority options.

Adding lxc.aa_allow_incomplete = 1 to the config, as the error message says, does not help, but produce a different error:

lxc_apparmor - lsm/apparmor.c:apparmor_process_label_set:234 - No such file or directory - failed to change apparmor profile to lxc-container-default-cgns
lxc_sync - sync.c:__sync_wait:57 - An error occurred in another process (expected sequence number 5)
lxc_container - lxccontainer.c:wait_on_daemonized_start:754 - Received container state "ABORTING" instead of "RUNNING"
lxc_start - start.c:__lxc_start:1530 - Failed to spawn container "test".
lxc_start_ui - tools/lxc_start.c:main:368 - The container failed to start.
lxc_start_ui - tools/lxc_start.c:main:370 - To get more details, run the container in foreground mode.
lxc_start_ui - tools/lxc_start.c:main:372 - Additional information can be obtained by setting the --logfile and --logpriority options.

Original Debian bug report: #880502

The text was updated successfully, but these errors were encountered:

terceiro · 2017-11-02T14:06:39Z

I can also reproduce this issue with 2.1.0-devel build from the sources

terceiro · 2017-11-02T14:11:47Z

And on Debian stable, booted with security=apparmor in the kernel command line

root@testlxc:~# lxc-start --version
2.0.7
root@testlxc:~# uname -r
4.9.0-4-amd64

brauner · 2017-11-02T14:36:31Z

This looks like the requested AppArmor profile has not been loaded into the kernel which means that AppArmor cannot do change_profile on container startup. This looks like Debian should load those profiles on boot. Can you show (as root):

aa-status

terceiro · 2017-11-02T16:03:35Z

yep, that looks like it

# aa-status 
apparmor module is loaded.
0 profiles are loaded.
0 profiles are in enforce mode.
0 profiles are in complain mode.
0 processes have profiles defined.
0 processes are in enforce mode.
0 processes are in complain mode.
0 processes are unconfined but have a profile defined.

evgeni · 2017-11-02T16:23:59Z

FWIW, on Stretch, I get the following with lxc and apparmor installed:

root@debian9:~# aa-status 
apparmor module is loaded.
1 profiles are loaded.
1 profiles are in enforce mode.
   /usr/bin/lxc-start
0 profiles are in complain mode.
0 processes have profiles defined.
0 processes are in enforce mode.
0 processes are in complain mode.
0 processes are unconfined but have a profile defined.

evgeni · 2017-11-02T16:24:54Z

Can you try re-installing lxc after apparmor was installed?

vvv-ca · 2017-12-04T09:22:39Z

Same issue on Debian for me:

apparmor module is loaded.
16 profiles are loaded.
16 profiles are in enforce mode.
   /usr/bin/lxc-start
   /usr/bin/man
   /usr/bin/man//filter
   /usr/bin/man//groff
   /usr/lib/cups/backend/cups-pdf
   /usr/sbin/cups-browsed
   /usr/sbin/cupsd
   /usr/sbin/cupsd//third_party
   /usr/sbin/libvirtd
   /usr/sbin/libvirtd//qemu_bridge_helper
   /usr/sbin/tcpdump
   lxc-container-default
   lxc-container-default-cgns
   lxc-container-default-with-mounting
   lxc-container-default-with-nesting
   virt-aa-helper
0 profiles are in complain mode.
3 processes have profiles defined.
3 processes are in enforce mode.
   /usr/sbin/cups-browsed (754) 
   /usr/sbin/cupsd (716) 
   /usr/sbin/libvirtd (942) 
0 processes are in complain mode.
0 processes are unconfined but have a profile defined.

however apparmor blocks mounts:

[ 4701.102345] audit: type=1400 audit(1512378301.275:876): apparmor="DENIED" operation="mount" info="failed mntpnt match" error=-13 profile="/usr/bin/lxc-start" name="/" pid=14856 comm="lxc-start" flags="rw, slave"
[ 4701.102578] audit: type=1400 audit(1512378301.275:877): apparmor="DENIED" operation="mount" info="failed mntpnt match" error=-13 profile="/usr/bin/lxc-start" name="/dev/" pid=14856 comm="lxc-start" flags="rw, slave"
[ 4701.102695] audit: type=1400 audit(1512378301.275:878): apparmor="DENIED" operation="mount" info="failed mntpnt match" error=-13 profile="/usr/bin/lxc-start" name="/dev/pts/" pid=14856 comm="lxc-start" flags="rw, slave"
[ 4701.102801] audit: type=1400 audit(1512378301.275:879): apparmor="DENIED" operation="mount" info="failed mntpnt match" error=-13 profile="/usr/bin/lxc-start" name="/dev/shm/" pid=14856 comm="lxc-start" flags="rw, slave"
[ 4701.102909] audit: type=1400 audit(1512378301.275:880): apparmor="DENIED" operation="mount" info="failed mntpnt match" error=-13 profile="/usr/bin/lxc-start" name="/dev/hugepages/" pid=14856 comm="lxc-start" flags="rw, slave"
[ 4701.103018] audit: type=1400 audit(1512378301.275:881): apparmor="DENIED" operation="mount" info="failed mntpnt match" error=-13 profile="/usr/bin/lxc-start" name="/dev/mqueue/" pid=14856 comm="lxc-start" flags="rw, slave"
[ 4701.103127] audit: type=1400 audit(1512378301.275:882): apparmor="DENIED" operation="mount" info="failed mntpnt match" error=-13 profile="/usr/bin/lxc-start" name="/run/" pid=14856 comm="lxc-start" flags="rw, slave"
[ 4701.103243] audit: type=1400 audit(1512378301.275:883): apparmor="DENIED" operation="mount" info="failed mntpnt match" error=-13 profile="/usr/bin/lxc-start" name="/run/lock/" pid=14856 comm="lxc-start" flags="rw, slave"
[ 4701.103347] audit: type=1400 audit(1512378301.275:884): apparmor="DENIED" operation="mount" info="failed mntpnt match" error=-13 profile="/usr/bin/lxc-start" name="/run/rpc_pipefs/" pid=14856 comm="lxc-start" flags="rw, slave"
[ 4701.103484] audit: type=1400 audit(1512378301.275:885): apparmor="DENIED" operation="mount" info="failed mntpnt match" error=-13 profile="/usr/bin/lxc-start" name="/run/user/1000/" pid=14856 comm="lxc-start" flags="rw, slave"

and I get this in the lxc container log:

lxc-start 20171204090501.277 ERROR    lxc_conf - conf.c:remount_all_slave:2965 - Permission denied - Failed to make / rslave
lxc-start 20171204090501.277 ERROR    lxc_conf - conf.c:remount_all_slave:2966 - Continuing...
lxc-start 20171204090501.277 ERROR    lxc_conf - conf.c:remount_all_slave:2965 - Permission denied - Failed to make /dev rslave
lxc-start 20171204090501.277 ERROR    lxc_conf - conf.c:remount_all_slave:2966 - Continuing...
lxc-start 20171204090501.278 ERROR    lxc_conf - conf.c:remount_all_slave:2965 - Permission denied - Failed to make /dev/pts rslave
lxc-start 20171204090501.278 ERROR    lxc_conf - conf.c:remount_all_slave:2966 - Continuing...
lxc-start 20171204090501.278 ERROR    lxc_conf - conf.c:remount_all_slave:2965 - Permission denied - Failed to make /dev/shm rslave
lxc-start 20171204090501.278 ERROR    lxc_conf - conf.c:remount_all_slave:2966 - Continuing...
lxc-start 20171204090501.278 ERROR    lxc_conf - conf.c:remount_all_slave:2965 - Permission denied - Failed to make /dev/hugepages rslave
lxc-start 20171204090501.278 ERROR    lxc_conf - conf.c:remount_all_slave:2966 - Continuing...
lxc-start 20171204090501.278 ERROR    lxc_conf - conf.c:remount_all_slave:2965 - Permission denied - Failed to make /dev/mqueue rslave
lxc-start 20171204090501.278 ERROR    lxc_conf - conf.c:remount_all_slave:2966 - Continuing...
lxc-start 20171204090501.278 ERROR    lxc_conf - conf.c:remount_all_slave:2965 - Permission denied - Failed to make /run rslave
lxc-start 20171204090501.278 ERROR    lxc_conf - conf.c:remount_all_slave:2966 - Continuing...
lxc-start 20171204090501.278 ERROR    lxc_conf - conf.c:remount_all_slave:2965 - Permission denied - Failed to make /run/lock rslave
lxc-start 20171204090501.278 ERROR    lxc_conf - conf.c:remount_all_slave:2966 - Continuing...
lxc-start 20171204090501.278 ERROR    lxc_conf - conf.c:remount_all_slave:2965 - Permission denied - Failed to make /run/rpc_pipefs rslave
lxc-start 20171204090501.278 ERROR    lxc_conf - conf.c:remount_all_slave:2966 - Continuing...
lxc-start 20171204090501.278 ERROR    lxc_conf - conf.c:remount_all_slave:2965 - Permission denied - Failed to make /run/user/1000 rslave
lxc-start 20171204090501.278 ERROR    lxc_conf - conf.c:remount_all_slave:2966 - Continuing...
lxc-start 20171204090501.278 ERROR    lxc_conf - conf.c:remount_all_slave:2965 - Permission denied - Failed to make /sys rslave
lxc-start 20171204090501.278 ERROR    lxc_conf - conf.c:remount_all_slave:2966 - Continuing...
lxc-start 20171204090501.279 ERROR    lxc_conf - conf.c:remount_all_slave:2965 - Permission denied - Failed to make /sys/kernel/security rslave
lxc-start 20171204090501.279 ERROR    lxc_conf - conf.c:remount_all_slave:2966 - Continuing...
lxc-start 20171204090501.279 ERROR    lxc_conf - conf.c:remount_all_slave:2965 - Permission denied - Failed to make /sys/fs/cgroup rslave
lxc-start 20171204090501.279 ERROR    lxc_conf - conf.c:remount_all_slave:2966 - Continuing...
lxc-start 20171204090501.279 ERROR    lxc_conf - conf.c:remount_all_slave:2965 - Permission denied - Failed to make /sys/fs/cgroup/unified rslave
lxc-start 20171204090501.279 ERROR    lxc_conf - conf.c:remount_all_slave:2966 - Continuing...
lxc-start 20171204090501.279 ERROR    lxc_conf - conf.c:remount_all_slave:2965 - Permission denied - Failed to make /sys/fs/cgroup/systemd rslave
lxc-start 20171204090501.279 ERROR    lxc_conf - conf.c:remount_all_slave:2966 - Continuing...
lxc-start 20171204090501.279 ERROR    lxc_conf - conf.c:remount_all_slave:2965 - Permission denied - Failed to make /sys/fs/cgroup/net_cls,net_prio rslave
lxc-start 20171204090501.279 ERROR    lxc_conf - conf.c:remount_all_slave:2966 - Continuing...
lxc-start 20171204090501.279 ERROR    lxc_conf - conf.c:remount_all_slave:2965 - Permission denied - Failed to make /sys/fs/cgroup/perf_event rslave
lxc-start 20171204090501.279 ERROR    lxc_conf - conf.c:remount_all_slave:2966 - Continuing...
lxc-start 20171204090501.279 ERROR    lxc_conf - conf.c:remount_all_slave:2965 - Permission denied - Failed to make /sys/fs/cgroup/cpuset rslave
lxc-start 20171204090501.279 ERROR    lxc_conf - conf.c:remount_all_slave:2966 - Continuing...
lxc-start 20171204090501.279 ERROR    lxc_conf - conf.c:remount_all_slave:2965 - Permission denied - Failed to make /sys/fs/cgroup/memory rslave
lxc-start 20171204090501.279 ERROR    lxc_conf - conf.c:remount_all_slave:2966 - Continuing...
lxc-start 20171204090501.279 ERROR    lxc_conf - conf.c:remount_all_slave:2965 - Permission denied - Failed to make /sys/fs/cgroup/devices rslave
lxc-start 20171204090501.280 ERROR    lxc_conf - conf.c:remount_all_slave:2966 - Continuing...
lxc-start 20171204090501.280 ERROR    lxc_conf - conf.c:remount_all_slave:2965 - Permission denied - Failed to make /sys/fs/cgroup/blkio rslave
lxc-start 20171204090501.280 ERROR    lxc_conf - conf.c:remount_all_slave:2966 - Continuing...
lxc-start 20171204090501.280 ERROR    lxc_conf - conf.c:remount_all_slave:2965 - Permission denied - Failed to make /sys/fs/cgroup/cpu,cpuacct rslave
lxc-start 20171204090501.280 ERROR    lxc_conf - conf.c:remount_all_slave:2966 - Continuing...
lxc-start 20171204090501.280 ERROR    lxc_conf - conf.c:remount_all_slave:2965 - Permission denied - Failed to make /sys/fs/cgroup/freezer rslave
lxc-start 20171204090501.280 ERROR    lxc_conf - conf.c:remount_all_slave:2966 - Continuing...
lxc-start 20171204090501.280 ERROR    lxc_conf - conf.c:remount_all_slave:2965 - Permission denied - Failed to make /sys/fs/cgroup/pids rslave
lxc-start 20171204090501.280 ERROR    lxc_conf - conf.c:remount_all_slave:2966 - Continuing...
lxc-start 20171204090501.280 ERROR    lxc_conf - conf.c:remount_all_slave:2965 - Permission denied - Failed to make /sys/fs/pstore rslave
lxc-start 20171204090501.281 ERROR    lxc_conf - conf.c:remount_all_slave:2966 - Continuing...
lxc-start 20171204090501.281 ERROR    lxc_conf - conf.c:remount_all_slave:2965 - Permission denied - Failed to make /sys/firmware/efi/efivars rslave
lxc-start 20171204090501.281 ERROR    lxc_conf - conf.c:remount_all_slave:2966 - Continuing...
lxc-start 20171204090501.281 ERROR    lxc_conf - conf.c:remount_all_slave:2965 - Permission denied - Failed to make /sys/kernel/debug rslave
lxc-start 20171204090501.281 ERROR    lxc_conf - conf.c:remount_all_slave:2966 - Continuing...
lxc-start 20171204090501.281 ERROR    lxc_conf - conf.c:remount_all_slave:2965 - Permission denied - Failed to make /sys/fs/fuse/connections rslave
lxc-start 20171204090501.281 ERROR    lxc_conf - conf.c:remount_all_slave:2966 - Continuing...
lxc-start 20171204090501.281 ERROR    lxc_conf - conf.c:remount_all_slave:2965 - Permission denied - Failed to make /proc rslave
lxc-start 20171204090501.281 ERROR    lxc_conf - conf.c:remount_all_slave:2966 - Continuing...
lxc-start 20171204090501.281 ERROR    lxc_conf - conf.c:remount_all_slave:2965 - Permission denied - Failed to make /proc/sys/fs/binfmt_misc rslave
lxc-start 20171204090501.282 ERROR    lxc_conf - conf.c:remount_all_slave:2966 - Continuing...
lxc-start 20171204090501.282 ERROR    lxc_conf - conf.c:remount_all_slave:2965 - Permission denied - Failed to make /boot/efi rslave
lxc-start 20171204090501.282 ERROR    lxc_conf - conf.c:remount_all_slave:2966 - Continuing...
lxc-start 20171204090501.282 ERROR    lxc_conf - conf.c:remount_all_slave:2965 - Permission denied - Failed to make /var/lib/lxcfs rslave
lxc-start 20171204090501.282 ERROR    lxc_conf - conf.c:remount_all_slave:2966 - Continuing...
lxc-start 20171204090501.286 ERROR    storage_utils - storage/storage_utils.c:mount_unknown_fs:335 - failed to determine fs type for '/dev/mycontainer-vg/root'
lxc-start 20171204090501.286 ERROR    lxc_conf - conf.c:lxc_setup_rootfs:1254 - Failed to mount rootfs "/dev/mycontainer-vg/root" onto "/usr/lib/x86_64-linux-gnu/lxc/rootfs" with options "(null)".
lxc-start 20171204090501.286 ERROR    lxc_conf - conf.c:do_rootfs_setup:3044 - failed to setup rootfs for 'mycontainer'
lxc-start 20171204090501.286 ERROR    lxc_conf - conf.c:lxc_setup:3085 - Error setting up rootfs mount after spawn
lxc-start 20171204090501.286 ERROR    lxc_start - start.c:do_start:1020 - Failed to setup container "mycontainer".
lxc-start 20171204090501.286 ERROR    lxc_sync - sync.c:__sync_wait:57 - An error occurred in another process (expected sequence number 5)
lxc-start 20171204090501.320 ERROR    lxc_start - start.c:__lxc_start:1530 - Failed to spawn container "mycontainer".
lxc-start 20171204090501.869 ERROR    lxc_start_ui - tools/lxc_start.c:main:368 - The container failed to start.
lxc-start 20171204090501.869 ERROR    lxc_start_ui - tools/lxc_start.c:main:372 - Additional information can be obtained by setting the --logfile and --logpriority options.

I did also re-install lxc after apparmor was installed.

Polve · 2017-12-12T07:30:05Z

I have the same problem, any workaround?

Without having to disable apparmor with kernel options, if possible

terceiro · 2017-12-12T10:47:33Z

lxc in debian testing and unstable at the moment disables apparmor for all containers by default while we don't have a fix for this. you want this;

lxc.aa_profile = unconfined

terceiro · 2017-12-19T12:34:30Z

This issue has been fixed in Debian by apparmor 2.11.1-4. See Debian bug 883703 for more information.

I am thus dropping the workaround I had put into the Debian lxc packages.

harridu · 2018-06-05T17:16:28Z

Dropping the workaround was a bad idea (imho). It ties lxc to a newer apparmor version not available in Stretch. Makes backporting lxc pretty difficult. Not to mention that lxc's control file doesn't set Conflicts: accordingly. I would suggest a line "Conflicts: libapparmor1 <= 2.11.1-4".

I highly appreciate the lxc.aa_profile = unconfined

terceiro · 2018-06-12T13:04:10Z

a bad idea is not making use of a security feature like apparmor when it is available.

your point about the dependency on libapparmor is valid, though, but "Conflicts:" is not the correct solution. I added an explicit dependency on the specific version of apparmor that has the fix we need.

khaefeli · 2018-08-08T15:51:56Z

best thing would probably be, to have the patched version in Debian Strech and not the old, buggy one.. :)

parithy · 2018-09-11T08:02:38Z

Issue is still persisting
parithy@debian-acer ~> uname -a
Linux debian-acer 4.17.0-3-amd64 #1 SMP Debian 4.17.17-1 (2018-08-18) x86_64 GNU/Linux
parithy@debian-acer ~> dpkg -l | grep armor
ii apparmor 2.13-8 amd64 user-space parser utility for AppArmor
ii libapparmor1:amd64 2.13-8 amd64 changehat AppArmor library
parithy@debian-acer ~>

lxc/lxc#1895 apparmor guvenlik siklastirma kapsaminda yuklenen bir tool

geez0x1 · 2018-12-19T14:36:48Z

@parithy I encountered a similar looking issue earlier today while trying to launch my first LXC container on Debian stable (using exclusively packages from the stable repositories). In the end, a simple systemctl restart apparmor fixed the issue. Perhaps try that?

edit: Later on I encountered a whole bunch of other issues, but with stopping containers. Their init process would become unkillable (even with kill -9 from the host), an issue which is easy to find on the internet but I could not find a consistent fix for. ~~Rebooting, somehow, fixed everything. Hence after installing LXC I highly recommend rebooting the machine, before working with any containers.~~ After a very, very long search this turned out to be my non-default umask (027).

See e.g.
https://lists.linuxcontainers.org/pipermail/lxc-users/2016-December/012612.html
https://bugs.launchpad.net/ubuntu/+source/lxc/+bug/1642767
#2277

fjleon1980 · 2019-05-07T02:07:01Z

debian 9 stretch on azure comes installed with libapparmor, but not apparmor itself so i couldn't start the container. i had to use the workaround given by terceiro

itoffshore · 2020-01-26T21:45:10Z

lxc.aa_profile is deprecated now - for lxc-3.2.1 in Arch Linux (using apparmor) I needed to use:

lxc.apparmor.profile = unconfined

to get an Alpine Linux lxc container from the download template to start

terceiro closed this as completed Dec 19, 2017

mt-caret mentioned this issue Feb 10, 2018

lxd: fails to launch containers NixOS/nixpkgs#25280

Closed

iercan pushed a commit to alptugay/MYS that referenced this issue Oct 10, 2018

lxc apparmor entegrasyon sorunu duzeltildi. Detaylar asagidaki linkte

d466674

lxc/lxc#1895 apparmor guvenlik siklastirma kapsaminda yuklenen bir tool

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cannot start containers on Debian with apparmor enabled in the kernel. #1895

cannot start containers on Debian with apparmor enabled in the kernel. #1895

terceiro commented Nov 2, 2017 •

edited

Loading

terceiro commented Nov 2, 2017

terceiro commented Nov 2, 2017

brauner commented Nov 2, 2017

terceiro commented Nov 2, 2017

evgeni commented Nov 2, 2017

evgeni commented Nov 2, 2017

vvv-ca commented Dec 4, 2017

Polve commented Dec 12, 2017 •

edited

Loading

terceiro commented Dec 12, 2017

terceiro commented Dec 19, 2017

harridu commented Jun 5, 2018 •

edited

Loading

terceiro commented Jun 12, 2018

khaefeli commented Aug 8, 2018

parithy commented Sep 11, 2018

geez0x1 commented Dec 19, 2018 •

edited

Loading

fjleon1980 commented May 7, 2019

itoffshore commented Jan 26, 2020 •

edited

Loading

cannot start containers on Debian with apparmor enabled in the kernel. #1895

cannot start containers on Debian with apparmor enabled in the kernel. #1895

Comments

terceiro commented Nov 2, 2017 • edited Loading

Required information

Issue description

terceiro commented Nov 2, 2017

terceiro commented Nov 2, 2017

brauner commented Nov 2, 2017

terceiro commented Nov 2, 2017

evgeni commented Nov 2, 2017

evgeni commented Nov 2, 2017

vvv-ca commented Dec 4, 2017

Polve commented Dec 12, 2017 • edited Loading

terceiro commented Dec 12, 2017

terceiro commented Dec 19, 2017

harridu commented Jun 5, 2018 • edited Loading

terceiro commented Jun 12, 2018

khaefeli commented Aug 8, 2018

parithy commented Sep 11, 2018

geez0x1 commented Dec 19, 2018 • edited Loading

fjleon1980 commented May 7, 2019

itoffshore commented Jan 26, 2020 • edited Loading

terceiro commented Nov 2, 2017 •

edited

Loading

Polve commented Dec 12, 2017 •

edited

Loading

harridu commented Jun 5, 2018 •

edited

Loading

geez0x1 commented Dec 19, 2018 •

edited

Loading

itoffshore commented Jan 26, 2020 •

edited

Loading