Failed to reset devices.list...Operation not permitted

[CalebEverett]

Required information

Distribution

Linux version 4.4.0-21-generic (buildd@lgw01-21) (gcc version 5.3.1 20160413 (Ubuntu 5.3.1-14ubuntu2) ) #37-Ubuntu SMP Mon Apr 18 18:33:37 UTC 2016

LXC Info

  driver: lxc
  driverversion: 2.0.0
  kernel: Linux
  kernelarchitecture: x86_64
  kernelversion: 4.4.0-21-generic
  server: lxd
  serverpid: 16430
  serverversion: 2.0.0
  storage: zfs
  storageversion: "5"
config:
  core.https_address: '[::]:8443'
  core.trust_password: true
  storage.zfs_pool_name: lxd

Issue description

Systemd in container fails to reset devices.list on system.slice when adding a new unit.

May  8 05:14:03 ubuntu rsyslogd: [origin software="rsyslogd" swVersion="8.16.0" x-pid="107" x-info="http://www.rsyslog.com"] start
May  8 05:14:03 ubuntu rsyslogd-2307: warning: ~ action is deprecated, consider using the 'stop' statement instead [v8.16.0 try http://www.rsyslog.com/e/2307 ]
May  8 05:14:03 ubuntu rsyslogd-2145: activation of module imklog failed [v8.16.0 try http://www.rsyslog.com/e/2145 ]
May  8 05:14:03 ubuntu systemd[1]: Started udev Kernel Device Manager.
May  8 05:14:03 ubuntu systemd[1]: Started Login Service.
May  8 05:14:03 ubuntu systemd[1]: Started System Logging Service.
May  8 05:14:03 ubuntu systemd[1]: Failed to reset devices.list on /system.slice: Operation not permitted
May  8 05:14:03 ubuntu mdadm[110]:  * Not starting MD monitoring service in container
May  8 05:14:03 ubuntu systemd[1]: Failed to reset devices.list on /system.slice/polkitd.service: Operation not permitted
May  8 05:14:03 ubuntu systemd[1]: Starting Authenticate and Authorize Users to Run Privileged Tasks...
May  8 05:14:03 ubuntu systemd[1]: Failed to reset devices.list on /user.slice: Operation not permitted
May  8 05:14:03 ubuntu systemd[1]: Failed to reset devices.list on /init.scope: Operation not permitted
May  8 05:14:03 ubuntu systemd[1]: Failed to reset devices.list on /system.slice/systemd-journald.service: Operation not permitted
May  8 05:14:03 ubuntu systemd[1]: Failed to reset devices.list on /system.slice/system-getty.slice: Operation not permitted
May  8 05:14:03 ubuntu systemd[1]: Failed to reset devices.list on /system.slice/proc-cpuinfo.mount: Operation not permitted
May  8 05:14:03 ubuntu systemd[1]: Failed to reset devices.list on /system.slice/proc-uptime.mount: Operation not permitted
May  8 05:14:03 ubuntu systemd[1]: Failed to reset devices.list on /system.slice/resolvconf.service: Operation not permitted
May  8 05:14:03 ubuntu systemd[1]: Failed to reset devices.list on /system.slice/sys-fs-fuse-connections.mount: Operation not permitted
May  8 05:14:03 ubuntu systemd[1]: Failed to reset devices.list on /system.slice/dev-mqueue.mount: Operation not permitted
May  8 05:14:03 ubuntu systemd[1]: Failed to reset devices.list on /system.slice/sys-kernel-debug.mount: Operation not permitted
May  8 05:14:03 ubuntu systemd[1]: Failed to reset devices.list on /system.slice/lxd-containers.service: Operation not permitted
May  8 05:14:03 ubuntu systemd[1]: Failed to reset devices.list on /system.slice/dev-urandom.mount: Operation not permitted
May  8 05:14:03 ubuntu systemd[1]: Failed to reset devices.list on /system.slice/dev-random.mount: Operation not permitted
May  8 05:14:03 ubuntu systemd[1]: Failed to reset devices.list on /system.slice/dev-null.mount: Operation not permitted
May  8 05:14:03 ubuntu systemd[1]: Failed to reset devices.list on /system.slice/dev-lxd.mount: Operation not permitted
May  8 05:14:03 ubuntu systemd[1]: Failed to reset devices.list on /system.slice/cloud-init-local.service: Operation not permitted
May  8 05:14:03 ubuntu systemd[1]: Failed to reset devices.list on /system.slice/systemd-journal-flush.service: Operation not permitted
May  8 05:14:03 ubuntu systemd[1]: Failed to reset devices.list on /system.slice/atd.service: Operation not permitted
May  8 05:14:03 ubuntu systemd[1]: Failed to reset devices.list on /system.slice/dev-full.mount: Operation not permitted
May  8 05:14:03 ubuntu systemd[1]: Failed to reset devices.list on /system.slice/dev-zero.mount: Operation not permitted
May  8 05:14:03 ubuntu systemd[1]: Failed to reset devices.list on /system.slice/rsyslog.service: Operation not permitted
May  8 05:14:03 ubuntu systemd[1]: Failed to reset devices.list on /system.slice/systemd-udevd.service: Operation not permitted
May  8 05:14:03 ubuntu systemd[1]: Failed to reset devices.list on /system.slice/dev-.lxd\x2dmounts.mount: Operation not permitted
May  8 05:14:03 ubuntu polkitd[147]: started daemon version 0.105 using authority implementation `local' version `0.105'
May  8 05:14:03 ubuntu systemd[1]: Failed to reset devices.list on /system.slice/proc-diskstats.mount: Operation not permitted
May  8 05:14:03 ubuntu systemd[1]: Failed to reset devices.list on /system.slice/systemd-random-seed.service: Operation not permitted
May  8 05:14:03 ubuntu accounts-daemon[108]: started daemon version 0.6.40
May  8 05:14:03 ubuntu systemd[1]: Failed to reset devices.list on /system.slice/systemd-update-utmp.service: Operation not permitted
May  8 05:14:03 ubuntu systemd[1]: Failed to reset devices.list on /system.slice/cron.service: Operation not permitted
May  8 05:14:03 ubuntu systemd[1]: Failed to reset devices.list on /system.slice/dbus.service: Operation not permitted
May  8 05:14:03 ubuntu systemd[1]: Failed to reset devices.list on /system.slice/ufw.service: Operation not permitted
May  8 05:14:03 ubuntu systemd[1]: Failed to reset devices.list on /system.slice/mdadm.service: Operation not permitted
May  8 05:14:03 ubuntu systemd[1]: Failed to reset devices.list on /system.slice/proc-swaps.mount: Operation not permitted
May  8 05:14:03 ubuntu systemd[1]: Failed to reset devices.list on /system.slice/apparmor.service: Operation not permitted
May  8 05:14:03 ubuntu systemd[1]: Failed to reset devices.list on /system.slice/proc-sys-fs-binfmt_misc.mount: Operation not permitted
May  8 05:14:03 ubuntu systemd[1]: Failed to reset devices.list on /system.slice/systemd-logind.service: Operation not permitted
May  8 05:14:03 ubuntu systemd[1]: Failed to reset devices.list on /system.slice/dev-tty.mount: Operation not permitted
May  8 05:14:03 ubuntu systemd[1]: Failed to reset devices.list on /system.slice/var-lib-lxcfs.mount: Operation not permitted
May  8 05:14:03 ubuntu systemd[1]: Failed to reset devices.list on /system.slice/systemd-tmpfiles-setup.service: Operation not permitted
May  8 05:14:03 ubuntu systemd[1]: Failed to reset devices.list on /system.slice/proc-stat.mount: Operation not permitted
May  8 05:14:03 ubuntu systemd[1]: Failed to reset devices.list on /system.slice/lvm2-monitor.service: Operation not permitted
May  8 05:14:03 ubuntu systemd[1]: Failed to reset devices.list on /system.slice/-.mount: Operation not permitted
May  8 05:14:03 ubuntu systemd[1]: Failed to reset devices.list on /system.slice/proc-meminfo.mount: Operation not permitted
May  8 05:14:03 ubuntu systemd[1]: Failed to reset devices.list on /system.slice/systemd-udev-trigger.service: Operation not permitted
May  8 05:14:03 ubuntu systemd[1]: Failed to reset devices.list on /system.slice/accounts-daemon.service: Operation not permitted
May  8 05:14:03 ubuntu systemd[1]: Failed to reset devices.list on /system.slice/pollinate.service: Operation not permitted
May  8 05:14:03 ubuntu systemd[1]: Started LSB: MD monitoring daemon.
May  8 05:14:03 ubuntu systemd[1]: Started Authenticate and Authorize Users to Run Privileged Tasks.
May  8 05:14:03 ubuntu systemd[1]: Started Accounts Service.
May  8 05:14:03 ubuntu pollinate[88]: WARNING: Network communication failed [0]\n  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current#012                                 Dload  Upload   Total   Spent    Left  Speed#012#015  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     005:14:03.631670 * Could not resolve host: entropy.ubuntu.com#01205:14:03.631786 * Closing connection 0#012curl: (6) Could not resolve host: entropy.ubuntu.com
May  8 05:14:03 ubuntu pollinate[88]: May  8 05:14:03 ubuntu <13>May  8 05:14:03 pollinate[88]: WARNING: Network communication failed [0]\n  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
May  8 05:14:03 ubuntu systemd[1]: Started Seed the pseudo random number generator on first boot.
May  8 05:14:03 ubuntu systemd[1]: Failed to reset devices.list on /system.slice/testproc.service: Operation not permitted
May  8 05:14:03 ubuntu systemd[1]: Started Test Process.
May  8 05:14:03 ubuntu systemd[1]: Failed to reset devices.list on /system.slice/mdadm.service: Operation not permitted
May  8 05:14:03 ubuntu testproc[178]: hello
May  8 05:14:03 ubuntu systemd[1]: Started LXD - container startup/shutdown.
May  8 05:14:04 ubuntu systemd[1]: Started Initial cloud-init job (pre-networking).
May  8 05:14:04 ubuntu systemd[1]: Failed to reset devices.list on /system.slice/networking.service: Operation not permitted
May  8 05:14:04 ubuntu systemd[1]: Starting Raise network interfaces...
May  8 05:14:04 ubuntu systemd[1]: Failed to reset devices.list on /system.slice/lxd-containers.service: Operation not permitted
May  8 05:14:05 ubuntu testproc[178]: hello
May  8 05:14:06 ubuntu systemd[1]: Stopping Test Process...
May  8 05:14:06 ubuntu systemd[1]: Stopped Test Process.
May  8 05:14:06 ubuntu systemd[1]: Failed to reset devices.list on /system.slice/testproc.service: Operation not permitted
May  8 05:14:06 ubuntu systemd[1]: Started Test Process.
May  8 05:14:06 ubuntu testproc[279]: hello

Steps to reproduce

Run this script:

#!/bin/bash

CONTAINER=testcon
lxc launch ubuntu:xenial $CONTAINER

lxc exec $CONTAINER -- /bin/bash -c "cat <<-EOF > /etc/systemd/system/testproc.service
[Unit]
Description=Test Process

[Service]
ExecStart=/bin/bash -c 'while true;do echo hello;sleep 2s;done'
Restart=always
RestartSec=10
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=testproc
User=root

[Install]
WantedBy=multi-user.target
EOF"

lxc exec $CONTAINER -- systemctl enable testproc.service
lxc exec $CONTAINER -- systemctl start testproc.service
sleep 3s
printf "\n" && lxc list $CONTAINER
lxc exec $CONTAINER -- systemctl restart testproc
lxc exec $CONTAINER -- cat /var/log/syslog | egrep -i 'fail|error|testproc|start|stop'

[stgraber]

Unprivileged containers cannot modify the devices cgroup configuration. That's a kernel policy not something that LXD enforces itself.

It may be that the new kernel cgroup hierarchy work (cgroupv2) will allow setting additional device restrictions inside unprivileged containers, but right now, the kernel just doesn't let you do it.

[stgraber]

There probably is an argument for making systemd a bit less verbose in such cases, you can file a bug against systemd in Ubuntu (https://launchpad.net/ubuntu/+source/systemd/+filebug) to try and improve things there.

[CalebEverett]

Happy to log it in over there if you think helpful.

[jason-lo]

I have also encountered that issue. Is there a solution already? Or where is the bug listed to improve this so that I can track the status?

[stgraber]

@jason-lo as mentioned in this report, this particular error message isn't fatal at all. If a systemd unit isn't working for you, it's because of something else.

The bug report I requested (which I don't believe was filed), was simply to hide that error message since it is non-fatal and clearly confuses users.

[jason-lo]

@stgraber I think you are right. The error message looks similar, but the root course might be totally different.
I should ask my question here:
I encountered this message when doing this:
snap install --channel=edge --devmode ubuntu-image

and part of the error messages are here:

Sep 12 02:58:14 jk-env-xenial systemd[1]: Failed to reset devices.list on /system.slice/dev-loop1.mount: Operation not permitted
Sep 12 02:58:14 jk-env-xenial systemd[1]: Failed to reset devices.list on /system.slice/apparmor.service: Operation not permitted
Sep 12 02:58:14 jk-env-xenial systemd[1]: Failed to reset devices.list on /system.slice/ssh.service: Operation not permitted
Sep 12 02:58:14 jk-env-xenial systemd[1]: Failed to reset devices.list on /system.slice/rsyslog.service: Operation not permitted
Sep 12 02:58:14 jk-env-xenial systemd[1]: Failed to reset devices.list on /system.slice/iscsid.service: Operation not permitted
Sep 12 02:58:14 jk-env-xenial systemd[1]: Failed to reset devices.list on /system.slice/proc-uptime.mount: Operation not permitted
Sep 12 02:58:14 jk-env-xenial systemd[1]: Failed to reset devices.list on /system.slice/systemd-tmpfiles-setup.service: Operation not permitted
Sep 12 02:58:14 jk-env-xenial systemd[1]: Failed to reset devices.list on /system.slice/polkitd.service: Operation not permitted
Sep 12 02:58:14 jk-env-xenial systemd[1]: Failed to reset devices.list on /system.slice/dev-.lxd\x2dmounts.mount: Operation not permitted
Sep 12 02:58:14 jk-env-xenial systemd[1]: snap-ubuntu\x2dcore-423.mount: Mount process exited, code=exited status=32
Sep 12 02:58:14 jk-env-xenial systemd[1]: Failed to mount Mount unit for ubuntu-core.
-- Subject: Unit snap-ubuntu\x2dcore-423.mount has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit snap-ubuntu\x2dcore-423.mount has failed.
-- 
-- The result is failed.

Is there any solution for this?

[stgraber]

You can't install snaps inside LXD containers right now.

Installing snaps inside LXD containers first requires support for apparmor profile stacking which is still being worked on by the apparmor guys. Once the feature is available in the Ubuntu kernels, LXD will be able to make use of it. That combined with the squashfuse support I contributed to snapd a while back will allow for snaps to be installed inside LXD containers.

We're hoping this will be unblocked soon, but right now we're waiting on a fixed kernel from the apparmor team without any clear ETA from them.

[jason-lo]

Thanks for the reply. At least now I know it is not out of some missing configuration on privileged or mount mapping. It is even better to know this is a plan in schedule, only soon or later.
👍

[gregsifr]

@stgraber I'm running a priveledged container and experiencing Failed to reset devices.list on /system.slice/apparmor.service: Operation not permitted.

To be specific, I executed the following commands:

lxc launch ubuntu: container
lxc config set container security.privileged true
lxc start container
lxc exec container bash

I then install a third party package using sudo however the service failed and upon inspecting sudo journalctl -xe I see the following:

Nov 30 10:03:25 container systemd[1]: snapd.refresh.timer: Adding 1h 54min 34.466397s random time.
Nov 30 10:03:25 container systemd[1]: apt-daily.timer: Adding 4h 3min 38.232571s random time.
Nov 30 10:03:25 container systemd[1]: Failed to reset devices.list on /system.slice/apparmor.service: Operation not permitted
Nov 30 10:03:25 container systemd[1]: Reloading LSB: AppArmor initialization.
-- Subject: Unit apparmor.service has begun reloading its configuration
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit apparmor.service has begun reloading its configuration
Nov 30 10:03:25 container systemd[1]: Failed to reset devices.list on /init.scope: Operation not permitted
Nov 30 10:03:25 container systemd[1]: Failed to reset devices.list on /user.slice: Operation not permitted
Nov 30 10:03:25 container systemd[1]: Failed to reset devices.list on /system.slice/dev-mqueue.mount: Operation not permitted
Nov 30 10:03:25 container systemd[1]: Failed to reset devices.list on /system.slice/ondemand.service: Operation not permitted
Nov 30 10:03:25 container systemd[1]: Failed to reset devices.list on /system.slice/snapd.service: Operation not permitted
Nov 30 10:03:25 container systemd[1]: Failed to reset devices.list on /system.slice/-.mount: Operation not permitted
Nov 30 10:03:25 container systemd[1]: Failed to reset devices.list on /system.slice/cron.service: Operation not permitted
Nov 30 10:03:25 container systemd[1]: Failed to reset devices.list on /system.slice/sys-kernel-debug.mount: Operation not permitted
Nov 30 10:03:25 container systemd[1]: Failed to reset devices.list on /system.slice/rc-local.service: Operation not permitted
Nov 30 10:03:25 container systemd[1]: Failed to reset devices.list on /system.slice/ufw.service: Operation not permitted
Nov 30 10:03:25 container systemd[1]: Failed to reset devices.list on /system.slice/var-lib-lxcfs.mount: Operation not permitted
Nov 30 10:03:25 container systemd[1]: Failed to reset devices.list on /system.slice/resolvconf.service: Operation not permitted
Nov 30 10:03:25 container systemd[1]: Failed to reset devices.list on /system.slice/proc-meminfo.mount: Operation not permitted
Nov 30 10:03:25 container systemd[1]: Failed to reset devices.list on /system.slice/cloud-init-local.service: Operation not permitted
Nov 30 10:03:25 container systemd[1]: Failed to reset devices.list on /system.slice/iscsid.service: Operation not permitted
Nov 30 10:03:25 container systemd[1]: Failed to reset devices.list on /system.slice/accounts-daemon.service: Operation not permitted
Nov 30 10:03:25 container systemd[1]: Failed to reset devices.list on /system.slice/console-getty.service: Operation not permitted
Nov 30 10:03:25 container systemd[1]: Failed to reset devices.list on /system.slice/systemd-journald.service: Operation not permitted
Nov 30 10:03:25 container systemd[1]: Failed to reset devices.list on /system.slice/polkitd.service: Operation not permitted
Nov 30 10:03:25 container systemd[1]: Failed to reset devices.list on /system.slice/systemd-journal-flush.service: Operation not permitted
Nov 30 10:03:25 container systemd[1]: Failed to reset devices.list on /system.slice/dbus.service: Operation not permitted
Nov 30 10:03:25 container systemd[1]: Failed to reset devices.list on /system.slice/system-getty.slice: Operation not permitted
Nov 30 10:03:25 container systemd[1]: Failed to reset devices.list on /system.slice/dev-lxd.mount: Operation not permitted
Nov 30 10:03:25 container systemd[1]: Failed to reset devices.list on /system.slice/system-apport\x2dforward.slice: Operation not permitted
Nov 30 10:03:25 container systemd[1]: Failed to reset devices.list on /system.slice/systemd-tmpfiles-setup.service: Operation not permitted
Nov 30 10:03:25 container systemd[1]: Failed to reset devices.list on /system.slice/dev-hugepages.mount: Operation not permitted
Nov 30 10:03:25 container systemd[1]: Failed to reset devices.list on /system.slice/sys-devices-virtual-net.mount: Operation not permitted
Nov 30 10:03:25 container systemd[1]: Failed to reset devices.list on /system.slice/apport.service: Operation not permitted
Nov 30 10:03:25 container systemd[1]: Failed to reset devices.list on /system.slice/lvm2-monitor.service: Operation not permitted
Nov 30 10:03:25 container systemd[1]: Failed to reset devices.list on /system.slice/dev-fuse.mount: Operation not permitted
Nov 30 10:03:25 container systemd[1]: Failed to reset devices.list on /system.slice/proc-diskstats.mount: Operation not permitted
Nov 30 10:03:25 container systemd[1]: Failed to reset devices.list on /system.slice/cloud-config.service: Operation not permitted
Nov 30 10:03:25 container systemd[1]: Failed to reset devices.list on /system.slice/open-iscsi.service: Operation not permitted
Nov 30 10:03:25 container systemd[1]: Failed to reset devices.list on /system.slice/rsyslog.service: Operation not permitted
Nov 30 10:03:25 container systemd[1]: Failed to reset devices.list on /system.slice/systemd-random-seed.service: Operation not permitted
Nov 30 10:03:25 container systemd[1]: Failed to reset devices.list on /system.slice/proc-uptime.mount: Operation not permitted
Nov 30 10:03:25 container systemd[1]: Failed to reset devices.list on /system.slice/cloud-final.service: Operation not permitted
Nov 30 10:03:25 container systemd[1]: Failed to reset devices.list on /system.slice/proc-sys-net.mount: Operation not permitted
Nov 30 10:03:25 container systemd[1]: Failed to reset devices.list on /system.slice/dev-net-tun.mount: Operation not permitted
Nov 30 10:03:25 container systemd[1]: Failed to reset devices.list on /system.slice/proc-stat.mount: Operation not permitted
Nov 30 10:03:25 container systemd[1]: Failed to reset devices.list on /system.slice/run-user-1000.mount: Operation not permitted
Nov 30 10:03:25 container systemd[1]: Failed to reset devices.list on /system.slice/proc-sys-fs-binfmt_misc.mount: Operation not permitted
Nov 30 10:03:25 container systemd[1]: Failed to reset devices.list on /system.slice/mdadm.service: Operation not permitted
Nov 30 10:03:25 container systemd[1]: Failed to reset devices.list on /system.slice/proc-cpuinfo.mount: Operation not permitted
Nov 30 10:03:25 container systemd[1]: Failed to reset devices.list on /system.slice/lxd-containers.service: Operation not permitted
Nov 30 10:03:25 container systemd[1]: Failed to reset devices.list on /system.slice/sys-fs-fuse-connections.mount: Operation not permitted
Nov 30 10:03:25 container systemd[1]: Failed to reset devices.list on /system.slice/dev-.lxd\x2dmounts.mount: Operation not permitted
Nov 30 10:03:25 container systemd[1]: Failed to reset devices.list on /system.slice/ssh.service: Operation not permitted
Nov 30 10:03:25 container systemd[1]: Failed to reset devices.list on /system.slice/systemd-user-sessions.service: Operation not permitted
Nov 30 10:03:25 container systemd[1]: Failed to reset devices.list on /system.slice/cloud-init.service: Operation not permitted
Nov 30 10:03:25 container systemd[1]: Failed to reset devices.list on /system.slice/sys-kernel-debug-tracing.mount: Operation not permitted
Nov 30 10:03:25 container systemd[1]: Failed to reset devices.list on /system.slice/systemd-logind.service: Operation not permitted
Nov 30 10:03:25 container systemd[1]: Failed to reset devices.list on /system.slice/proc-sysrq\x2dtrigger.mount: Operation not permitted
Nov 30 10:03:25 container systemd[1]: Failed to reset devices.list on /system.slice/networking.service: Operation not permitted
Nov 30 10:03:25 container systemd[1]: Failed to reset devices.list on /system.slice/irqbalance.service: Operation not permitted
Nov 30 10:03:25 container systemd[1]: Failed to reset devices.list on /system.slice/ntp.service: Operation not permitted
Nov 30 10:03:25 container systemd[1]: Failed to reset devices.list on /system.slice/proc-swaps.mount: Operation not permitted
Nov 30 10:03:25 container systemd[1]: Failed to reset devices.list on /system.slice/systemd-update-utmp.service: Operation not permitted
Nov 30 10:03:25 container systemd[1]: Failed to reset devices.list on /system.slice/snapd.firstboot.service: Operation not permitted
Nov 30 10:03:25 container systemd[1]: Failed to reset devices.list on /system.slice/atd.service: Operation not permitted
Nov 30 10:03:25 container apparmor[15611]: /etc/init.d/apparmor: 256: /etc/init.d/apparmor: cannot open /sys/kernel/security/apparmor/.ns_stacked: Permission denied
Nov 30 10:03:25 container apparmor[15611]:  * Not reloading AppArmor in container
Nov 30 10:03:25 container apparmor[15611]:    ...done.
Nov 30 10:03:25 container systemd[1]: Reloaded LSB: AppArmor initialization.
-- Subject: Unit apparmor.service has finished reloading its configuration
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit apparmor.service has finished reloading its configuration
-- 
-- The result is done.

What needs to happen to allow the LXD container this behaviour. To provide further context, I had no issues doing this is privileged LXC containers.

Update:
Upon stopping apparmor on the host using /etc/init.d/apparmor stop, I was able to get past the systemd permission failure however I then ran into other permission problems such as Nov 30 10:41:43 container ntpd[661]: adj_systime: Operation not permitted

Update 2:
Using lxc config set container raw.lxc lxc.aa_profile=unconfined did not resolve the matter.

[gattytto]

lxc.aa_profile: unconfined
lxc.cgroup.devices.allow: a
lxc.cap.drop:
lxc.cgroup.devices.deny:
lxc.mount.auto: proc:rw sys:ro cgroup:ro
lxc.kmsg: 0
lxc.autodev: 1

[Eddie85]

Dear gattytto
Could you please provide the whole profile for it?
I got this error:
lxc 20170920225750.923 ERROR lxc_apparmor - lsm/apparmor.c:apparmor_process_label_set:234 - No such file or directory - failed to change apparmor profile to unconfined lxc.cgroup.devices.allow=a lxc.mount.auto= proc:rw sys:ro cgroup:ro lxc.kmsg= 0 lxc.autodev= 1

This is how my profiles looks like:
...
config:
boot.autostart: "true"
boot.autostart.delay: "30"
boot.autostart.priority: "10"
limits.cpu: "4"
limits.memory: 1024MB
limits.memory.swap: "false"
raw.lxc: lxc.aa_profile = unconfined lxc.cgroup.devices.allow=a lxc.mount.auto=
proc:rw sys:ro cgroup:ro lxc.kmsg= 0 lxc.autodev= 1
security.privileged: "true"
description: ""
.....

[brauner]

This line is wrong. It needs to be:

raw.lxc: |-
  lxc.aa_profile=unconfined
  lxc.cgroup.devices.allow=a
  lxc.mount.auto=proc:rw sys:ro cgroup:ro
  lxc.kmsg=0
  lxc.autodev=1

Also, please note that lxc.kmsg is deprecated as of liblxc 2.1 and lxc.autodev shouldn't need to be set as this is the default anyway. Additionally, cgroup:ro will currently only be respected if cgroup namespaces are not supported by the running kernel. If they are this statement has no effect since the isolation due to cgroup namespaces should be sufficient. However, we might enable this option in future liblxc versions.