apparmor profile breaks systemd PrivateNetwork=yes

[gibmat]

Required information

• Distribution: Debian
• Distribution version: bookworm
• lxc version: 5.0.2

Issue description

It has been discovered that lxc containers with the default apparmor profile are preventing systemd services with PrivateNetwork=yes from starting. There is a bug in Debian that I've been able to reproduce with the steps below; another bug has been reported against LXD which I think has the same root cause.

On the host, the following message appears in the kernel logs when the service fails to start:

[ 2594.539830] audit: type=1400 audit(1692491371.352:50): apparmor="DENIED" operation="file_lock" profile="lxc-bookworm_</var/lib/lxc>" pid=16113 comm="(true)" family="unix" sock_type="dgram" protocol=0 requested_mask="send"
[ 2594.539851] audit: type=1400 audit(1692491371.352:51): apparmor="DENIED" operation="file_lock" profile="lxc-bookworm_</var/lib/lxc>" pid=16113 comm="(true)" family="unix" sock_type="dgram" protocol=0 requested_mask="send"

I am aware of #820, but as that's five years old I'm hoping there is some other reason why this isn't working properly.

I have tried to modify the generated apparmor profile snippet as defined in src/lxc/lsm/apparmor.c's AA_PROFILE_UNIX_SOCKETS with unix (send) and even just unix, but that didn't seem to help. Admittedly, I'm not super experienced with apparmor, so maybe that's not the correct way to allow the access that is currently being denied within the container.

Steps to reproduce

This is easy to reproduce in a container with the default apparmor profile enabled. From the Debian bug report:

root@bookworm:/# cat > /lib/systemd/system/test.service <<EOF
[Unit]
Description=A test
[Service]
Type=oneshot
ExecStart=/bin/true
PrivateNetwork=yes
EOF
root@bookworm:/# systemctl daemon-reload
root@bookworm:/# systemctl start test.service
Job for test.service failed because the control process exited with error code.
See "systemctl status test.service" and "journalctl -xeu test.service" for details.
root@bookworm:/# systemctl status test.service
× test.service - A test
     Loaded: loaded (/lib/systemd/system/test.service; static)
     Active: failed (Result: exit-code) since Sun 2023-08-20 02:22:37 UTC; 1min 28s ago
    Process: 138 ExecStart=/bin/true (code=exited, status=225/NETWORK)
   Main PID: 138 (code=exited, status=225/NETWORK)
        CPU: 1ms

Aug 20 02:22:37 bookworm (true)[138]: test.service: Failed at step NETWORK spawning /bin/true: Permission denied
Aug 20 02:22:37 bookworm systemd[1]: Starting test.service - A test...
Aug 20 02:22:37 bookworm systemd[1]: test.service: Main process exited, code=exited, status=225/NETWORK
Aug 20 02:22:37 bookworm systemd[1]: test.service: Failed with result 'exit-code'.
Aug 20 02:22:37 bookworm systemd[1]: Failed to start test.service - A test.

Known workarounds

• Disable apparmor entirely for the container (not ideal)
• Set PrivateNetwork=no in affected service files (requires manual editing of configuration files)

[stgraber]

The workaround in our images is to have Privatenetwork=no on systemd >= 254.
That's been in our images for a few weeks now so hopefully new containers created from the images using lxc-download will just behave.

The real fix is running an updated apparmor along with the change in LXC profile that we merged a few days ago.
The catch there is that if you run that profile on a system that doesn't have the fixed apparmor, then you introduce security vulnerabilities that would allow escalation to system-wide root privileges and container escape (from priv containers anyway).

Closing as there's nothing actionable at this stage. We have already merged the apparmor policy change that would fix that and we've already been shipping with a workaround in our images for weeks now.

[mbiebl]

I'm running into this issue as well as it made the systemd autopkgtest fail on Debian stable and eventually arrived at this bug report after some digging.
See https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1050256

The real fix is running an updated apparmor along with the change in LXC profile that we merged a few days ago.

I wonder what can be done about this in Debian stable.
Do you have links to the relevant patches?
With "updated apparmor", do you mean AA in the kernel or the actual "apparmor" package?

[stgraber]

@mbiebl it was an apparmor parser bug which @mihalicyn fixed a little while back.
The AppArmor issue is CVE-2016-1585 which was fixed in AppArmor 3.0.10 with subsequent fixes in 3.0.11 and 3.0.12.

So you need to either get the apparmor userspace updated to the 3.0.12 bugfix release or have the content of 3.0.10, 3.0.11 and 3.0.12 applied to your current apparmor version.

Once that's done, then it should be safe to apply 890de07 to LXC.

[mbiebl]

@stgraber ok, the issue I'm seeing is a different issue then, because upgrading apparmor to 3.0.12 and applying the above commit on top of lxc 5.0.2 did not help.

The only thing that helped so far was upgrading the kernel from 6.1.38 (bookworm) to trixie (6.4.11).

The apparmor denial log messages are exactly the same, which is why I thought it's the same issue.

[gibmat]

Maybe this is a different issue... in a clean minimal install of bookworm (apparmor 3.0.8, lxc 5.0.2, kernel 6.1.38) I get the error initially reported. However, if I install kernel 6.4.4 from bookworm-backports, then starting a service with PrivateNetwork=yes works just fine.

[mbiebl]

@gibmat I think you and I are seeing the same issue.

What I was referring to is @stgraber 's reply who mentioned that upgrading lxc and apparmor would help fix the issue (which it doesn't)

[gibmat]

Yes, that's what I meant -- sorry for any confusion!