New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

systemd-networkd.service in unprivileged lxc guest not working since systemd v 240 #2778

Open
n8v8R opened this Issue Jan 10, 2019 · 14 comments

Comments

2 participants
@n8v8R
Copy link

n8v8R commented Jan 10, 2019

@n8v8R n8v8R changed the title systemd-networkd.service in unprivileged lxc guest not working since systems 240 systemd-networkd.service in unprivileged lxc guest not working since systems v 240 Jan 10, 2019

@n8v8R n8v8R changed the title systemd-networkd.service in unprivileged lxc guest not working since systems v 240 systemd-networkd.service in unprivileged lxc guest not working since systed v 240 Jan 10, 2019

@n8v8R n8v8R changed the title systemd-networkd.service in unprivileged lxc guest not working since systed v 240 systemd-networkd.service in unprivileged lxc guest not working since systemd v 240 Jan 10, 2019

@Blub

This comment has been minimized.

Copy link
Member

Blub commented Jan 10, 2019

I suspect you're getting log entries such as:
audit: type=1400 audit(1547119806.282:61): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-414_</var/lib/lxc>" name="/" pid=18681 comm="(networkd)" flags="rw, rslave"

particularly the DENIED, operation=mount, flags=rw, rslave parts, judging by the error message in the bug entry you linked to.

While we'd like to allow such mounts we cannot do so until the apparmor_parser is fixed to handle them correctly.

@n8v8R

This comment has been minimized.

Copy link

n8v8R commented Jan 10, 2019

@Blub No such entries in either host or guest. In latter

systemd-networkd.service: Failed to set up mount namespacing: Permission denied
systemd-networkd.service: Failed at step NAMESPACE spawning /usr/lib/systemd/systemd-networkd: Permission denied

@Blub

This comment has been minimized.

Copy link
Member

Blub commented Jan 10, 2019

Curious. Do you generally get apparmor audit messages in your host's journal? That's at least what I get when trying to reproduce it, and the service's journal also has the Failed at step NAMESPACE... error in it...

@n8v8R

This comment has been minimized.

Copy link

n8v8R commented Jan 10, 2019

Uhm, my bad (on the host syslog-ng is handling logs and not journald ...) , on the host indeed

audit: type=1400 audit(1547125168.853:722): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-container-default-cgns" name="/" pid=8426 comm="(networkd)" flags="rw, rslave"

@n8v8R

This comment has been minimized.

Copy link

n8v8R commented Jan 10, 2019

While we'd like to allow such mounts we cannot do so until the apparmor_parser is fixed to handle them correctly.

Who is supposed to fix that, I am afraid that is not clear to me but suppose that are the AA people at ubuntu and thus requires another bug report (now the 4th then)?

https://bugs.launchpad.net/ubuntu/+source/apparmor/+bug/1811248

@n8v8R

This comment has been minimized.

Copy link

n8v8R commented Jan 11, 2019

@Blub would you be so kind to elaborate what

such mounts

are? The AA crew seems to be unclear about that:

I don't know if AppArmor's log message is sufficient to reconstruct the actual mount() syscall that the process has performed -- and I don't know if the extra parameters that may be in the syscall are important or not.

@Blub

This comment has been minimized.

Copy link
Member

Blub commented Jan 11, 2019

What systemd wants to do is the equivalent of executing mount --make-rslave / on the commandline. The syscall from systemd specifically AFAICT is: mount(NULL, "/", NULL, MS_REC|MS_SLAVE, NULL);
As for the AppArmor profile rule, see https://github.com/lxc/lxc/blob/master/config/apparmor/abstractions/container-base.in#L94

I've pinged jjohansen from the AppArmor devs on irc about it and am hoping he's gonna find the time to dig into this soon.

@n8v8R

This comment has been minimized.

Copy link

n8v8R commented Jan 11, 2019

Thank you! 👍

Shall we close the issue here then, perhaps also on systemd, since it all seems to be bearing down to AA. Or is there anything left to do in the lxc code once AA got patched other than un-commenting those lines?

@Blub

This comment has been minimized.

Copy link
Member

Blub commented Jan 11, 2019

We can keep it open as a reminder to track the update of our apparmor profile. Not sure what systemd wants to do - it's definitely not a bug on their end.

@Blub Blub added the Blocked label Jan 11, 2019

@Blub Blub self-assigned this Jan 11, 2019

@n8v8R

This comment has been minimized.

Copy link

n8v8R commented Jan 16, 2019

@Blub Have you heard back from the AppArmor devs? Asking because the issue seems to be accelerating/cascading to the extent that that the lxc arch linux guest is now entirely dead ☹️

https://bugs.archlinux.org/task/61428

@n8v8R

This comment has been minimized.

Copy link

n8v8R commented Jan 16, 2019

@Blub

While we'd like to allow such mounts we cannot do so until the apparmor_parser is fixed to handle them correctly.

In my simpleton logic stopping/disabling AppArmor on the host

/etc/init.d/apparmor status

● apparmor.service - AppArmor initialization
Loaded: loaded (/lib/systemd/system/apparmor.service; disabled; vendor preset: enabled)
Active: inactive (dead)

then should not invoke the issue since apparmor_parser is supposedly removed from the equation. But it turns out the issue still prevails.

Which now begs the question of who is producing if it is not AppArmor?

audit: type=1400 audit(1547650543.083:21): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-container-default-cgns" name="/" pid=3850 comm="(ostnamed)" flags="rw, rslave"

Logical explanation would be LXC and which would make sense in light of https://github.com/lxc/lxc/blob/master/config/apparmor/abstractions/container-base.in#L94

Which then would make it an LXC issue rather than an AppArmor one.


Since disabling AppArmor on the host does not provide a viable workaround I have to hold any further updates for Arch Linux guests, in order to prevent them from being bricked ☹️ , until LXC supports these type of mounts, considering that the other workaround of partial updates in Arch Linux is not viable either

arch linux partial updates

@n8v8R

This comment has been minimized.

Copy link

n8v8R commented Jan 16, 2019

As a consequence any new Arch Linux guest on a Ubuntu host will fail connectivity if created from a recent image via -t download, e.g. as currently offered 20190116_01:27

@Blub

This comment has been minimized.

Copy link
Member

Blub commented Jan 17, 2019

lxc sets the apparmor profile manually, you can use lxc.apparmor.profile = unchanged to disable that functionality, but I do not recommend that.

If you insist on a workaround, you can add mount options=(rw,make-rslave), to /etc/apparmor.d/abstractions/lxc/container-base, which is the rule we want, but currently equivalent to allowing all mounts. And I'd rather have that part fixed first on the apparmor_parser side (because it's been that way for far too long already, and is only part of the issues we're facing currently...)

@n8v8R

This comment has been minimized.

Copy link

n8v8R commented Jan 17, 2019

@Blub Thanks for the feedback 👍

Well, it is not insisting but the choices I am left with are neither desirable.

  • hold the updates in the arch linux containers until this issue is fixed.
  • set lxc.apparmor.profile = unchanged
  • set mount options=(rw,make-rslave)

Probably holding the updates for a while is the least comprising one, at least for a short while. But who knows when this will be sorted as the AppArmor dev team seems rather unperturbed.
And that only works for existing guests, of which some got bricked already beyond recovery, but not new guests since the latest guest image comes already loaded with systemd v240.x upon inception.

Likely that the majority of users deploy guests that are homogeneous with the host and rather not mix like I do. But at some point Ubuntu is likely to update systemd in their branch repo to v 240 as well and I am wondering what that will be causing then even in more homogeneous nodes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment