systemd "Failed to set up mount namespacing"

[avsdev-cw]

This is less of a bug, more of an "Information for others with the same problem".

Recently I've been upgrading all our containers from debian Buster to debian Bullseye and a whole load of systemd services started failing, for example:

May 31 10:02:27 hostyhost systemd[474]: logrotate.service: Failed to set up mount namespacing: /run/systemd/unit-root/proc: Permission denied
May 31 10:02:27 hostyhost systemd[474]: logrotate.service: Failed at step NAMESPACE spawning /usr/sbin/logrotate: Permission denied

After plenty of googling, it seems that the consensus is "Allow nesting in the container" which for various reasons I don't want to do. After my own tinkering I finally found that in the new containers there is a file (/etc/systemd/system-generators/lxc). Copying this file (chmod 755) fixes all the service issues.

For anyone interested in the file, I've placed it in a gist: https://gist.github.com/avsdev-cw/37de3dfbe0b369ba60efac7c5f680bb0

The file is generated by lxc/distrobuilder (https://github.com/lxc/distrobuilder/blob/db120181f69811c4b9aabe7e0842d9100dc13ad9/distrobuilder/main.go#L540-L766)

If anyone with inside knowledge wants to expand, please do!

[torstenfohrer]

Hi, distrobuilder creates this file (embedded in main.go).

A additional question from me, do you use unprivileged or privileged containers? I'm stuck with same isse with unprivileged containers!

[avsdev-cw]

Thanks for the tip on the source. Link for others is: https://github.com/lxc/distrobuilder/blob/db120181f69811c4b9aabe7e0842d9100dc13ad9/distrobuilder/main.go#L540-L766

I use both privileged and unprivileged (mostly the latter).

Personally, I loathe systemd - it's become a monolithic disaster which claims to make things simpler and more functional and fails to do either. Previously 10 seconds of googling and you would have an answer in a beautiful 2 or 3 lines; now 10 minutes of reading multiple systemd docs and scraping the internet for examples and you can achieve half of what you set out to do with a bunch of new files and 10's more lines than necessary! A long way of saying: I don't even pretend to know why the file is necessary or even chunks of what its doing or why its doing those things.

[torstenfohrer]

Thanks, for answer, are you perhaps not using apparmor?

I use as host a Ubuntu 20.04, and containers with a apparmor profile. If i patch this systemd service generator to disable ProtectHome/Systemd and so it works (one way to do it). Or otherwise running unprivileged container without apparmor profile.

[avsdev-cw]

I am using apparmor. Our hosts are all debian-minimal based hosts.

[BartVB]

@avsdev-cw You are a lifesaver, thanks!

Just spend a couple of hours figuring out what went wrong with LXC after I upgraded my host from Debian Bullseye to Debian Bookworm. For some reason services like systemd-logind got stuck in Failed at step NAMESPACE.

Did find some other people running into that, but the 'solution' that was recommended was enabling nesting which seemed rather heavy handed.

I was trying to figure out why a fresh Debian Bullseye container worked (main symptom was that ssh took ages to connect) and my 'old' containers didn't.

/etc/systemd/system-generators/lxc was the answer! Thanks a lot!

Now the question is; how should I have handled this? There is lxc-update-config but that only touches the config file. As far as I know the only way tot obtain the generators file is by manually copying that from a fresh container? Also no idea how I should have learned about the need for this file? And are there other 'magic' things that distrobuilder takes care of that are missing in my current containers?

[kongomongo]

sorry for possibly sounding stupid, but I am in a container and have no control of the host, can I fix this by generating the file under /etc/systemd/system-generators/lxc or is that only possible on the host?

[stgraber]

Yeah, you can add/update the generator to your container and systemd should run it on reboot.

[mihalicyn]

Hi @avsdev-cw,

thanks for sharing your experience with others.

Enabling nesting for unprivileged containers is totally fine and we are thinking about making it a default in the future.
For privileged containers, you are right, enabling nesting creates additional security risks (making container escape-able, basically). We also have an idea to deprecate privileged containers in the future... (btw, if you have a usecase for them -> please, share it so we can think about how to achive the same without a privileged mode).

I'm closing this, cause there is nothing to do, really.

[paulocoghi]

@mihalicyn when enabling nesting for unprivileged containers by default, the only important thing is to still allow to disable it via config file. This is most probably what you will do, but I'm just giving my 2c just in case.