containers: reload fails with user namespace enabled

[uvNikita]

Issue description

Reloading of containers fails with:

org.freedesktop.DBus.Error.NoReply: Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken.

I think it's connected to #57083 because adding -U to nsenter command seem to fix an error (not 100% sure yet).

Steps to reproduce

configuration.nix:

containers.test = {
  extraFlags = [ "-U" ];
  config = {};
};

# systemctl start test
# systemctl reload test # hangs for a few seconds
# journalctl -M test
# systemctl status container@test.service
...
Process: 13923 ExecReload=/nix/store/agr8dhkpx995hfbgwdx43if3j0vwl242-reload-container (code=exited, status=2)
...

Technical details

 - system: `"x86_64-linux"`
 - host os: `Linux 4.14.104, NixOS, 18.09pre-git (Jellyfish)`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.1.3`
 - channels(root): `"nixos-server-18.09.1834.9d608a6f592, nixos-18.03"`
 - nixpkgs: `/var/src/nixpkgs`

[uvNikita]

Further tests shows that adding -U argument to nsenter indeed fixes the original error, but the container still can't reload configuration with these errors in journalctl:

Mar 08 23:05:56 server systemd[1]: Reloading Container 'test'.
Mar 08 23:05:56 server container test[18863]: WARNING: bad ownership on /nix/var/nix/profiles/per-user/, should be 0
Mar 08 23:05:56 server container test[18863]: activating the configuration...
Mar 08 23:05:56 server container test[18863]: setting up /etc...
Mar 08 23:05:56 server container test[18863]: mount: /dev: permission denied.
Mar 08 23:05:56 server container test[18863]: mount: /dev/pts: permission denied.
Mar 08 23:05:56 server container test[18863]: mount: /dev/shm: permission denied.
Mar 08 23:05:56 server container test[18863]: mount: /proc: permission denied.
Mar 08 23:05:56 server container test[18863]: mount: /run: permission denied.
Mar 08 23:05:56 server container test[18863]: mount: /run/keys: permission denied.
Mar 08 23:05:56 server container test[18863]: mount: /run/wrappers: permission denied.
Mar 08 23:05:56 server container test[18863]: Activation script snippet 'specialfs' failed (32)
Mar 08 23:05:56 server container test[18863]: chmod: changing permissions of '/var/empty': Operation not permitted
Mar 08 23:05:56 server container test[18863]: chown: changing ownership of '/var/empty': Operation not permitted
Mar 08 23:05:56 server container test[18863]: Activation script snippet 'var' failed (1)
Mar 08 23:05:57 server container test[18863]: ln: failed to create symbolic link '/nix/var/nix/gcroots/current-system': Permission denied
Mar 08 23:05:57 server container test[18863]: setting up tmpfiles
Mar 08 23:05:57 server systemd[1]: container@test.service: Control process exited, code=exited status=2

The reason for these errors, I think, I that /nix has owner nobody:nogroup inside the container (because of user namespace). Manually stopping and starting the container works as expected though.

Here is the code that bind mounts nix store inside the container:

nixpkgs/nixos/modules/virtualisation/containers.nix

Lines 123 to 132 in 37694c8

	exec ${config.systemd.package}/bin/systemd-nspawn \
	--keep-unit \
	-M "$INSTANCE" -D "$root" $extraFlags \
	$EXTRA_NSPAWN_FLAGS \
	--notify-ready=yes \
	--bind-ro=/nix/store \
	--bind-ro=/nix/var/nix/db \
	--bind-ro=/nix/var/nix/daemon-socket \
	--bind="/nix/var/nix/profiles/per-container/$INSTANCE:/nix/var/nix/profiles" \
	--bind="/nix/var/nix/gcroots/per-container/$INSTANCE:/nix/var/nix/gcroots" \

[uvNikita]

Considering that systemd-nspawn --bind option doesn't work well together with -U [1], does it mean that nixos containers can't be unprivileged?

[arianvp]

had issues with enabling user namespaces when working on my own take on nixos-container and if I recall correctly it had to do with the fact that mounting stuff from the host into a user namespace doesn't work well or had issues with permissions. See comment I left here in my own rewrite of nixos-container

https://github.com/arianvp/nixos-stuff/blob/293aa02546fa87ef8cedc4943e9fa2655857f027/modules/containers-v2.nix#L71

Basically the way we mount nix store and gcroots doesn't work well with user namespaces

So I came to the same conclusion that unpriveleged NixOS containers isn't possible

[uvNikita]

@arianvp thanks for the input! It's unfortunate that this is the case. During my tests so far everything works correctly as long as you stop and start containers when configuration changes instead of reloading them. So I'm considering if that's an acceptable compromise for me to get containers with user namespace :)

[arianvp]

If you're not interested in nix commands working inside the container, and only are interested in declarative containers , then unpriveleged containers should work I think

[uvNikita]

Yes, that's my use case exactly -- fully declarative containers, so read-only store should be enough.

I checked the module you linked and it looks pretty good, are you planning to create a PR to push it upstream at some point? :) As far as I can tell, it's much easier to customize it (e.g. add or remove bind mounts and change other parameters of nspawn) than in the current containers module.

[stale]

Thank you for your contributions.

This has been automatically marked as stale because it has had no activity for 180 days.

If this is still important to you, we ask that you leave a comment below. Your comment can be as simple as "still important to me". This lets people see that at least one person still cares about this. Someone will have to do this at most twice a year if there is no other activity.

Here are suggestions that might help resolve this more quickly:

1. Search for maintainers and people that previously touched the related code and @ mention them in a comment.
2. Ask on the NixOS Discourse.
3. Ask on the #nixos channel on irc.freenode.net.