Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cannot use nix --store runtime which is selected by default #98

Open
orzFly opened this issue Apr 15, 2024 · 9 comments
Open

cannot use nix --store runtime which is selected by default #98

orzFly opened this issue Apr 15, 2024 · 9 comments

Comments

@orzFly
Copy link

orzFly commented Apr 15, 2024

Using nix-portable v011

$ nix-portable nix run nixpkgs#htop
error: setting up a private mount namespace: Operation not permitted

By the way, bwrap runs fine.

$ NP_RUNTIME=bwrap nix-portable nix run nixpkgs#htop

Diagnostics

$ zgrep CONFIG_USER_NS /proc/config.gz
CONFIG_USER_NS=y
CONFIG_USER_NS_UNPRIVILEGED=y

$ cat /proc/sys/kernel/unprivileged_userns_clone
1

$ unshare -r -n echo YES
YES

$ uname -a
Linux mipha 6.7.8-arch1-1 #1 SMP PREEMPT_DYNAMIC Sun, 03 Mar 2024 00:30:36 +0000 x86_64 GNU/Linux
$ NP_DEBUG=2 nix-portable nix run nixpkgs#htop 2>&1 | tee np_debug.log

np_debug.log

$ NP_DEBUG=2 strace nix-portable nix run nixpkgs#htop 2>&1 | tee strace.log

strace.log

$ grep unshare strace.log
unshare(CLONE_NEWNS|CLONE_NEWUSER)      = 0
unshare(CLONE_NEWNS|CLONE_NEWUSER)      = -1 EPERM (Operation not permitted)
unshare(CLONE_NEWNS)                    = -1 EPERM (Operation not permitted)

$ strace unshare -r -m echo YES 2>&1 | egrep 'unshare|YES'
execve("/usr/bin/unshare", ["unshare", "-r", "-m", "echo", "YES"], 0x7ffc6e302bd0 /* 79 vars */) = 0
unshare(CLONE_NEWNS|CLONE_NEWUSER)      = 0
execve("/home/user/perl5/bin/echo", ["echo", "YES"], 0x7ffdf5a71d28 /* 79 vars */) = -1 ENOENT (No such file or directory)
execve("/opt/asdf-vm/bin/echo", ["echo", "YES"], 0x7ffdf5a71d28 /* 79 vars */) = -1 ENOENT (No such file or directory)
execve("/home/user/sbin/echo", ["echo", "YES"], 0x7ffdf5a71d28 /* 79 vars */) = -1 ENOENT (No such file or directory)
execve("/home/user/bin/echo", ["echo", "YES"], 0x7ffdf5a71d28 /* 79 vars */) = -1 ENOENT (No such file or directory)
execve("/usr/local/sbin/echo", ["echo", "YES"], 0x7ffdf5a71d28 /* 79 vars */) = -1 ENOENT (No such file or directory)
execve("/usr/local/bin/echo", ["echo", "YES"], 0x7ffdf5a71d28 /* 79 vars */) = -1 ENOENT (No such file or directory)
execve("/usr/bin/echo", ["echo", "YES"], 0x7ffdf5a71d28 /* 79 vars */) = 0
write(1, "YES\n", 4YES
@DavHau
Copy link
Owner

DavHau commented Apr 15, 2024

Thanks for the detailed report. I'm not sure at the moment why the nix local store doesn't work but bubblewrap does.

Anyways, v012 should fix the issue with nix-portable not falling back to bwrap automatically. Can you confirm that it works now?

@orzFly
Copy link
Author

orzFly commented Apr 15, 2024

Anyways, v012 should fix the issue with nix-portable not falling back to bwrap automatically. Can you confirm that it works now?

Still no luck 😿 same error

I updated nix-portable, nuked ~/.nix-portable and tried:

$ nix-portable nix run nixpkgs#htop
error: setting up a private mount namespace: Operation not permitted

Removing last_auto_runtime for a completer log

$ rm /home/user/.nix-portable/conf/last_auto_runtime
$ NP_DEBUG=2 nix-portable nix run nixpkgs#htop >np_debug_v012.log 2>&1

np_debug_v012.log

@orzFly
Copy link
Author

orzFly commented Apr 15, 2024

Worth noting in np_debug_v012.log:

+ /home/user/.nix-portable/bin/nix --store /home/user/.nix-portable/tmp/__store shell -f /home/user/.nix-portable/mini-drv.nix -c /home/user/.nix-portable/bin/nix store add-file --store /home/user/.nix-portable/tmp/__store /home/user/.nix-portable/tmp/testfile
this derivation will be built:
  /nix/store/mbwnbda8s0f5rlpprm2km8m69gkq94nm-foo.drv
building '/nix/store/mbwnbda8s0f5rlpprm2km8m69gkq94nm-foo.drv'...
+ chmod -R +w /home/user/.nix-portable/tmp/__store
+ rm -r /home/user/.nix-portable/tmp/__store
+ debug 'nix --store works on this system -> will use nix as runtime'
+ echo nix --store works on this system '->' will use nix as runtime
nix --store works on this system -> will use nix as runtime

I am not sure why the second call to unshare is failed. I am not familiar with unshare-y things.

$ strace nix-portable nix run nixpkgs#htop 2>&1 | grep unshare
unshare(CLONE_NEWNS|CLONE_NEWUSER)      = 0
unshare(CLONE_NEWNS|CLONE_NEWUSER)      = -1 EPERM (Operation not permitted)
unshare(CLONE_NEWNS)                    = -1 EPERM (Operation not permitted)

This command can success. I am not sure if this helps.

$ strace /usr/bin/unshare -r -m /usr/bin/unshare -r -m echo YES 2>&1 | grep unshare
execve("/usr/bin/unshare", ["/usr/bin/unshare", "-r", "-m", "/usr/bin/unshare", "-r", "-m", "echo", "YES"], 0x7ffcfe206b78 /* 79 vars */) = 0
unshare(CLONE_NEWNS|CLONE_NEWUSER)      = 0
execve("/usr/bin/unshare", ["/usr/bin/unshare", "-r", "-m", "echo", "YES"], 0x7ffce18616e0 /* 79 vars */) = 0
unshare(CLONE_NEWNS|CLONE_NEWUSER)      = 0

@DavHau
Copy link
Owner

DavHau commented Apr 15, 2024

Which version of which distro are you running? Maybe I can add a test for it to the CI pipeline.

@orzFly
Copy link
Author

orzFly commented Apr 15, 2024

Arch Linux. Up-to-dated (with an old kernel for ZFS).


However, this is a little embarrassing to say... Actually, I am new to nix. Due to my oversatisfaction to bundling applications with nix-portable as single executable, I just installed Nix (multiple users install)! With the real /nix directory, things has changed.

$ uname -a
Linux mipha 6.7.8-arch1-1 #1 SMP PREEMPT_DYNAMIC Sun, 03 Mar 2024 00:30:36 +0000 x86_64 GNU/Linux

$ sudo zfs unmount rpool/nix
$ mount | grep /nix
$ ls -alh /nix
total 10K
drwxr-xr-x  2 root root  2 Apr 16 02:10 .
drwxr-xr-x 18 root root 24 Apr 16 02:10 ..
$ NP_RUNTIME=nix NP_DEBUG=2 strace ./nix-portable-x86_64 nix run nixpkgs#hello 2>&1 | egrep unshare\|error:\|Hello
unshare(CLONE_NEWNS|CLONE_NEWUSER)      = 0
unshare(CLONE_NEWNS|CLONE_NEWUSER)      = -1 EPERM (Operation not permitted)
unshare(CLONE_NEWNS)                    = -1 EPERM (Operation not permitted)
write(2, "error: setting up a private moun"..., 69error: setting up a private mount namespace: Operation not permitted

$ sudo zfs mount rpool/nix
$ mount | grep /nix
rpool/nix on /nix type zfs (rw,relatime,xattr,posixacl,casesensitive)
$ ls -alh /nix
total 1.1M
drwxr-xr-x   4 root root      4 Apr 16 02:11 .
drwxr-xr-x  18 root root     24 Apr 16 02:10 ..
drwxrwxr-t 461 root nixbld 4.2K Apr 16 02:27 store
drwxr-xr-x   4 root root      4 Apr 16 02:14 var
$ NP_RUNTIME=nix NP_DEBUG=2 strace ./nix-portable-x86_64 nix run nixpkgs#hello 2>&1 | egrep unshare\|error:\|Hello
unshare(CLONE_NEWNS|CLONE_NEWUSER)      = 0
unshare(CLONE_NEWNS|CLONE_NEWUSER)      = 0
write(1, "Hello, world!\n", 14Hello, world!

@orzFly
Copy link
Author

orzFly commented Apr 15, 2024

Reproducible with ArchLinux Netboot Release 2024.04.01. Here's the rough session transcript:

# uname -a
Linux archiso 6.8.2-arch2-1 #1 SMP PREEMPT_DYNAMIC Thu, 28 Mar 2024 17:06:35 +0000 x86_64 GNU/Linux
# mount -o remount,size=50% /run/archiso/cowspace
# pacman -Sy wget strace
# useradd user
# mkdir /home/user
# chown -R user:user /home/user
# sudo -uuser -Hs
$ wget https://github.com/DavHau/nix-portable/releases/download/v012/nix-portable-x86_64
$ chmod +x nix-portable-x86_64
$ ./nix-portable-x86_64 nix run nixpkgs\#hello                          # let it download and fail
$ NP_RUNTIME=nix NP_DEBUG=2 strace ./nix-portable-x86_64 nix run nixpkgs\#hello 2>&1 | grep -E unshare\|error:\|Hello
unshare(CLONE_NEWNS|CLONE_NEWUSER)      = 0
unshare(CLONE_NEWNS|CLONE_NEWUSER)      = -1 EPERM (Operation not permitted)
unshare(CLONE_NEWNS)                    = -1 EPERM (Operation not permitted)
write(2, "error: setting up a private moun"..., 69error: setting up a private mount namespace: Operation not permitted
$ NP_RUNTIME=bwrap ./nix-portable-x86_64 nix run nixpkgs\#hello
Hello, world!

@orzFly
Copy link
Author

orzFly commented Apr 16, 2024

Maybe I can add a test for it to the CI pipeline.

Maybe you should try adding test for nix run nixpkgs#hello to the CI pipeline...

Producible on Debian GNU/Linux 11 (bullseye)
orzfly@media-sorter:~$ NP_RUNTIME=bwrap ./nix-portable nix-shell -p hello --run hello
Hello, world!

orzfly@media-sorter:~$ NP_RUNTIME=bwrap ./nix-portable nix run nixpkgs#hello
Hello, world!

orzfly@media-sorter:~$ NP_RUNTIME=nix ./nix-portable nix-shell -p hello --run hello
Hello, world!

orzfly@media-sorter:~$ NP_RUNTIME=nix ./nix-portable nix run nixpkgs#hello
error: setting up a private mount namespace: Operation not permitted

orzfly@media-sorter:~$ uname -a
Linux media-sorter 6.1.0-10-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.37-1 (2023-07-03) x86_64 GNU/Linux

orzfly@media-sorter:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description:    Debian GNU/Linux 11 (bullseye)
Release:        11
Codename:       bullseye

@ccornix
Copy link

ccornix commented Apr 17, 2024

Reproducible on Debian GNU/Linux 12 (bookworm) as well, using v012.

ccornix@debian:~$ NP_RUNTIME=bwrap ./nix-portable nix run nixpkgs#hello
Hello, world!

ccornix@debian:~$ NP_RUNTIME=nix ./nix-portable nix run nixpkgs#hello
error: setting up a private mount namespace: Operation not permitted

ccornix@debian:~$ uname -a
Linux debian 6.1.0-20-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.85-1 (2024-04-11) x86_64 GNU/Linux

ccornix@debian:~$ lsb_release -a
No LSB modules are available.
Distributor ID:	Debian
Description:	Debian GNU/Linux 12 (bookworm)
Release:	12
Codename:	bookworm

@chaserhkj
Copy link

Considering NixOS/nix#6853 is present and probably will be around for some time as well. Maybe we should just switch to bwrap runtime as default in nix-portable ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants