"operation not permitted" opening a tun/tap device

[mediocregopher]

Hi! I'm trying to run nebula from within a nix-bundle'd binary (not using the AppImage builder yet) and am getting the following error:

ERRO[0000] Failed to get a tun/tap device                error="operation not permitted"

When I run the entrypoint script directly, not using nix-bundle, it works fine. Afaict everything about the bundled binary is working correctly, it only seems to fail at making a new tun/tap device. This error occurs when running the binary via sudo and from a real root shell.

I have also managed to smuggle in my outer PATH environment variable so that it's set within the entrypoint script, but that didn't affect anything. Similarly when I called ${pkgs.iproute2}/bin/ip link from the entrypoint script it correctly printed out all my network devices.

You can see the code in nebula where it's creating the tun device here. Afaict it's doing everything through system calls, and not via some exec call to an external binary, so I don't think there's any issue here with the process not being able to access some outside resource due to the chroot.

The only thing I can think is that the chroot is preventing write access to /dev in some way? It appears to support read operations on /dev (based on my iproute2 test), but maybe the mapping code in the chroot utility doesn't account for writes. Does that make sense as a theory? My C++ isn't very good so I'm not very confident in it.

In any case thanks for making such a useful tool! 🙏

[mediocregopher]

Ok I came up with a much easier repro case:

test.nix

{
    pkgs ? import (fetchTarball "https://github.com/NixOS/nixpkgs/archive/d50923ab2d308a1ddb21594ba6ae064cab65d8ae.tar.gz") {},
    system ? builtins.currentSystem,
    nixBundleSrc ? (fetchTarball "https://github.com/matthewbauer/nix-bundle/archive/8e396533ef8f3e8a769037476824d668409b4a74.tar.gz"),
}:
    {
        nixBundle = (import "${nixBundleSrc}/release.nix") {};

        testScript  = (pkgs.writeScriptBin "test-script" ''
            #!${pkgs.stdenv.shell}
            set -e
            ${pkgs.iproute2}/bin/ip tuntap add mode tun dev tuntest0
        '');
    }

Then:

> nix-build -A nixBundle test.nix

> ./result/bin/nix-bundle '((import ./test.nix) {}).testScript' /bin/test-script
these derivations will be built:
  /nix/store/509liv8hrsdaxm0knl1q8ngpmgmw9m89-test-script.drv
  /nix/store/pf36b8z3ck9fsvfasxybn3fw88jlrhli-startup.drv
  /nix/store/5k8y7c940is6lci4v9bwrkk5425r0wvp-maketar.drv
  /nix/store/4cr87ncc9lqdfid163kj92dgl98mcblz-arx.drv
building '/nix/store/509liv8hrsdaxm0knl1q8ngpmgmw9m89-test-script.drv'...
building '/nix/store/pf36b8z3ck9fsvfasxybn3fw88jlrhli-startup.drv'...
building '/nix/store/5k8y7c940is6lci4v9bwrkk5425r0wvp-maketar.drv'...
tar: Removing leading `/' from member names
building '/nix/store/4cr87ncc9lqdfid163kj92dgl98mcblz-arx.drv'...
Nix bundle created at test-script.

> sudo ./test-script
ioctl(TUNSETIFF): Operation not permitted

Running test-script outside of the bundle adds a tun device just fine. Hope this helps!

[danobi]

I hate to bump old issues but I am running into this as well. I've been working on bundling bpftrace (bpftrace/bpftrace#2595) and anything that requires root seems to fail:

$ nix bundle
$ sudo ./bpftrace -e 'BEGIN { print("hi") }'
ERROR: Unknown error -1: couldn't set RLIMIT_MEMLOCK for bpftrace. If your program is not loading, you can try "ulimit -l 8192" to fix the problem
Attaching 1 probe...
ERROR: failed to create  map: Operation not permitted
Creation of the required BPF maps has failed.
Make sure you have all the required permissions and are not confined (e.g. like
snapcraft does). `dmesg` will likely have useful output for further troubleshooting