[CA-drvs] random errors when invoking nix build in parallel

[fogti]

Describe the bug

Sometimes I run both 'nix build ...#...' as a trusted user, and 'nixos-rebuild ... switch' as root. If both start to build in a short amount of time, and try to build the seemingly same derivation, both builds crash hard. Two workers seem to somewhat step on each other or hit some ulimit (maybe some worker uid is reused?, or the sandbox parameters are messed up in-between?)
It looks like the sandboxing gets corrupted.

example output from one build (I didn't save the other build output, sorry)

gimp> gcc: fatal error: cannot execute 'as': execvp: No such file or directory
gimp> compilation terminated.
gimp> make[3]: Leaving directory '/build/gimp-2.10.24/libgimp'
gimp> make[2]: Leaving directory '/build/gimp-2.10.24/libgimp'
gimp> make[1]: Leaving directory '/build/gimp-2.10.24'
gimp> /nix/store/4gvspv40pgvigigykx07nrr34ljgpbdk-stdenv-linux/setup: line 101: /dev/null: No such file or directory
gimp> /nix/store/4gvspv40pgvigigykx07nrr34ljgpbdk-stdenv-linux/setup: line 82: /dev/null: No such file or directory
gimp> /nix/store/4gvspv40pgvigigykx07nrr34ljgpbdk-stdenv-linux/setup: line 84: /dev/null: No such file or directory
error: builder for '/nix/store/pj2dx839d6iyj8r865wdzzs9c6g3ww0a-gimp-2.10.24.drv' failed with exit code 2;
       last 10 log lines:
       > make[3]: *** [Makefile:1400: gimpdrawablecolor_pdb.lo] Error 1
       > make[3]: Leaving directory '/build/gimp-2.10.24/libgimp'
       > make[2]: *** [Makefile:1182: all] Error 2
       > make[2]: Leaving directory '/build/gimp-2.10.24/libgimp'
       > make[1]: *** [Makefile:852: all-recursive] Error 1
       > make[1]: Leaving directory '/build/gimp-2.10.24'
       > make: *** [Makefile:753: all] Error 2
       > /nix/store/4gvspv40pgvigigykx07nrr34ljgpbdk-stdenv-linux/setup: line 101: /dev/null: No such file or directory
       > /nix/store/4gvspv40pgvigigykx07nrr34ljgpbdk-stdenv-linux/setup: line 82: /dev/null: No such file or directory
       > /nix/store/4gvspv40pgvigigykx07nrr34ljgpbdk-stdenv-linux/setup: line 84: /dev/null: No such file or directory
       For full logs, run 'nix log /nix/store/pj2dx839d6iyj8r865wdzzs9c6g3ww0a-gimp-2.10.24.drv'.
error (ignored): error: opening directory '/nix/store/pj2dx839d6iyj8r865wdzzs9c6g3ww0a-gimp-2.10.24.drv.chroot/nix/store/s6irxlhycj1h0czp9ksjjzfd9y231jw4-perl5.32.1-HTTP-Negotiate-6.01': Bad file descriptor
error: opening file '/nix/store/qskswsv9n0ccqyf5nd927aqdyf7g8p24-ostree-2021.1.drv.chroot/etc/passwd': No such file or directory

Steps To Reproduce

Non-deterministic, as far as I know, but really annoying.

Expected behavior

Neither build should fail and they shouldn't interfere with each other.

nix-env --version output
nix-env (Nix) 2.4pre20210715_db4d4cf

Additional context

happened with the following nix.conf:

# WARNING: this file is generated from the nix.* options in
# your NixOS configuration, typically
# /etc/nixos/configuration.nix.  Do not edit it!
build-users-group = nixbld
max-jobs = 3
cores = 4
sandbox = true
extra-sandbox-paths = 
substituters = 
trusted-substituters = 
trusted-public-keys = cache.ngi0.nixos.org-1:KqH5CBLNSyX184S9BKZJo1LxrxJ9ltnY2uAs5c/f1MA= cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
auto-optimise-store = true
require-sigs = true
trusted-users = zseri
allowed-users = *
builders =

system-features = nixos-test benchmark big-parallel kvm
sandbox-fallback = false

experimental-features = nix-command ca-derivations ca-references flakes
keep-outputs = true
keep-derivations = true
flake-registry = /nix/store/8smmncch59idkjk3f9vmxhzzmsjiavrb-flake-empty.json
builders-use-substitutes = true

[thufschmitt]

Mh, I guess that’s linked to this.

Does #5030 fix it?