[OS X] Derivation fails with sandbox

[eamsden]

Describe the bug

When running a sandboxed build on OS X, a derivation fails with sandbox-exec: pattern serialization length 71710 exceeds maximum (65535)

If you have a problem with a specific package or NixOS,
you probably want to file an issue at https://github.com/NixOS/nixpkgs/issues.

Steps To Reproduce

Create a derivation with a sufficiently large number of inputs, and attempt to build it.

Expected behavior

The derivation builds

nix-env --version output
2.3.7

Additional context

I have done a bit of digging and it seems most likely that this is due to the fact that the OS X sandbox config is created by building a pattern mapping every path in the dependency closure of the derivation to a path in the sandbox individually:

nix/src/libstore/build.cc

Lines 3706 to 3729 in d761485

	/* Our inputs (transitive dependencies and any impurities computed above)
	without file-write* allowed, access() incorrectly returns EPERM
	*/
	sandboxProfile += "(allow file-read* file-write* process-exec\n";
	for (auto & i : dirsInChroot) {
	if (i.first != i.second.source)
	throw Error(
	"can't map '%1%' to '%2%': mismatched impure paths not supported on Darwin",
	i.first, i.second.source);
	string path = i.first;
	struct stat st;
	if (lstat(path.c_str(), &st)) {
	if (i.second.optional && errno == ENOENT)
	continue;
	throw SysError("getting attributes of path '%s", path);
	}
	if (S_ISDIR(st.st_mode))
	sandboxProfile += fmt("\t(subpath \"%s\")\n", path);
	else
	sandboxProfile += fmt("\t(literal \"%s\")\n", path);
	}
	sandboxProfile += ")\n";

[siraben]

For me the derivation that fails when I try to build with sandbox on macOS is my home-manager config.

these derivations will be built:
  /nix/store/xfsgnyj7m6pmhjf1qzxawwvsd0nhppif-home-manager-path.drv
  /nix/store/r5s9k5m0apmlk6hn1bysb98pha3aikcm-activation-script.drv
  /nix/store/yclx6cws1qc2gyxn0j1pv70nr802ck69-home-manager-generation.drv
building '/nix/store/xfsgnyj7m6pmhjf1qzxawwvsd0nhppif-home-manager-path.drv'...
sandbox-exec: pattern serialization length 87092 exceeds maximum (65535)
builder for '/nix/store/xfsgnyj7m6pmhjf1qzxawwvsd0nhppif-home-manager-path.drv' failed with exit code 65
cannot build derivation '/nix/store/yclx6cws1qc2gyxn0j1pv70nr802ck69-home-manager-generation.drv': 1 dependencies couldn't be built
error: build of '/nix/store/yclx6cws1qc2gyxn0j1pv70nr802ck69-home-manager-generation.drv' failed

[bbbbbailey]

Bumping because I just experienced this issue with Nix 2.4 (2.4pre20201201_5a6ddb3) when building a home-manager config via nix flakes.

error: --- Error ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- nix
builder for '/nix/store/v75p8mffw7l7pg2pxdwqjfzx89rks8b2-home-manager-path.drv' failed with exit code 65; last 1 log lines:
  sandbox-exec: pattern serialization length 66460 exceeds maximum (65535)
error: --- Error ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- nix
1 dependencies of derivation '/nix/store/8hpb5b7l19y2267dn11cbi4ilx67ribk-home-manager-generation.drv' failed to build
error: --- Error ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- nix
1 dependencies of derivation '/nix/store/pjjh7a238hwjkvkvhmbmazrzk41jsyz5-user-environment.drv' failed to build
error: --- Error ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- nix
1 dependencies of derivation '/nix/store/vskz1pxb41sv66irwx2689f39b8hnkld-activation-bbuscarino.drv' failed to build
error: --- Error ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- nix
1 dependencies of derivation '/nix/store/ak7vwn1z37ymjv192iyl1v2fc5ibkwnx-darwin-system-21.03.20210209.74335e0+darwin4.dca3806.drv' failed to build

[hlolli]

hashtag me2

> nix run .#main-client-build
 warning: unknown setting 'allowUnfree'
warning: Git tree '/Users/hlodversigurdsson/Documents/testapp' is dirty
warning: unknown setting 'allowUnfree'
error: builder for '/nix/store/99swavz43gwjd27w7zx5a2q5gxbrmbxk-node-dependencies-_at_testapp_slash_main-client-1.0.0.drv' failed with exit code 65;
       last 1 log lines:
       > sandbox-exec: pattern serialization length 89274 exceeds maximum (65535)
       For full logs, run 'nix log /nix/store/99swavz43gwjd27w7zx5a2q5gxbrmbxk-node-dependencies-_at_testapp_slash_main-client-1.0.0.drv'.
error: 1 dependencies of derivation '/nix/store/mgi4b6a83rxqmzkv0603mia72fccwj1c-node-shell-_at_testapp_slash_main-client-1.0.0.drv' failed to build
> nix log /nix/store/99swavz43gwjd27w7zx5a2q5gxbrmbxk-node-dependencies-_at_testapp_slash_main-client-1.0.0.drv
warning: unknown setting 'allowUnfree'
sandbox-exec: pattern serialization length 89274 exceeds maximum (65535)
> nix --version
warning: unknown setting 'allowUnfree'
nix (Nix) 2.4pre20210503_6d2553a

[hlolli]

So it seems the generated sandbox profile has an s-expression exceeding the limit

when running with nix --debug

executing builder '/nix/store/ldfws5lqyz0irmb3bcvnjawyhsva38xq-bash-4.4-p23/bin/bash'
sandbox setup: Generated sandbox profile:
sandbox setup: (version 1)
sandbox setup: (deny default (with no-log))
sandbox setup: (import "sandbox-defaults.sb")
sandbox setup: (allow file-read* file-write* process-exec
sandbox setup:      (subpath "/nix/store/8xywf3whgq08045glmv99v2bgxmhvh00-node-dependencies-_at_testapp_slash_main-client-1.0.0")
sandbox setup: )
sandbox setup: (allow file-read* file-write* process-exec
sandbox setup:        (subpath "/System/Library/Frameworks")
sandbox setup:  (subpath "/System/Library/PrivateFrameworks")
sandbox setup:   (literal "/bin/bash")
sandbox setup:   (literal "/bin/sh")
sandbox setup:     (literal "/dev/random")
sandbox setup:         (literal "/dev/urandom")
sandbox setup:        (literal "/dev/zero")
sandbox setup:   (literal "/nix/store/006jl3mw0xiqwrfnmm0dn6zxkpaqkdar-http2-wrapper-1.0.3.tgz")
sandbox setup:         (literal "/nix/store/011f5148hdsm6yznj4rl30j4jhzlhrb6-is-buffer-1.1.6.tgz")
sandbox setup:     (literal "/nix/store/026dnivlzqrsdrf8pkc47hh5f3arm9kk-final-form-4.20.2.tgz")
....

I'm assuming splitting these statements into smaller blocks will fix this.

[domenkozar]

@toonn something to put on the list for https://opencollective.com/nix-macos

[domenkozar]

@toonn I went through Nix issues concerning macOS sandboxing and this is the only one left reported to fix.

It needs research when the limit kicks in and what's the workaround (maybe we have to use import and split the expression if it becomes too long).

[hlolli]

@toonn I went through Nix issues concerning macOS sandboxing and this is the only one left reported to fix.

It needs research when the limit kicks in and what's the workaround (maybe we have to use import and split the expression if it becomes too long).

I'd not be surprised that the limit is 65535, but that's hard to determine from counting the packages, because I think there's one sandbox item entry for every nix store path. I did read a bit about the sandbox specification from osx developer manual, and started attempting to write a fix by batching these in multiple files (which is totally possible, importing .sb files). But my lack of c++ knowledge, and especially modern c++ as is in the nix repo, got the better of me.

sandbox-exec: pattern serialization length 89274 exceeds maximum (65535)

[stale]

I marked this as stale due to inactivity. → More info

[thefloweringash]

Having a sandbox on macOS is still a worthy goal.

and started attempting to write a fix by batching these in multiple files (which is totally possible, importing .sb files).

For the record, were you able to confirm that importing from multiple files is sufficient to bypass the limit? I tried experimenting a little, and I think the serialization limit is at a lower level.

I made two files with ~800 store paths in them, and a top-level file to import them.

• If I include either file by itself, sandbox-exec works.
• If I include both, the computed size is 97830 and this produces the pattern serialization length 97830 exceeds maximum (65535) error.
• If I combine the files, the error message is exactly the same, down to the same computed size.
• If I include all three files (slice 1, slice 2, combined), the error message is still the same. Which to me suggests some kind of unique representation for each path, and a format not related to the input.

If it's not possible to encode the strict nix sandboxing scheme, could we still have a sandbox that prevents access to things outside of the store path? I think this could still help a lot with purity.

[lilyball]

I just ran into this myself.

What I find especially interesting/frustrating is that I ran into it while performing a nix flake check after enabling additional services on one of my outputs.nixosConfigurations. That output isn't for the current platform, so I've configured Nix with a remote builder for aarch64-linux. It looks like it's failing on a trivial wrapper /nix/store/bbblnlsl5hwxbv4s145cf7vh8wyxqnqp-deploy-rs-check-activate.drv which is an aarch64-darwin derivation that just runs a simple shell script over an aarch64-linux derivation /nix/store/x499w8xz2zrjniz1lb5y3cmrnm3zi8si-activatable-nixos-system-pyra-22.05.20220428.6766fb6.drv. So it's generating a massive sandbox profile with 1213 literal/subpath entries all so it can just check for the presence of 2 files.

I really want to keep the macOS sandbox. If there's no way to get around this by splitting things up, I wonder if we could do something like generate regexes that match multiple paths with a single pattern. I don't know what regex dialect it is but I've never seen one that doesn't have alternations. At the very we could remove /nix/store duplication by merging multiple paths together. We could also truncate each (subpath "/nix/store/…") entry after the store hash, such as (regex #"^/nix/store/zs08rlx67p6vjjbk4y3c75ylr9r56xms-.*") instead of (subpath "/nix/store/zs08rlx67p6vjjbk4y3c75ylr9r56xms-iana-etc-20211124"), as that would save some characters that are presumably contributing to the serialization size as well.

Assuming this works, there is a limit to how much space could be saved this way (and I question whether constructing a massive regex with alternations would affect performance; if it internally converts literals and subpaths into regexes already then there should be no difference), but anything to increase the size would help. And if we can accurately predict the serialization size we could also have a degraded fallback mode where we start allowing extra files if it means writing fewer rules in order to stay under the size. Jumping all the way to (subpath "/nix/store") if we can't reasonably stay in the limit is still significantly better than abandoning the sandbox.

[lilyball]

Incidentally it sounds like older versions of macOS had a command sandbox-simplify, it doesn't exist anymore but if anyone has a sufficiently old version of macOS that still has it I'm wondering if it does anything to a large Nix-generated sandbox profile.

[2xsaiko]

Just hit this as well on a pkgs.linkFarm with ~3000 entries:

nix build 'git+https://git.dblsaiko.net/nix-mc?rev=3e53544c6bb8758d3ff5357c7ba6dee616ad1ac4#minecraftPackages.aarch64-darwin.1_19_1.assets'

[n8henrie]

If anyone is looking to reproduce the error manually, there is likely a better way, but here are my steps:

1. darwin-rebuild --flake . switch gives me:

error: builder for '/nix/store/8c1h3pifzs3xpfazfvcqbikgd5mf61j9-darwin-system-23.05.20230908.3fbd2a2+darwin4.511177f.drv' failed with exit code 65;                                                                                         │
│       last 1 log lines:                                                                                                                                                                                                                    │
│       > sandbox-exec: pattern serialization length 75123 exceeds maximum (65535)                                                                                                                                                           │
│       For full logs, run 'nix log /nix/store/8c1h3pifzs3xpfazfvcqbikgd5mf61j9-darwin-system-23.05.20230908.3fbd2a2+darwin4.511177f.drv'.                  

2. nix build --debug that derivation:

$ nix build --debug \
    /nix/store/8c1h3pifzs3xpfazfvcqbikgd5mf61j9-darwin-system-23.05.20230908.3fbd2a2+darwin4.511177f.drv \
    &> sandbox.sb

3. Manually edit sandbox.sb to strip out the scheme code, starting with (version 1)
4. Using src/libstore/build/local-derivation-goal.cc as an example:

$ sandbox-exec -f ./sandbox.sb -D _GLOBAL_TMP_DIR=/tmp/foo echo fake                
sandbox-exec: pattern serialization length 75123 exceeds maximum (65535)                                                                                                                        

(Would likely replace echo fake with nix build the_derivation.drv once the sandbox error is fixed).

[n8henrie]

Trying @lilyball's suggestion:

We could also truncate each (subpath "/nix/store/…") entry after the store hash, such as (regex #"^/nix/store/zs08rlx67p6vjjbk4y3c75ylr9r56xms-.*") instead of (subpath "/nix/store/zs08rlx67p6vjjbk4y3c75ylr9r56xms-iana-etc-20211124"), as that would save some characters that are presumably contributing to the serialization size as well.

I used the following nvim replacement:

:%s:subpath "\(/nix/store/[^-]*\)-.*":regex #"\1-.*": 

Running the below line count pre and post gives the same result, so it doesn't look like any additional paths can be removed (I thought perhaps there would be multiple binaries from the same package listed):

$ sed -e 's/^[[:blank:]]*//' -e 's/[[:blank:]]*$//' < sandbox.sb  | sort -u | wc -l                                                                                                                               
1268

Afterwards, I get farther(?), so it seems like this is probably a valid temporizing measure:

$ sandbox-exec \
    -f ./sandbox.sb \
    -D _GLOBAL_TMP_DIR=/tmp/foo \
    $(type -p nix) build /nix/store/8c1h3pifzs3xpfazfvcqbikgd5mf61j9-darwin-system-23.05.20230908.3fbd2a2+darwin4.511177f.drv
sandbox-exec: execvp() of '/run/current-system/sw/bin/nix' failed: Operation not permitted      

Rereading the debug output, looks like I'm using the wrong approach for building, but this doesn't work either (even though I have execute permissions for this bash):

$ sandbox-exec \
    -f ./sandbox.sb \
    -D _GLOBAL_TMP_DIR=/tmp/foo \
    /nix/store/7avcas47b0g25c9bk7m4xb7p5p636qs7-bash-5.2-p15/bin/bash \
    -e /nix/store/6xg259477c90a229xwmb53pdfkn6ig3g-default-builder.sh
sandbox-exec: execvp() of '/nix/store/7avcas47b0g25c9bk7m4xb7p5p636qs7-bash-5.2-p15/bin/bash' failed: Operation not permitted

[jsoo1]

Is it possible that this could be fixed with one more layer of indirection?

Suppose we were to create a chroot and mount in all the store dependencies, could we reduce the number of store paths in the sandbox profile to only the chroot directory?

Edit: I suppose the lack of bind mounts would make this impossible.

[lilyball]

I do wonder how much mileage we could get by doing things like having a single regex that matches multiple store paths (as an extension on the idea of "use a regex to match up to the hash") to save even more characters, but I don't know how the "pattern serialization length" is calculated and it's plausible that it's already basically joining all paths with the same perms into a single pattern internally.

[jsoo1]

I found a gist that had some notes on experiments done for this problem which seemed to suggest the patterns were all reduced prior to sending to the kernel. Let me see if I can dig it up...

https://gist.github.com/rrbutani/16c7c66202ed47b2a5dd8cb5e916fb0a

Their experiments seemed to suggest that sandbox-simplify was subsumed by sandbox-exec.

I do wonder what might be done with sandbox.h after some perusal. Though I have the hardest time digging up apple's headers (if indeed it is public).

[n8henrie]

That gist is interesting. Mentions:

NOTE this restriction only seems to exist if you use (version 1); (version 2) and (version 3) happily accept more atoms in subexpressions

I quickly patched nix to use version 3 and to remove the no-log statement that it chokes on, but it results in

 > sandbox-exec: execvp() of '/nix/store/49wn01k9yikhjlxc1ym5b6civ29zz3gv-bash-5.1-p16/bin/bash' failed: Operation not permitted

when I try to darwin-rebuild switch. My DDG-fu is failing me on finding much info on version 2 or version 3 of the sandbox file format. Anyone know of additional resources?

[jsoo1]

The sandbox format is famously undocumented. As far as I know the best resource is the pdf linked in the bottom of the gist.

[lilyball]

I do wonder what might be done with sandbox.h after some perusal. Though I have the hardest time digging up apple's headers (if indeed it is public).

The file exists in the macOS SDK (as provided by Xcode), but it really doesn't have very much in it, just the declarations of the stuff documented in man sandbox_init.

[jsoo1]

The file exists in the macOS SDK (as provided by Xcode), but it really doesn't have very much in it, just the declarations of the stuff documented in man sandbox_init.

I see. Welp. I'm definitely at a loss. Maybe there could be something insome posts about reverse engineering sandbox/private.h.

[szlend]

How about something along the lines of:

1. Optimize subpath match rules to only include the hash (^/nix/store/zs08rlx67p6vjjbk4y3c75ylr9r56xms-.*)
2. Now we have a fixed length per store path so we can make an educated guess of how many paths the sandbox can handle in the worst case.
3. We can throw an error (or warning) at evaluation time if the derivation includes too many paths, making it easy to find all instances of where this can happen.
4. We introduce some sort of annotation e.g. _sandboxProfileIncludeFullStore = true which simplifies all subpath matching rules to just ^/nix/store/.* and apply it where necessary.

Builders like buildEnv and symlinkJoin could include this annotation by default. They're the most likely to run into this issue. Or they could turn it on dynamically based on the number of paths included.

This gets us one step closer to enabling the sandbox on by default. It doesn't have to be perfect. Any step we can make towards the sandbox being more usable is a worthy step imo.

[n8henrie]

I made a little patch to tinker on this, including the below:

diff --git a/src/libstore/unix/build/local-derivation-goal.cc b/src/libstore/unix/build/local-derivation-goal.cc
index 72125cb82..0ff9739d0 100644
--- a/src/libstore/unix/build/local-derivation-goal.cc
+++ b/src/libstore/unix/build/local-derivation-goal.cc
@@ -2012,7 +2012,14 @@ void LocalDerivationGoal::runChild()
 #if __APPLE__
         else {
             /* This has to appear before import statements. */
-            std::string sandboxProfile = "(version 1)\n";
+            debug("I should be outputting version 3 here");
+            std::string sandboxProfile = "(version 3)\n";
+            sandboxProfile +=
+                #include "system.sb"
+                ;
+            sandboxProfile +=
+                #include "dyld-support.sb"
+                ;
 
             if (useChroot) {
 
@@ -2045,11 +2052,7 @@ void LocalDerivationGoal::runChild()
                 }
 
                 /* Violations will go to the syslog if you set this. Unfortunately the destination does not appear to be configurable */
-                if (settings.darwinLogSandboxViolations) {
-                    sandboxProfile += "(deny default)\n";
-                } else {
-                    sandboxProfile += "(deny default (with no-log))\n";
-                }
+                sandboxProfile += "(deny default)\n";
 
                 sandboxProfile +=
                     #include "sandbox-defaults.sb"

However, when I run nix build and then use ./result/bin/nix --debug build /path/to/my/flake#darwinConfigurations.mymachine.config.system.build.toplevel, I still see it outputing a version 1 sandbox (and it omits my I should be outputting version 3 here). What is that? The sandbox profile is generated by nix (and not something within my system flake itself), right?

I assume it can't be caching some kind of build artifact (because this is nix) but I tried a make clean and rm -rf the cache directory listed in .gitignore anyway, no change in behavior.

[toonn]

@n8henrie, isn't libstore used by the daemon rather than nix the command? You'd have to replace your daemon, not just invoke the command based on that change.

[n8henrie]

Thanks, that's what I was wondering, just confused as the src/libstore code seems to live in this repo.

Unfortunately it seems like a dead end; in the interim I copied a "serialization too long" sandbox file changed it to version 3 and it dies with the same error.

[n8henrie]

Following @lilyball from above:

We could also truncate each (subpath "/nix/store/…") entry after the store hash, such as (regex #"^/nix/store/zs08rlx67p6vjjbk4y3c75ylr9r56xms-.*") instead of (subpath "/nix/store/zs08rlx67p6vjjbk4y3c75ylr9r56xms-iana-etc-20211124"), as that would save some characters that are presumably contributing to the serialization size as well.

Might it also be possible / make sense to save the repetition of subpath "/nix/store by first limiting to only paths in /nix/store, then subsequently by / + the hash? I suppose there would be the possibility of a path /nix/store/not-what-you-wanted/zs08rlx67p6vjjbk4y3c75ylr9r56xms... but that seems fairly unlikely.

[n8henrie]

Not a solution to the bigger issue here, but tonight I finally sat down and looked at the sandbox profiles in the nix build --debug output and discovered that there are 3 sandbox builds; the first two were both under 250 lines, but the one for my home-manager stuff was 1197 lines!

I whipped together a little script to try to see if there was a common theme that was making it so big:

$ awk -F'"' '/subpath/ { print $2 }' sandbox-three.sb |
    awk -F- '{ for (i=2;i<=NF;i++) { print $i }}' |
    grep -v -e '^[0-9.]\+$' -e 'rev=' |
    sort |
    uniq -c |
    sort -n |
    tail

     14 c
     14 lib
     16 man
     32 perl5.38.2
     39 framework
     40 apple
     51 python3.11
    278 treesitter
    278 vimplugin
    559 grammar

559 lines of treesitter grammars (due to using pkgs.vimPlugins.nvim-treesitter.withAllGrammars)!

Curiously, I pared this down to about 60 languages that I ever use, and it only brought my sandbox file down to 1067 lines (but seems to have dramatically reduces the number of grammar lines):

  21 man
     28 lib
     32 perl5.38.2
     40 framework
     42 apple
     46 ruby3.1
     62 treesitter
     62 vimplugin
     70 python3.11
    127 grammar

Regardless, it made enough of a difference to avoid the sandbox error for now (instead of just using --option sandbox false like I've been doing for the past many months).