Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Recursive Nix support #3205

Merged
merged 5 commits into from Dec 2, 2019
Merged

Recursive Nix support #3205

merged 5 commits into from Dec 2, 2019

Conversation

@edolstra
Copy link
Member

edolstra commented Nov 4, 2019

This allows Nix builders to call Nix to build derivations, with some limitations.

Example:

let nixpkgs = fetchTarball channel:nixos-18.03; in
    
with import <nixpkgs> {};
    
runCommand "foo"
  {
    buildInputs = [ nix jq ];
    NIX_PATH = "nixpkgs=${nixpkgs}";
  }
  ''
    hello=$(nix-build -E '(import <nixpkgs> {}).hello.overrideDerivation (args: { name = "hello-3.5"; })')
    
    $hello/bin/hello
    
    mkdir -p $out/bin
    ln -s $hello/bin/hello $out/bin/hello
    
    nix path-info -r --json $hello | jq .
  ''

This derivation makes a recursive Nix call to build GNU Hello and symlinks it from its $out, i.e.

# ll ./result/bin/
lrwxrwxrwx 1 root root 63 Jan  1  1970 hello -> /nix/store/s0awxrs71gickhaqdwxl506hzccb30y5-hello-3.5/bin/hello
    
# nix-store -qR ./result
/nix/store/hwwqshlmazzjzj7yhrkyjydxamvvkfd3-glibc-2.26-131
/nix/store/s0awxrs71gickhaqdwxl506hzccb30y5-hello-3.5
/nix/store/sgmvvyw8vhfqdqb619bxkcpfn9lvd8ss-foo

This is implemented as follows:

  • Before running the outer builder, Nix creates a Unix domain socket .nix-socket in the builder's temporary directory and sets $NIX_REMOTE to point to it. It starts a thread to process connections to this socket. (Thus you don't need to have nix-daemon running.)

  • The daemon thread uses a wrapper store (RestrictedStore) to keep track of paths added through recursive Nix calls, to implement some restrictions (see below), and to do some censorship (e.g. for
    purity, queryPathInfo() won't return impure information such as signatures and timestamps).

  • After the build finishes, the output paths are scanned for references to the paths added through recursive Nix calls (in addition to the inputs closure). Thus, in the example above, $out has a reference to $hello.

The main restriction on recursive Nix calls is that they cannot do arbitrary substitutions. For example, doing

nix-store -r /nix/store/kmwd1hq55akdb9sc7l3finr175dajlby-hello-2.10

is forbidden unless /nix/store/kmwd... is in the inputs closure or previously built by a recursive Nix call. This is to prevent irreproducible derivations that have hidden dependencies on substituters or the current store contents. Building a derivation is fine, however, and Nix will use substitutes if available. In other words, the builder has to present proof that it knows how to build a desired store path from scratch by constructing a derivation graph for that path.

Probably we should also disallow instantiating/building fixed-output derivations (specifically, those that access the network, but currently we have no way to mark fixed-output derivations that don't
access the network). Otherwise sandboxed derivations can bypass sandbox restrictions and access the network.

When sandboxing is enabled, we make paths appear in the sandbox of the builder by entering the mount namespace of the builder and bind-mounting each path. This is tricky because we do a pivot_root() in the builder to change the root directory of its mount namespace, and thus the host /nix/store is not visible in the mount namespace of the builder. To get around this, just before doing pivot_root(), we branch a second mount namespace that shares its /nix/store mountpoint
with the parent.

Recursive Nix currently doesn't work on macOS in sandboxed mode (because we can't change the sandbox policy of a running build) and on Linux in non-root mode (because setns() barfs).

This PR also adds some ccache-like functionality to Nix's makefiles that wraps GCC calls in Nix derivations to enable caching and remote builds. This requires recursive Nix when you want to do this inside a Nix build.

Implements #13.

@zimbatm

This comment has been minimized.

Copy link
Member

zimbatm commented Nov 4, 2019

Assuming that the inner build is quite large. How would the build be distributed to the same set of remote builders as the outer build?

One thing I was wondering is, if it would make sense for the inner build to just return a drv file in $out instead, and then let the outer scheduler update its build plan accordingly. This would be a bit closer to IFD but where the evaluation happens in a builder instead of all in the client.

@Ericson2314

This comment has been minimized.

Copy link
Member

Ericson2314 commented Nov 4, 2019

@edolstra I am worry about exposing nix-build-in-nix-build before we do nix-instantiate-in-nix-build (my RFC). Your commit messages do mention putting behind a feature flag, and if we ban it in Nxpkgs initially that alleviates my worries. But, let me lay out those worries.

In short, while I think nix-build-in-nix-build is a decent last to retrofit existing shody stuff, it's never the way anything should strive to work. nix-build-in-nix-build is worse because:

  • Timing is observable: derivations can observe how long nested nix-builds take, and therefore whether the thing they want to build was built before. This weakens our purity/caching. Even if we aren't building adversarial stuff that tries to exploit this, it still could could painful, hard to repoduce bugs.

  • "By default", it's sequential: you have to wait for the last nix-build to finish in your script, or waste effort writing your own parallelization code. With "ret cont" you don't need to put in effort to get better scheduling.

  • Resource usage: Derivations that idle waiting for nested nix-build waste space. I call nix-instantiate-innix-build "ret cont" because compared to this you are morally serializing your continuation in the "sucessor" derivation. That successor derivation will almost surely be a lot smaller.

I don't want to be in a position where people write a bunch of stuff that uses nix-build-in-nix-build because of this PR, and then no one has the energy to rewrite it for nix-instantiate-in-nix-build. Even worse than the code is everybody getting excited about this, and then learning a bunch of other stuff---I'll admit ret-cont is weirder up front and more of an "unlearning" step. I'd like to avoid that tech debt and culture whiplash.

@edolstra

This comment has been minimized.

Copy link
Member Author

edolstra commented Nov 5, 2019

I've made an initial version of a nix-ccache flake: https://github.com/edolstra/nix-ccache. It provides a wrapper around gcc/g++ that executes the compilation of the preprocessed source in a recursive nix-build call.

edolstra added 5 commits Oct 2, 2018
This allows Nix builders to call Nix to build derivations, with some
limitations.

Example:

  let nixpkgs = fetchTarball channel:nixos-18.03; in

  with import <nixpkgs> {};

  runCommand "foo"
    {
      buildInputs = [ nix jq ];
      NIX_PATH = "nixpkgs=${nixpkgs}";
    }
    ''
      hello=$(nix-build -E '(import <nixpkgs> {}).hello.overrideDerivation (args: { name = "hello-3.5"; })')

      $hello/bin/hello

      mkdir -p $out/bin
      ln -s $hello/bin/hello $out/bin/hello

      nix path-info -r --json $hello | jq .
    ''

This derivation makes a recursive Nix call to build GNU Hello and
symlinks it from its $out, i.e.

  # ll ./result/bin/
  lrwxrwxrwx 1 root root 63 Jan  1  1970 hello -> /nix/store/s0awxrs71gickhaqdwxl506hzccb30y5-hello-3.5/bin/hello

  # nix-store -qR ./result
  /nix/store/hwwqshlmazzjzj7yhrkyjydxamvvkfd3-glibc-2.26-131
  /nix/store/s0awxrs71gickhaqdwxl506hzccb30y5-hello-3.5
  /nix/store/sgmvvyw8vhfqdqb619bxkcpfn9lvd8ss-foo

This is implemented as follows:

* Before running the outer builder, Nix creates a Unix domain socket
  '.nix-socket' in the builder's temporary directory and sets
  $NIX_REMOTE to point to it. It starts a thread to process
  connections to this socket. (Thus you don't need to have nix-daemon
  running.)

* The daemon thread uses a wrapper store (RestrictedStore) to keep
  track of paths added through recursive Nix calls, to implement some
  restrictions (see below), and to do some censorship (e.g. for
  purity, queryPathInfo() won't return impure information such as
  signatures and timestamps).

* After the build finishes, the output paths are scanned for
  references to the paths added through recursive Nix calls (in
  addition to the inputs closure). Thus, in the example above, $out
  has a reference to $hello.

The main restriction on recursive Nix calls is that they cannot do
arbitrary substitutions. For example, doing

  nix-store -r /nix/store/kmwd1hq55akdb9sc7l3finr175dajlby-hello-2.10

is forbidden unless /nix/store/kmwd... is in the inputs closure or
previously built by a recursive Nix call. This is to prevent
irreproducible derivations that have hidden dependencies on
substituters or the current store contents. Building a derivation is
fine, however, and Nix will use substitutes if available. In other
words, the builder has to present proof that it knows how to build a
desired store path from scratch by constructing a derivation graph for
that path.

Probably we should also disallow instantiating/building fixed-output
derivations (specifically, those that access the network, but
currently we have no way to mark fixed-output derivations that don't
access the network). Otherwise sandboxed derivations can bypass
sandbox restrictions and access the network.

When sandboxing is enabled, we make paths appear in the sandbox of the
builder by entering the mount namespace of the builder and
bind-mounting each path. This is tricky because we do a pivot_root()
in the builder to change the root directory of its mount namespace,
and thus the host /nix/store is not visible in the mount namespace of
the builder. To get around this, just before doing pivot_root(), we
branch a second mount namespace that shares its /nix/store mountpoint
with the parent.

Recursive Nix currently doesn't work on macOS in sandboxed mode
(because we can't change the sandbox policy of a running build) and in
non-root mode (because setns() barfs).
Derivations that want to use recursion should now set

  requiredSystemFeatures = [ "recursive-nix" ];

to make the daemon socket appear.

Also, Nix should be configured with "experimental-features =
recursive-nix".
@edolstra edolstra force-pushed the recursive-nix branch from eb7131b to 69326f3 Nov 5, 2019
@matthewbauer

This comment has been minimized.

Copy link
Member

matthewbauer commented Nov 6, 2019

I've made an initial version of a nix-ccache flake: https://github.com/edolstra/nix-ccache. It provides a wrapper around gcc/g++ that executes the compilation of the preprocessed source in a recursive nix-build call.

Very cool! This looks a lot like what @layus talks about at NixCon. How costly is it to run every compilation in nix-build, though? Perhaps we need some heuristic to determine whether a C file is big enough to be cacheable, otherwise we impose a constant builder setup for every .c file.

@edolstra

This comment has been minimized.

Copy link
Member Author

edolstra commented Nov 7, 2019

A quick unscientific measurement suggests the overhead is ~0.15s per GCC call on my laptop. (This also depends on the size of the preprocessor output, since it needs to be copied to the Nix store.) This is enough to make configure scripts much slower, so right now there is a special check to disable building through recursive Nix when the input is called "conftest". A heuristic like you suggest might be better.

@volth

This comment has been minimized.

Copy link
Contributor

volth commented Nov 14, 2019

Wouldn't it be more flexible to add it in form of builtins.nixBuild accepting string of nix code to eval and returning a list of derivations?

Your example will be just

- hello=$(nix-build -E '(import <nixpkgs> {}).hello.overrideDerivation (args: { name = "hello-3.5"; })')
+ hello=${builtins.nixBuild ''(import <nixpkgs> {}).hello.overrideDerivation (args: { name = "hello-3.5"; })''}

Advantages are:

  1. inner nix-build's settings will be consistent with the outer, in terms that the inner won't have own --cores or $TMPDIR
  2. not having the full rich set of command-line options of nix-build, it would be easier to restrict what the inner build is allowed to do (for example, evaluate nix code only from a string, not from an arbitrary file)
  3. returning the list of derivations instead of raw stdout capture of nix-build executable, it could be used in buildInputs = builtins.nixBuild "some nix code". for example, when building java apps, the inner nix-build could look up into pom.xml and build a list of fetchurls for maven dependencies
@Ericson2314

This comment has been minimized.

Copy link
Member

Ericson2314 commented Nov 14, 2019

@volth you just reinvented import from derivation :) But a big benefit of recursive (to me at least) is trying to leverage eval less not more, i.e. nix-exprs can just be one unprivileged way to get drv files.

Other than that< I think you might prefer NixOS/rfcs#40.

  1. inner nix-build's settings will be consistent with the outer

Ret-cont recursive also does this.

  1. returning the list of derivations instead of raw stdout capture of nix-build executable

Never need to go stdout->store path either.

@edolstra

This comment has been minimized.

Copy link
Member Author

edolstra commented Nov 14, 2019

Wouldn't it be more flexible to add it in form of builtins.nixBuild accepting string of nix code to eval and returning a list of derivations?

Well, hello was just a toy example. The main usefulness of recursive Nix is if you don't know the inner derivations in advance. For example, in the case of nix-ccache (which wraps gcc invocations in nix builds), it's typically a makefile driving the build process, so you don't know at the outer expression level which inner builds are going to be done, or in what order.

@edolstra edolstra merged commit 69326f3 into master Dec 2, 2019
@edolstra edolstra deleted the recursive-nix branch Dec 2, 2019
@domenkozar domenkozar mentioned this pull request Dec 20, 2019
@volth volth mentioned this pull request Jan 10, 2020
5 of 13 tasks complete
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

5 participants
You can’t perform that action at this time.