remote instantiation (NIX_REMOTE, --store) is an order of magnitude slower than local instantiation + nix copy

[Atemu]

Describe the bug

A clear and concise description of what the bug is.

When doing remote builds, instantiation time is unreasonably long (many seconds for a simple package). About an order of magnitude slower in fact:

$ nix run nixpkgs#hyperfine -- -w 1 -- 'nix copy $(nix-instantiate -A hello --system x86_64-linux) --to ssh-ng://remote'
Benchmark 1: nix copy $(nix-instantiate -A hello --system x86_64-linux) --to ssh-ng://remote
  Time (mean ± σ):     810.2 ms ±  34.2 ms    [User: 103.5 ms, System: 26.9 ms]
  Range (min … max):   768.9 ms … 873.3 ms    10 runs
 

$ nix run nixpkgs#hyperfine -- -w 1 -- "nix-instantiate -A hello --system x86_64-linux --store ssh-ng://remote"
Benchmark 1: nix-instantiate -A hello --system x86_64-linux --store ssh-ng://remote
  Time (mean ± σ):      8.616 s ±  0.263 s    [User: 0.632 s, System: 0.328 s]
  Range (min … max):    8.070 s …  8.949 s    10 runs
 

Steps To Reproduce

1. nix copy $(nix-instantiate -A hello --system x86_64-linux) --to ssh-ng://remote
2. nix-instantiate -A hello --store ssh-ng://remote
3. Observe difference

Expected behavior

A clear and concise description of what you expected to happen.

Remote instantiation should be the same speed or faster than local instantiation + copy.

Rationale for faster than local + copy: .drv files should not need to be written locally during a remote build (certainly not in the hot path), saving many IOPS.

nix-env --version output

nix-env (Nix) 2.18.1

$ uname -a
Darwin hostname 23.3.0 Darwin Kernel Version 23.3.0: Wed Dec 20 21:30:44 PST 2023; root:xnu-10002.81.5~7/RELEASE_ARM64_T6000 arm64 arm Darwin

Additional context

During remote instantiation, there appears to be a constant 1Mbit/s upload stream, no matter whether I previously instantiated the exact same drv or not.

Priorities

Add 👍 to issues you find important.

[thufschmitt]

@Atemu does that also happen with --store ssh-ng://remote --eval-store auto?

[Atemu]

Indeed it does not!

$ nix run nixpkgs#hyperfine -- -w 1 -- "nix-instantiate -A hello --system x86_64-linux --store ssh-ng://remote --eval-store auto"
Benchmark 1: nix-instantiate -A hello --system x86_64-linux --store ssh-ng://remote --eval-store auto
  Time (mean ± σ):     261.7 ms ±   9.7 ms    [User: 134.4 ms, System: 46.2 ms]
  Range (min … max):   251.1 ms … 284.9 ms    11 runs
 

Thanks :)

What is this wizardry and why don't I cast it by default?

Can I make that the default somehow? I don't want to have to specify it on every invocation that does a remote build.

[thufschmitt]

What is this wizardry and why don't I cast it by default?

--store sets the store used by Nix, both for building and for evaluation. This means that just setting it will make every store-operation during the evaluation use the remote store (this includes writing the .drv file for every call to builtins.derivation, uploading every interpolated path, etc..). And because the evaluator is strongly sequential, this can be really slow.

Using --eval-store auto makes nix use the local store for the evaluation (auto means “daemon or direct write to a local store”), and copy the resulting drv tree to the remote store – so very similar to nix-instantiate && nix copy.
Your last benchmark is probably a bit biased btw because I suspect that nix-instantiate --eval-store foo --store bar won't copy the closure to bar as it doesn't require building it.

Can I make that the default somehow? I don't want to have to specify it on every invocation that does a remote build.

Apparently not, because eval-store is just a cli flag, not an option – but I think that could be reasonably easy to fix.

More broadly, this store thing is a bit of a footgun, it would be nice to find a good interface so that people don't run into issues like the current one. Any suggestion to improve it is welcome :)

[Atemu]

Related: #6893

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2024-03-04-nix-team-meeting-minute-130/40830/1