Nix Rust build benchmark

This repository compares minimal Nix wrappers for building the same Rust application with:

• cargo-nix-plugin;
• crate2nix;
• Crane with compressed incremental artifacts (use-zstd);
• Crane with symlink-deduplicated incremental artifacts (use-symlink).

The application happens to be SelfCI. It is a fixed benchmark specimen here, not the subject of this repository.

Results

All values below use Cargo's dev profile on the machine recorded in each result bundle. Five-sample warm-store medians are exploratory; project-cold values are single functional samples. They are not portable performance rankings.

Warm-store end-to-end median:

Builder	Clean project	Source edit	Dependency change
cargo-nix-plugin	5.29 s	4.91 s	5.40 s
Crane use-zstd	7.68 s	7.13 s	7.60 s
Crane use-symlink	7.37 s	7.42 s	7.51 s
crate2nix, checked graph	7.12 s	7.23 s	9.52 s

The two Crane rows are a contemporaneous randomized n=5 run. The plugin and crate2nix rows come from earlier runs, so cross-builder differences remain indicative. The crate2nix dependency clock includes about 2.05 seconds of graph regeneration.

Project-cold realization and absolute artifact size:

Builder	Build time	Final output	Compiled dependencies	Compiled total	All paths added
Crane use-zstd	21.49 s	13.02 MiB	109.63 MiB	122.65 MiB	122.65 MiB
Crane use-symlink	25.48 s	13.02 MiB	474.65 MiB	487.67 MiB	487.67 MiB
cargo-nix-plugin	26.60 s	84.12 MiB	613.46 MiB	697.58 MiB	711.79 MiB
crate2nix, checked graph	31.80 s	70.71 MiB	491.96 MiB	562.67 MiB	576.87 MiB

Sizes are additive NAR sizes of distinct store paths, not physical disk use and not sums of overlapping closure sizes. “Compiled total” is the disjoint sum of final project outputs and compiled dependency outputs. “All paths added” is the primary absolute total: every post-build path absent from the explicit pre-build seed. Columns are nested sets and must not be added together.

Project-cold means a fresh rooted local Nix store seeded with toolchains, native tools, builders, derivation recipes, and fixed/fetched source inputs. Project and compiled Rust dependency outputs are asserted absent, and substitution is disabled during realization. It does not mean OS- or machine-cold: kernel page cache, filesystem cache, scheduling, and thermal state remain uncontrolled.

Crane artifact modes

The original wrapper measured use-zstd. It did not set installCargoArtifactsMode; pinned Crane therefore used its use-zstd default. The original pinned flakebox revision also explicitly defaulted its enhanced Crane arguments to use-zstd.

The current wrappers set the mode on both buildDepsOnly and buildPackage. use-zstd stores a compressed target.tar.zst; use-symlink stores a target tree and symlink-deduplicates unchanged files against previous artifacts.

In this project, both modes had similar warm incremental time. use-zstd was about four seconds faster in the single project-cold sample and used 122.65 MiB instead of 487.67 MiB for newly compiled artifacts. Prefer use-zstd here. use-symlink may still trade store space for less compression work on other, larger workloads; measure the actual project rather than generalizing this sample.

Recommendation

Use Crane use-zstd as the conservative default for this project. It has a simple wrapper, avoids cargo-nix-plugin's exact Nix plugin-ABI requirement, and produced the smallest project-cold artifact set. cargo-nix-plugin's lower observed warm medians justify a larger randomized follow-up if its ABI and registry-prefetch operational costs are acceptable. Checked crate2nix makes the dependency graph explicit but adds a regeneration step when dependencies change.

Do not select from the timing table alone. Runs used a developer machine with an uncontrolled global store for warm lanes, separate historical runs for some rows, and fewer than the 20 randomized samples required for p95.

Reproduce

Install Nix with flakes enabled, clone this repository, then run:

git clone https://github.com/dpc/crane-eval
cd crane-eval

# Five warm incremental samples for every lane.
nix run .#benchmark -- \
  --suite warm --repetitions 5 \
  --output "$PWD/benchmark-results/warm"

# One isolated project-cold sample for every lane. This copies several stores
# and takes substantially longer.
nix run .#benchmark -- \
  --suite project-cold --repetitions 1 \
  --output "$PWD/benchmark-results/project-cold"

To compare only the two Crane modes:

nix run .#benchmark -- \
  --solution crane-modes --suite warm --repetitions 5 \
  --output "$PWD/benchmark-results/crane-warm"

nix run .#benchmark -- \
  --solution crane-modes --suite project-cold --repetitions 1 \
  --output "$PWD/benchmark-results/crane-project-cold"

Accepted --solution values are all, crane-modes, crane-use-zstd, crane-use-symlink, crate2nix, and cargo-nix-plugin. crane remains a compatibility alias for crane-use-zstd. Run nix run .#benchmark -- --help for the complete option summary.

Each output contains raw events, exact flake/environment metadata, summaries, and store evidence. Keep the complete output when reporting new numbers.

Build an individual package with:

nix build .#crane-use-zstd
nix build .#crane-use-symlink
nix build .#crate2nix
./bench/nix-with-plugin build \
  .#legacyPackages.x86_64-linux.benchmark.dev.cargo-nix-plugin

See BENCHMARK.md for the full protocol and results for retained evidence.