How does one use cachix to share nix-shell environments?

[cocreature]

First of all, thanks for cachix! It’s an awesome project!

Now to my question: I have a bunch of projects that have fairly expensive (in terms of build time) development environments (e.g. they require debugging builds of LLVM) so cachix would be perfect for sharing these environments with other developers.
However, I’m not really sure what the best way of doing that is.

[domenkozar]

As far as I'm aware to upload all build-time dependencies you'd need to do:

a) nix-build shell.nix to build everything
b) nix-store -qR --include-outputs $(nix-instantiate shell.nix) | cachix push <name>

[cocreature]

Seems to work perfectly, thanks!

[domenkozar]

Let's leave this open until it's documented :)

[edude03]

Sorry this is a bit off topic, but how should your shell.nix look to make it buildable with nix-build?

My shells are usually pretty simple just

with import <nixpkgs>{}; 

stdenv.mkDerivation rec {
  name = "some-shell";
  buildInputs = [
    wget # or whatever
  ]; 
}

This doesn't work with nix build however because there is no $src and even if you disable the standard phases there is no output.

[domenkozar]

In that case doing just nix-shell shell.nix should suffice.

[edude03]

Ah! Thanks! I see

Edit: I whipped up a little script that I'm going to use in CI to keep the cache up to date

#! /usr/bin/env nix-shell
#! nix-shell -i bash https://github.com/NixOS/nixpkgs/tarball/db557aab7b690f5e0e3348459f2e4dc8fd0d9298 -A cachix
# Build the shell environment and exit on completion
nix-shell --command exit
# get a list of all the derivations used in the shell environment and push them to the cache
nix-store -qR --include-outputs $(nix-instantiate shell.nix) | cachix push $1

Hope it's useful to someone

[zimbatm]

Another strategy used by https://github.com/cachix/cachix-action is:

cache_name=foobar
# record all the existing paths in the store
nix path-info --all | grep -v '\.drv$' > /tmp/store-path-pre-build

# run your commands here. eg:
nix-shell --run "echo OK"

# at the very end, dump the diff to cachix
nix path-info --all | grep -v '\.drv$' | cat - /tmp/store-path-pre-build | sort | uniq -u  | cachix push ${cache_name}

The nice thing about this approach is that it doesn't need to carefully make sure that all the outputs or collected. Which can be quite difficult when using import-from-derivation or any nix wrappers.

[freeman42x]

@domenkozar Hi, has this been documented? I have the same issue right now, trying to push all nix-shell build dependencies to cachix.

[freeman42x]

Will the following warning cause any issues?

neo@nixos ~/P/haskell-editor-setup> nix-store -qR --include-outputs (nix-instantiate shell.nix) | cachix push fairy-tale-agi-solutions
warning: you did not specify '--add-root'; the result might be removed by the garbage collector

[jappeace]

@razvan-flavius-panda

It means the garbage collector will clean the result. But it doesn't matter since you pushed it to cache anyway.

[domenkozar]

This is now documented at https://docs.cachix.org/faq.html#is-there-a-way-to-cache-nix-shell

[fzakaria]

@zimbatm @domenkozar @edude03 the only issue with your suggestion above & in the cachix documentation is it includes all transitive dependencies to build; which is too broad

Consider this simple example:

let nixpkgs = import <nixpkgs> {};
in
with nixpkgs;
with stdenv;
with stdenv.lib;
mkShell {
  name = "example-shell";
  buildInputs = [chromium];
}

Ideally, I'd only want to cache chromium for my nix-shell.

However if you run the snippet given you get way more:

nix-store -qR --include-outputs $(nix-instantiate shell.nix) | head -n 10
warning: you did not specify '--add-root'; the result might be removed by the garbage collector
/nix/store/b7irlwi2wjlx5aj1dghx4c8k3ax6m56q-busybox.drv
/nix/store/bzq60ip2z5xgi7jk6jgdw8cngfiwjrcm-bootstrap-tools.tar.xz.drv
/nix/store/wzdwpgqf2384hr2npma78mqillg5lv08-unpack-bootstrap-tools.sh
/nix/store/0zhkga32apid60mm7nh92z2970im5837-bootstrap-tools.drv
/nix/store/9krlzvny65gdc8s7kpb6lkx8cd02c25b-default-builder.sh
/nix/store/1i5y55x4b4m9qkx5dqbmr1r6bvrqbanw-multiple-outputs.sh
/nix/store/4ygqr4w06zwcd2kcxa6w3441jijv0pvx-strip.sh
/nix/store/6k829xz9v508ncxfcf2p1b8g1wq54946-patch-shebangs.sh
/nix/store/8zxndz5ag0p6s526c2xyllhk1nrn4c3i-audit-tmpdir.sh
/nix/store/cickvswrvann041nqxb0rxilc46svw1n-prune-libtool-files.sh

nix-store -qR --include-outputs $(nix-instantiate shell.nix) | wc
warning: you did not specify '--add-root'; the result might be removed by the garbage collector
   2093    2093  139650

Is there a sensible way to only query the output + runtime?

Traditionally nix-store --query --requisites returns the runtime closure if given a store output path.
The problem is that mkShell itself cannot be built.

[domenkozar]

That could be fixed by making shell.nix build a noop.

Currently there's no easy way to query that in Nix: NixOS/nix#1245

[fzakaria]

@domenkozar I suspect a noop won't be enough.

A dummy file should be the derivation output that just has the list of the buildInputs in it.

This is because Nix greps for the buildInput hashes to determine the runnable set.

That might be a worthy improvement to mkShell to always output.

[fzakaria]

@domenkozar and others -- I opened NixOS/nixpkgs#95191 with a proposed solution.
Please let me know what you think.

[fzakaria]

For those reading this pullrequest here is a much saner transitive set to cache for shell.nix

nix-store --query --references $(nix-instantiate shell.nix) | \
xargs nix-store --realise | xargs nix-store --query --requisites | cachix push $NAME

[avanov]

What is the right way of using this pipe with nix copy? it reports error: path '/nix/store/fzmkblzbsmqlsw7dvch92k92mp7q5vlh-nix-shell' is not valid to me