New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

nix-shell dependencies can be garbage collected any time now / persistent nix-shell envs #2208

Open
goodwillcoding opened this Issue Jun 4, 2018 · 6 comments

Comments

Projects
None yet
3 participants
@goodwillcoding

goodwillcoding commented Jun 4, 2018

There does not seem to be a way to prevent nix-shell dependencies from being garbage collected at any time. Related to this, for a while now there been possible to create persistent nix-shell environments that would be spared garbage collection. Neither of the two methods (see below) are now working.

shell.nix

with import <nixpkgs> {};
stdenv.mkDerivation {
    src = "./.";
    name = "test-0.1";
    buildInputs = [ hello ];
}

Previous Method 1 (https://nixos.org/nix-dev/2016-November/022173.html):

nix-instantiate shell.nix --show-trace  --indirect --add-root "$(pwd)/"shell.drv
nix-shell $(readlink $(pwd)/shell.drv)

While this does create a gcroot to the derivatation, it does not work (my guess cause it is not a gcroot to the build output)

$ tree /nix/var/nix/gcroots/auto/
/nix/var/nix/gcroots/auto/
└── rz3awpwi7lxpwfjmhqbab0rdfc3nj94m -> /home/user/tmp/persist-nix-shell/shell.drv

If nix-collect-gc is run while the shell is active it cleans up everything this derivation depends on

Previous Method 2:

mkdir gcroot
nix-shell shell.nix --indirect --add-root "gcroot/gc"

This used to create symlinks to dependencies, how ever since --indirect is now ignored as of 19477e8 this also no longer works as gcroot to the output is no longer created.

Please advise on this

@goodwillcoding

This comment has been minimized.

goodwillcoding commented Jun 6, 2018

@edolstra any thoughts? suggestion on this ^

@srghma

This comment has been minimized.

srghma commented Aug 11, 2018

Method 1 makes gcroot for shell derivation, this indeed doesnt prevent it dependencies to be garbage collected
Method 2 should makes gcroot for each dependency of shell derivation, but it's not working indeed

I have made script that makes same as method 2

#! /bin/sh -eu

# make gcroot for each dependency of shell and run nix-shell

mkdir -p .gcroots
nix-instantiate shell.nix --indirect --add-root $PWD/.gcroots/shell.drv
nix-store --indirect --add-root $PWD/.gcroots/shell.dep --realise $(nix-store --query --references $PWD/.gcroots/shell.drv)
exec nix-shell $(readlink $PWD/.gcroots/shell.drv)

tnx to http://nix-dev.science.uu.narkive.com/QjeSH005/truly-persistent-nix-shell

@lheckemann

This comment has been minimized.

Member

lheckemann commented Aug 12, 2018

Do you have the keep-derivations and keep-outputs options set to true in your nix.conf? This is necessary for this to work iirc.

@srghma

This comment has been minimized.

srghma commented Aug 13, 2018

@lheckemann no, I dont have this options

 ~  cat /etc/nix/nix.conf
# WARNING: this file is generated from the nix.* options in
# your NixOS configuration, typically
# /etc/nixos/configuration.nix.  Do not edit it!
build-users-group = nixbld
max-jobs = 40
cores = 0
sandbox = true
extra-sandbox-paths =
substituters = https://cache.nixos.org https://cachix.cachix.org https://srghma.cachix.org
trusted-substituters =
trusted-public-keys = cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY= cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY= cachix.cachix.org-1:eWNHQldwUO7G2VkjpnjDbWwy4KQ/HNxht7H4SSoMckM= srghma.cachix.org-1:EUHKjTh/WKs49hFtw6bwDE9oQLeX5afml0cAKc97MbI=
auto-optimise-store = false
require-sigs = true

trusted-users = root srghma
allowed-users = *
builders =

auto-optimise-store = true
@lheckemann

This comment has been minimized.

Member

lheckemann commented Aug 13, 2018

keep-derivations (sorry, got the name wrong in my previous comment) defaults to true, so that should be fine.
Try adding nix.extraOptions = "keep-outputs = true"; to your configuration.nix and rebuilding, then see if your shell dependencies still get gc'd when you use method 1.

keep-outputs
    If true, the garbage collector will keep the outputs of non-garbage derivations. If false (default), outputs
    will be deleted unless they are GC roots themselves (or reachable from other roots).
    In general, outputs must be registered as roots separately. However, even if the output of a derivation is
    registered as a root, the collector will still delete store paths that are used only at build time (e.g., the C
    compiler, or source tarballs downloaded from the network). To prevent it from doing so, set this option to true.
@srghma

This comment has been minimized.

srghma commented Aug 13, 2018

@lheckemann with keep-outputs and using this script

#! /bin/sh -eu

mkdir -p .gcroots

# add shell as gc-root in /nix/var/nix/gcroots/auto
nix-instantiate shell.nix --indirect --add-root $PWD/.gcroots/shell.drv

exec nix-shell $(readlink $PWD/.gcroots/shell.drv) "$@"

garbage collector indeed keep dependencies of shell, neat trick, but I still use the previous version of the script since keep-outputs = true is not defaut

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment