New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Nix garbage collects build dependencies even when gc root is added. #979

Closed
vrthra opened this Issue Jul 18, 2016 · 9 comments

Comments

Projects
None yet
7 participants
@vrthra
Member

vrthra commented Jul 18, 2016

Steps:

  • I have a development environment set under $PWD/etc/shell.nix.
  • I do nix-instantiate ./etc/shell.nix --indirect --add-root $PWD/etc/shell.drv
  • Then nix-shell --pure $PWD/etc/shell.drv
  • Exit nix-shell, then nix-collect-garbage -d
  • Try nix-shell --pure $PWD/etc/shell.drv

Result: the entire set of packages specified under shell.nix is downloaded again

  • nixos-version 16.03.1083.d745044 (Emu)
  • nix-env --version (Nix) 1.11.2
  • nix-instantiate --eval '<nixpkgs>' -A lib.nixpkgsVersion 16.09pre-git
@vrthra

This comment has been minimized.

Member

vrthra commented Jul 19, 2016

So, this seems to require this incantation to make sure that the development environment is not gc-d:

mkdir -p .gcroots
nix-shell . --pure --indirect --add-root .gcroots/dep

@vrthra vrthra closed this Jul 25, 2016

@domenkozar

This comment has been minimized.

Member

domenkozar commented Jul 25, 2016

@vrthra can you elaborate?

@vrthra

This comment has been minimized.

Member

vrthra commented Jul 25, 2016

After seeing that a command exists (nix-shell . --pure --indirect --add-root .gcroots/dep) for creating a non-gcd environment, I was not sure that this was a bug any more. (When I raised the bug I did not know of the above command.). Should I reopen this?

@Ericson2314

This comment has been minimized.

Member

Ericson2314 commented Aug 12, 2016

I think this is expected behavior, but there should be an easier way to create a root for the build environment without having to open a shell.

@ashgillman

This comment has been minimized.

ashgillman commented Sep 12, 2016

I am having a similar issue, and I think I can shed some light. The issue crops up when you make a derivation with no src. nix-instantiate will then make a .drv, and nix-shell will happily load it, however you can't build the derivation as it has no output. I guess when the GC looks at the .drv it uses the same mechanism as the building and can't determine the dependencies.

@vcunat

This comment has been minimized.

Member

vcunat commented Sep 18, 2016

No, I don't think the mechanisms are close. Instead, I suspect that nix-shell . does some additional magic that vanilla nix-instantiate shell.nix doesn't cover.

@ashgillman

This comment has been minimized.

ashgillman commented Sep 18, 2016

@vcunat That's what I mean, and that he garbage collector (i.e., nix-store) also doesn't include the additional magic (I am guessing), and so won't correctly prevent collection.

Here's a MWE:

{9:06}ash@daftpunk-rbh:~/tmp ➭ cat py33.nix 
let
  pkgs = import <nixpkgs> {};
in
{ stdenv ? pkgs.stdenv }:

stdenv.mkDerivation {
  name = "python33-test";
  buildInputs = (with pkgs; [
  ]) ++ (with pkgs.python33Packages; [
    python
  ]);
}

{9:07}ash@daftpunk-rbh:~/tmp ➭ nix-store --gc --print-dead | grep python3-3
finding garbage collector roots...
determining live/dead paths...
/nix/store/3lnk8y1xn4idajsn7ajlg6fc2shm1ck7-python3-3.3.6.drv
/nix/store/5dq1nr4s0xh5iry548w09423zwpkk9w7-python3-3.3.6
{9:07}ash@daftpunk-rbh:~/tmp ➭ nix-instantiate py33.nix --indirect --add-root $PWD/py33.drv 
/home/ash/tmp/py33.drv
{9:08}ash@daftpunk-rbh:~/tmp ➭ ls -l /nix/var/nix/gcroots/auto | grep py33
lrwxrwxrwx 1 ash ash 22 Sep 19 09:08 ppnsyyqpfxh29bhprd744wd9yr1hk87w -> /home/ash/tmp/py33.drv
{9:08}ash@daftpunk-rbh:~/tmp ➭ nix-store --gc --print-dead | grep python3-3                
finding garbage collector roots...
determining live/dead paths...
/nix/store/5dq1nr4s0xh5iry548w09423zwpkk9w7-python3-3.3.6
{9:09}ash@daftpunk-rbh:~/tmp ➭ nix-shell py33.nix --indirect --add-root .gcroots/py33dep

[nix-shell:~/tmp]$ exit
{9:10}ash@daftpunk-rbh:~/tmp ➭ ls -l /nix/var/nix/gcroots/auto | grep py33
lrwxrwxrwx 1 ash ash 32 Sep 19 09:10 9nz7jcakc31gx6d1di5gh4ffb8ras07c -> /home/ash/tmp/.gcroots/py33dep-4
lrwxrwxrwx 1 ash ash 36 Sep 19 09:10 b93splkqrpj8lj216kgqjxmyj238q4pd -> /home/ash/tmp/.gcroots/py33dep-3-doc
lrwxrwxrwx 1 ash ash 32 Sep 19 09:10 bhy9zmzmgpzqalrvvi212z215wbxawaj -> /home/ash/tmp/.gcroots/py33dep-3
lrwxrwxrwx 1 ash ash 32 Sep 19 09:10 bm9ml0zamryscgs9pyqlqh16y5gxmv4r -> /home/ash/tmp/.gcroots/py33dep-2
lrwxrwxrwx 1 ash ash 30 Sep 19 09:10 m03hlcvvmpaabhhwi6vvb20idmav2mla -> /home/ash/tmp/.gcroots/py33dep
lrwxrwxrwx 1 ash ash 22 Sep 19 09:08 ppnsyyqpfxh29bhprd744wd9yr1hk87w -> /home/ash/tmp/py33.drv
{9:10}ash@daftpunk-rbh:~/tmp ➭ nix-store --gc --print-dead | grep python3-3
finding garbage collector roots...
removing stale link from ‘/nix/var/nix/gcroots/auto/q19dnb49593w3gj7bjvxrlz8239z9d3a’ to ‘/run/user/1000/nix-shell.gpLNg8/derivation’
determining live/dead paths...
{9:10}ash@daftpunk-rbh:~/tmp ➭ 

The GC still wants to delete the py33 installation when you use the traditional nix-instantiate. So the nix-shell with --indirect --add-root seems to fix (I wasn't aware of this earlier). But the nix-instantiate is still what seems to be the recommended option in the docs, which can cause a lot of pain.

barrucadu added a commit to barrucadu/dotfiles that referenced this issue Mar 13, 2017

nix-instantiate doesn't create gc roots in the way I need.
Just using `nix-instantiate --add-root` doesn't prevent built things
from being garbage collected, see NixOS/nix#979.

Using `nix-shell --indirect --add-root` fixes the issue.

barrucadu added a commit to barrucadu/dotfiles that referenced this issue Mar 13, 2017

nix-instantiate doesn't create gc roots in the way I need.
Just using `nix-instantiate --add-root` doesn't prevent built things
from being garbage collected, see NixOS/nix#979.

Using `nix-shell --indirect --add-root` fixes the issue.
@CMCDragonkai

This comment has been minimized.

CMCDragonkai commented Oct 31, 2017

How come these options --indirect and --add-root are not documented inside nix-shell? It would be great if nix-shell's manual came with some mention about how to prevented GCing shell environments.

@Ralith

This comment has been minimized.

Ralith commented Jun 5, 2018

The nix-shell options quoted above seem to be silently ignored today. I'm accomplishing something similar by parsing the output of nix show-derivation somewhat laboriously. It would be much nicer for nix tools to have some first-class way to reference a build-time closure.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment