Bash build failure on 15.09

[deepfire]

During nix-shell invocation:

make[1]: Leaving directory `/tmp/nix-build-bash-4.3-p39.drv-0/bash-4.3/lib/malloc'
building bash
rm -f bash
gcc -L./builtins -L./lib/readline -L./lib/readline -L./lib/glob -L./lib/tilde -L./lib/malloc -L./lib/sh  -rdynamic  -g -O2 -o bash shell.o eval.o y.tab.o general.o make_cmd.o print_cmd.o  dispose_cmd.o execute_cmd.o variables.o copy_cmd.o error.o expr.o flags.o jobs.o subst.o hashcmd.o hashlib.o mailcheck.o trap.o input.o unwind_prot.o pathexp.o sig.o test.o version.o alias.o array.o arrayfunc.o assoc.o braces.o bracecomp.o bashhist.o bashline.o  list.o stringlib.o locale.o findcmd.o redir.o pcomplete.o pcomplib.o syntax.o xmalloc.o  -lbuiltins -lglob -lsh  -lhistory  -ltilde -lmalloc    -ldl 
ls -l bash
-rwxr-xr-x 1 nixbld nixbld 2786742 Aug 28 22:38 bash
size bash
   text    data     bss     dec     hex filename
 713871   13440   18248  745559   b6057 bash
building bashbug
building .made
installing
install flags: install SHELL=/nix/store/k0vqprjmxybr7clvfljk13zsdjwklcch-bootstrap-tools/bin/sh    
building .build

          ***********************************************************
          *                                                         *
          * GNU bash, version 4.3.39(1)-release (x86_64-unknown-linux-gnu)
          *                                                         *
          ***********************************************************

building y.tab.h
building installdirs
mkdir -p -- /nix/store/548wfw1i43glkx8lkyjmbg59h6127qky-bash-4.3-p39/share/doc/bash
mkdir: cannot create directory '/nix/store/548wfw1i43glkx8lkyjmbg59h6127qky-bash-4.3-p39/share/doc': Permission denied
make: *** [installdirs] Error 1
builder for ‘/nix/store/d0s90931blfav7f9v3kz1443jiaszc8w-bash-4.3-p39.drv’ failed with exit code 2
error: build of ‘/nix/store/d0s90931blfav7f9v3kz1443jiaszc8w-bash-4.3-p39.drv’ failed

[wkennington]

Yeah, I've been having the same issues ever since multiple-outputs were added.

cc @edolstra @vcunat

[vcunat]

Since e1f78bf? Does this happen only without chrooting, or why doesn't Hydra at least have the probem?

[vcunat]

What kind of nix-shell invocation is it? (I wonder if it's nix-shell specific.)

[wkennington]

I believe it has to do with the case where you pull the binary for one of
the derivations in a multi-derivation output, and then you try and use one
of the other derivations without a binary substitute available. Nix tries
to build the package, and then fails to install to the already existing
derivation causing the whole thing to fail.

On Fri, Aug 28, 2015 at 11:21 PM Vladimír Čunát notifications@github.com
wrote:

What kind of nix-shell invocation is it? (I wonder if it's nix-shell
specific.)

—
Reply to this email directly or view it on GitHub
#9504 (comment).

[wkennington]

Ex.
I build pkgs.bash.out. I now have both pkgs.bash.{out,doc} in my nix store.
I run a garbage collect which cleans out pkgs.bash.doc but keeps
pkgs.bash.out.
I now try and build pkgs.bash.doc. This fails because it has to build both
pkgs.bash.{out,doc} and pkgs.bash.out already exists and is protected from
being overwritten.

On Fri, Aug 28, 2015 at 11:27 PM William Kennington william@wkennington.com
wrote:

I believe it has to do with the case where you pull the binary for one of
the derivations in a multi-derivation output, and then you try and use one
of the other derivations without a binary substitute available. Nix tries
to build the package, and then fails to install to the already existing
derivation causing the whole thing to fail.

On Fri, Aug 28, 2015 at 11:21 PM Vladimír Čunát notifications@github.com
wrote:

What kind of nix-shell invocation is it? (I wonder if it's nix-shell
specific.)

—
Reply to this email directly or view it on GitHub
#9504 (comment).

[vcunat]

There certainly was code for handling this in nix IIRC, but it's possible it doesn't cover all cases yet.

[deepfire]

The store itself is fundamentally impure, and so its state-space {(0,0),(1,0),(0,1),(1,1)} comes and bites us..
Before it was just {0, 1}, and so trivial to handle..

The state of a derivation evolved from a boolean to an N-tuple.

The state-space of a derivation evolved, in the general case, from {0, 1} to an infinite set of N-tuples of booleans.

[deepfire]

The default.nix/shell.nix that caused nix-shell to fail:

{ mkDerivation, stdenv
, python, pythonPackages, curl, wsmancli
, base, base-unicode-symbols, bytestring, containers
, directory, GenericPretty, ghc-prim, hashable, interpolatedstring-perl6, MissingH, pretty
, process, shake, tuple, unordered-containers
}:
mkDerivation {
  pname = "foo";
  src = ./.;
  buildDepends = [
    python pythonPackages.paramiko pythonPackages.netifaces curl wsmancli
    base base-unicode-symbols bytestring containers
    directory GenericPretty ghc-prim hashable interpolatedstring-perl6 MissingH pretty
    process shake tuple unordered-containers
  ];
}

and..

{ nixpkgs ? import <nixpkgs> {}
, compiler ? "ghc7102"
}:
let
  pkgs = nixpkgs.pkgs;
  ghc  = pkgs.haskell.packages.${compiler};
  pkgf = import ./.;
  drv  = ghc.callPackage pkgf {};
  ghci-ng      = nixpkgs.pkgs.haskell.lib.overrideCabal ghc.ghci-ng (oldAttrs: {
    buildDepends = [ ghc.syb ];
    src = pkgs.fetchgit {
      url = https://github.com/chrisdone/ghci-ng.git;
      rev = "738f66f3d1f1a3b7ba574fb9c83da793179a42c3";
      sha256 = "03kz2ysmglgnhcbzw03wrgjsf8pmh87f1ajgvdny3rm2mbg37iig";
    };
  });
  hell-git = pkgs.haskell-ng.lib.overrideCabal pkgs.haskellngPackages.hell (oldAttrs: {
    src = pkgs.fetchgit {
      url = https://github.com/chrisdone/hell.git;
      rev = "442b0fc7cd7e2e9a5b2e43b14540467ddd32d29b";
      sha256 = "0q046zixqz5hd8c2pd4m6gcz3qyl032z8i9xxs0658c36fkk788f";
     };
  });
in with pkgs;
  (pkgs.haskell.lib.addBuildTools drv [
    ghc.cabal-install
    ghci-ng hell-git
    emacs git gitAndTools.git-extras silver-searcher w3m
  ]).env

[deepfire]

@vcunat, I have nix.useChroot = true;

[copumpkin]

If there's an issue with nix's implementation of multiple outputs, has anyone filed a bug against the NixOS/nix repo?

[domenkozar]

Did anyone reproduce this one?

[deepfire]

@domenkozar I suspect it went like this:

1. build bash-for-bin -> {bin, doc}
2. nix-store --gc -> doc dies
3. build bash-for-doc -> install fails, because bin is already present

Now, why would something want bash-for-doc, don't ask me : -)

[peti]

@domenkozar, I had the same error on hydra.cryp.to, i.e. http://hydra.cryp.to/build/1132944.

[domenkozar]

@peti I don't think it's the same, the build just timed out due to sqlite being busy

[peti]

Actually, I believe that the original log file was lost when I restarted one of those Hydra builds. 😞 Unfortunately, Hydra destroys the original build log and stores only the one of the re-run -- which succeeded. Anyway, the the build says clearly "failed"; it doesn't say "timed out".

[vcunat]

(Off topic) IMHO the log storage is handled on nix-level already. (I know very little about Hydra, but sometimes I do find myself regretting the overwriting of logs in nix.)

[edolstra]

Hydra no longer uses Nix for storing logs, but it still overwrites old logs.

[vcunat]

(Off topic, I'm sorry.) Is there a reason not to just extend the handling of logs on the nix level already?

[edolstra]

Well, the presumption is that builds are pure, so repeated builds will produce the same log output, and then there is no reason to store multiple logs.

[domenkozar]

if doc output is GCed and later referenced, it's not pure anymore in this case.

[edolstra]

That's still pure (modulo Nix bugs and impurities in the build itself).

[domenkozar]

Yes, so can we fix this in Nix? We could require all mulitple outputs to be present at build time. Not sure how hard is that to achieve in current code.