Inconistent Haskell Binaries

[jagajaga]

Using the latest nixos-unstable (15.05pre61966.75ebc3c (Dingo))
Trying nix-env -iA nixos.pkgs.haskellngPackages.idris
Getting

installing ‘haskell-idris-0.9.17.1’
these derivations will be built:
  /nix/store/brd7m3c8dy3803cdxp61n2x6g64ylxzz-haskell-idris-0.9.17.1.drv
building path(s) ‘/nix/store/fi6xncgmy756v3w39p22zq661jxi8zag-haskell-idris-0.9.17.1’
setupCompilerEnvironmentPhase
Build with /nix/store/x6vp7154r4m8hc7fhgngakkbg8y8pqnm-ghc-7.10.1.
unpacking sources
unpacking source archive /nix/store/gmhl8ya5c73ml4gfixz4v58fxxgpzird-idris-0.9.17.1.tar.gz
source root is idris-0.9.17.1
patching sources
Run jailbreak-cabal to lift version restrictions on build inputs.
compileBuildDriverPhase
setupCompileFlags: -package-db=/tmp/nix-build-haskell-idris-0.9.17.1.drv-0/package.conf.d -j1 -threaded
[1 of 1] Compiling Main             ( Setup.hs, /tmp/nix-build-haskell-idris-0.9.17.1.drv-0/Main.o )
Linking Setup ...
configuring
configureFlags: --verbose --prefix=/nix/store/fi6xncgmy756v3w39p22zq661jxi8zag-haskell-idris-0.9.17.1 --libdir=$prefix/lib/$compiler --libsubdir=$pkgid --with-gcc=gcc --package-db=/tmp/nix-build-haskell-idris-0.9.17.1.drv-0/package.conf.d --ghc-option=-optl=-Wl,-rpath=/nix/store/fi6xncgmy756v3w39p22zq661jxi8zag-haskell-idris-0.9.17.1/lib/ghc-7.10.1/idris-0.9.17.1 --ghc-option=-j1 --enable-split-objs --disable-library-profiling --enable-shared --enable-library-vanilla --enable-executable-dynamic --enable-tests -fffi -fgmp --extra-include-dirs=/nix/store/4cnb5zg0xziq80s9pqz3z1r9vvnj65ia-boehm-gc-7.2f/include --extra-lib-dirs=/nix/store/4cnb5zg0xziq80s9pqz3z1r9vvnj65ia-boehm-gc-7.2f/lib --extra-include-dirs=/nix/store/fjkshii315nzy8bh74wv0x51di09iadj-gmp-5.1.3/include --extra-lib-dirs=/nix/store/fjkshii315nzy8bh74wv0x51di09iadj-gmp-5.1.3/lib
Configuring idris-0.9.17.1...
Flags chosen: execonly=False, ci=False, freestanding=False, release=True,
curses=False, gmp=True, ffi=True
Dependency annotated-wl-pprint -any: using annotated-wl-pprint-0.6.0
Dependency ansi-terminal -any: using ansi-terminal-0.6.2.1
Dependency ansi-wl-pprint -any: using ansi-wl-pprint-0.6.7.2
Dependency base -any: using base-4.8.0.0
Dependency base64-bytestring -any: using base64-bytestring-1.0.0.1
Dependency binary -any: using binary-0.7.3.0
Dependency blaze-html -any: using blaze-html-0.7.1.0
Dependency blaze-markup -any: using blaze-markup-0.6.3.0
Dependency bytestring -any: using bytestring-0.10.6.0
Dependency cheapskate -any: using cheapskate-0.1.0.3
Dependency containers -any: using containers-0.5.6.2
Dependency deepseq -any: using deepseq-1.4.1.1
Dependency directory -any: using directory-1.2.2.0
Dependency filepath -any: using filepath-1.4.0.0
Dependency fingertree -any: using fingertree-0.1.0.2
Dependency haskeline -any: using haskeline-0.7.2.1
Dependency idris -any: using idris-0.9.17.1
Dependency lens -any: using lens-4.9.1
Dependency libffi <0.2: using libffi-0.1
Dependency mtl -any: using mtl-2.2.1
Dependency network -any: using network-2.6.1.0
Dependency optparse-applicative -any: using optparse-applicative-0.11.0.2
Dependency parsers -any: using parsers-0.12.1.1
Dependency pretty -any: using pretty-1.1.2.0
Dependency process -any: using process-1.2.3.0
Dependency safe -any: using safe-0.3.8
Dependency split -any: using split-0.2.2
Dependency text -any: using text-1.2.0.4
Dependency time -any: using time-1.5.0.1
Dependency transformers -any: using transformers-0.4.2.0
Dependency transformers-compat -any: using transformers-compat-0.4.0.4
Dependency trifecta -any: using trifecta-1.5.1.3
Dependency uniplate -any: using uniplate-1.6.12
Dependency unix <2.8: using unix-2.7.1.0
Dependency unordered-containers -any: using unordered-containers-0.2.5.1
Dependency utf8-string -any: using utf8-string-0.3.8
Dependency vector -any: using vector-0.10.12.3
Dependency vector-binary-instances -any: using vector-binary-instances-0.2.1.0
Dependency xml -any: using xml-1.3.14
Dependency zlib -any: using zlib-0.5.4.2
Setup: The following installed packages are broken because other packages they
depend on are missing. These broken packages must be rebuilt before they can
be used.
package blaze-html-0.7.1.0 is broken due to missing package
text-1.2.0.4-98506efb1b9ada233bb5c2b2db516d91
package blaze-markup-0.6.3.0 is broken due to missing package
text-1.2.0.4-98506efb1b9ada233bb5c2b2db516d91
package charset-0.3.7.1 is broken due to missing package
semigroups-0.16.2.2-9e371001c95378abb50aaaa29107f7f8
package cheapskate-0.1.0.3 is broken due to missing package
text-1.2.0.4-98506efb1b9ada233bb5c2b2db516d91
package css-text-0.1.2.1 is broken due to missing package
attoparsec-0.12.1.6-2dceb091855fa2d2aff2ca4101afc493,
text-1.2.0.4-98506efb1b9ada233bb5c2b2db516d91
package keys-3.10.1 is broken due to missing package
comonad-4.2.5-7eef189f60ef3ade00c34373cbc94c20,
free-4.11-d4d8f661de2b62a913b6aa8952ccaf4f,
semigroupoids-4.3-7d27766d0e4e2d3668a87a193a1b7711,
semigroups-0.16.2.2-9e371001c95378abb50aaaa29107f7f8
package parsers-0.12.1.1 is broken due to missing package
attoparsec-0.12.1.6-2dceb091855fa2d2aff2ca4101afc493,
text-1.2.0.4-98506efb1b9ada233bb5c2b2db516d91
package pointed-4.2 is broken due to missing package
comonad-4.2.5-7eef189f60ef3ade00c34373cbc94c20,
kan-extensions-4.2.1-cfecbba103d74d41abe43728892ba932,
semigroupoids-4.3-7d27766d0e4e2d3668a87a193a1b7711,
semigroups-0.16.2.2-9e371001c95378abb50aaaa29107f7f8
package reducers-3.10.3.1 is broken due to missing package
comonad-4.2.5-7eef189f60ef3ade00c34373cbc94c20,
semigroupoids-4.3-7d27766d0e4e2d3668a87a193a1b7711,
semigroups-0.16.2.2-9e371001c95378abb50aaaa29107f7f8,
text-1.2.0.4-98506efb1b9ada233bb5c2b2db516d91
package tagsoup-0.13.3 is broken due to missing package
text-1.2.0.4-98506efb1b9ada233bb5c2b2db516d91
package trifecta-1.5.1.3 is broken due to missing package
comonad-4.2.5-7eef189f60ef3ade00c34373cbc94c20,
lens-4.9.1-86f47912bc0540e77144b84a216cfee5,
semigroups-0.16.2.2-9e371001c95378abb50aaaa29107f7f8
package xml-1.3.14 is broken due to missing package
text-1.2.0.4-98506efb1b9ada233bb5c2b2db516d91
package xss-sanitize-0.3.5.5 is broken due to missing package
attoparsec-0.12.1.6-2dceb091855fa2d2aff2ca4101afc493,
text-1.2.0.4-98506efb1b9ada233bb5c2b2db516d91
builder for ‘/nix/store/brd7m3c8dy3803cdxp61n2x6g64ylxzz-haskell-idris-0.9.17.1.drv’ failed with exit code 1
error: build of ‘/nix/store/brd7m3c8dy3803cdxp61n2x6g64ylxzz-haskell-idris-0.9.17.1.drv’ failed

cc @peti

[peti]

package blaze-html-0.7.1.0 is broken due to missing package
text-1.2.0.4-98506efb1b9ada233bb5c2b2db516d91

These packages have become inconsistent on your machine. (Cc: @Fuuzetsu might be interested.) The only way to recover from this issue is to garbage collect all GHC 7.10.x related packages and to re-build / download everything from scratch.

[Fuuzetsu]

Pretty sure it's specialisation bullshit but if you want to to publish the contents of some of the store paths here compared to the storepaths on Hydra or whatever then it might be helpful. If not, that's OK.

Honestly this bug is notorious as hell. I want to fix it in GHC but have no time for a while more. It sucks. I'll try to raise the issue on ghc-dev and see if anyone wants to step up.

[puffnfresh]

I had this same problem yesterday, except text was broken because of a bad link to bytestring. I deleted all my references to GHC and built again, but got the exact same problem once again. I deleted the references and rebuilt with --option build-use-substitutes false, which took forever but I finally had a working install.

I have a feeling something on Hydra is weird. But yeah, the real solution is to fix GHC:

https://ghc.haskell.org/trac/ghc/ticket/4012

[Fuuzetsu]

Sent a little bump to the mailing list http://mail.haskell.org/pipermail/ghc-devs/2015-May/008992.html

[jagajaga]

So what are my actions now?

[Fuuzetsu]

Wipe out GHC and try again, hoping everything comes from a binary cache or is otherwise consistent. If you want to be sure it works then run with --option build-use-substitutes false which will make the packages consistent at least on your system. Of course anything that later mixes those with a binary cache will potentially be broken…

[jagajaga]

How can I get what depends on ghc-7.10? Because I get error that it is alive.

[peti]

$ nix-store -q --referrers /nix/store/pwgwx7p0r262rz8v26rp1mjw1z6cj8vb-ghc-7.10.1

gives you the list of packages that depend on GHC. Basically, you have to remove all Haskell packages from all profiles in /nix/var/nix/profiles. Then nix-store --delete ... the GHC derivation to remove all of it. Then re-install with --option binary-caches http://hydra.cryp.to xor --option binary-caches http://hydra.nixos.org xor --option build-use-substitutes false. Personally, I take my Linux/x86_64 binaries from hydra.cryp.to, and I'm pretty sure they're consistent. They may be inconsistent with hydra.nixos.org, though, which is out of my control.

[jagajaga]

@peti Thank you!

[peti]

@puffnfresh, recommending that our users switch back to GHC 7.8.4 seems a bit premature. I don't think it's fair to say that the non-determinism bug has gotten worse in 7.10.x based on the evidence we have so far. GHC 7.10.x clearly didn't eliminate the problem -- as we had hoped --, but that's not the same thing as saying that it got worse.

[Fuuzetsu]

@peti , it did get worse. Previously the main source of non-determinism was that some names in the interface file could come out different in presence of parallel builds. At that point we had a workaround which was switching parallel builds off.

The issue in 7.10.1 is a different problem. Whether it existed in 7.8.x and whether it still exists in GHC HEAD I don't know, but vanilla 7.10.1 is worse non-determinism wise because we don't even have a workaround this time.

[peti]

@Fuuzetsu, we've had inconsistent 7.8.x packages without parallel builds too. Parallel building greatly increased the chances of hitting the non-determinism bug, but even without it the chances of hitting that bug were greater than zero. In some cases, 7.8.x hits the non-determinism bug with a chance of almost 100%! For example, revert d42d643 and try compiling haskell.packages.ghc784.conduit-extra. On my Linux/x86_64 machine, that build always fails, saying:

package conduit-1.2.4.2 is broken due to missing package
void-0.7-b1f172356aa99e91a6e5f75a6e9c6280

No amount of tweaking helped; it seems flat out impossible to get a consistent build of conduit-1.2.4.2 with GHC 7.8.4, whereas 7.10.1 compiles it without any problem (with parallel building enabled). I'm sure there were other cases, too, but I don't remember them right now.

From my point of view, GHC 7.10.x has been more stable than 7.8.x so far.

[Fuuzetsu]

For example, revert d42d643 and try compiling haskell.packages.ghc784.conduit-extra. On my Linux/x86_64 machine, that build always fails

Ah, I did not know about this. I don't mean to say that 7.8.4 had no issues when parallel builds weren't enabled but I did not know about any actual cases.

From my point of view, GHC 7.10.x has been more stable than 7.8.x so far.

Sure thing though I wonder if that's because you build the Haskell package set yourself as nixpkgs moves so you end up with consistent env that you then use. Most people are probably going to end up mixing official hydra, local builds and maybe your Hydra too. If you say we should stick with 7.10.1 for now then I trust your decision.

Ideally someone steps up to help on Trac 4012 and we have more reliable 7.10.2 of course. SPJ seems to have some idea of what may be going on here so if someone wants to attempt this, just leave a comment on that Trac ticket. I may have time myself in about a week and already suggested that the ticket should probably be a blocker for 7.10.2.

[peti]

[Your builds are stable] because you build the Haskell package set yourself as nixpkgs moves so you end up with consistent env that you then use. Most people are probably going to end up mixing official hydra, local builds and maybe your Hydra too.

Yes, it's true that mixing packages from different Hydra servers is more likely to cause trouble. It's a good idea to download Haskell packages from one source only to reduce the likelihood of hitting this bug. As of now, the official nixos-unstable channel doesn't have any Haskell binaries (those are built in http://hydra.nixos.org/jobset/nixpkgs/haskell-updates only), so it's actually possible to chose either hydra.cryp.to or hydra.nixos.org as a source.

[peti]

Instructions how to recover from this issue are now available at https://nixos.org/wiki/Haskell#Recovering_from_GHC.27s_non-deterministic_library_ID_bug. Please feel free to improve the Wiki page! I'll close this ticket, because it's not anything actionable for us, i.e. we can't do anything in Nix about this issue other than document it and help our users avoid it and/or recover from it -- what we've done.

[Gabriella439]

The link for recovering from the determinism bug has moved to: http://nixos.org/nixpkgs/manual/#how-to-recover-from-ghcs-infamous-non-deterministic-library-id-bug

[domenkozar]

According to https://ghc.haskell.org/trac/ghc/ticket/4012#comment:232 GHC 8.0.2 will fix these issues. Could we spin up haskell rebuild with ghc HEAD to assure this works?

[deepfire]

I believe I get the same symptoms, but given that this is in the context of an entirely new GHC definition (#34023), I'm not really sure the bug is the same.

But, for the record, the symptoms are:

• attempt to enter a nix-shell fails, during a ghc-pkg check
• the reflex package is said to be broken
• ..because MemoTrie couldn't be found:

There are problems in package reflex-0.5:
  Warning: haddock-interfaces: /nix/store/s6il1wrbkvrnlnz5y97hs6f4d2y8d1k7-reflex-0.4.0.1/share/doc/x86_64-linux-ghc-8.4.20180115/reflex-0.5/html/reflex.haddock doesn't exist or isn't a file
  Warning: haddock-html: /nix/store/s6il1wrbkvrnlnz5y97hs6f4d2y8d1k7-reflex-0.4.0.1/share/doc/x86_64-linux-ghc-8.4.20180115/reflex-0.5/html doesn't exist or isn't a directory
  dependency "MemoTrie-0.6.8-93uZB8uRGVUJYCCxwHIALD" doesn't exist

The interesting part, though, is that ghc-pkg refers to the missing MemoTrie package by an ID that is qualified with a hash -- and the reflex's MemoTrie dependency has that ID:

# 1. reflex path examined by ghc-pkg`
nix-store --query --deriver /nix/store/s6il1wrbkvrnlnz5y97hs6f4d2y8d1k7-reflex-0.4.0.1
/nix/store/7v64jygdwk53glhhkfx5r0kwki5s3ghl-reflex-0.4.0.1.drv

# 2. reflex derivation -> MemoTrie derivation
nix-store --query --references /nix/store/7v64jygdwk53glhhkfx5r0kwki5s3ghl-reflex-0.4.0.1.drv | grep MemoTrie
/nix/store/1mr7qkj150pmschj0qs0kyrp5vr1jg7a-MemoTrie-0.6.8.drv

nix-store --realise /nix/store/1mr7qkj150pmschj0qs0kyrp5vr1jg7a-MemoTrie-0.6.8.drv
warning: you did not specify ‘--add-root’; the result might be removed by the garbage collector
/nix/store/vl9cvc6k4yi4bwnj96ynbqkx1cyb20lg-MemoTrie-0.6.8

ls /nix/store/vl9cvc6k4yi4bwnj96ynbqkx1cyb20lg-MemoTrie-0.6.8/lib/ghc-8.4.20180115/package.conf.d/
MemoTrie-0.6.8-93uZB8uRGVUJYCCxwHIALD.conf

I.e. the MemoTrie-0.6.8-93uZB8uRGVUJYCCxwHIALD, that ghc-pkg claims that it doesn't exist -- does exist, but is somehow inaccessible in the GHC environment built by nix-shell?