Benchmarks fail due to "shadowed dependencies"

[crockeea]

This problem is described in a Stackage Server issue, but the problem should really be reported here. The problem is this:

library a
library b depends on a
benchmarks for library a depend on a and b

stack bench a results in the error:

> stack bench a
...
Preprocessing library a-0.0.1...
Preprocessing benchmark 'a-benches' for a-0.0.1...
<command line>: cannot satisfy -package-id b-0.0.1-2uppVa4SsB06R8hiHGD1I8: 
    b-0.0.1-2uppVa4SsB06R8hiHGD1I8 is unusable due to shadowed dependencies:
      a-0.0.1-B41UI81aQaCEjW3sk56eaB
    (use -v for more information)
Progress: 3/4
--  While building package a-0.0.1 using:
      /home/eric/.stack/setup-exe-cache/x86_64-linux/setup-Simple-Cabal-1.24.0.0-ghc-8.0.1 --builddir=.stack-work/dist/x86_64-linux/Cabal-1.24.0.0 build bench:a-benches --ghc-options " -ddump-hi -ddump-to-file"
    Process exited with code: ExitFailure 1

Note that the benchmarks compile if a-benches only depends on a or b, but not both. I don't see any reason why the benchmarks should fail to compile when they depend on both a and b; it's clear what the compilation order should be.

As indicated in the Stackage Server issue, this problem affects many packages including

• text
• unordered-containers
• clock
• options
• attoparsec
• wai-logger

A minimal example is in shadowed-deps.tar.gz.

[snoyberg]

@lehins pointed this out to me. I'm looking at logs that look like this:

$ stack bench a
Downloaded lts-7.0 build plan.    
a-0.0.1: configure
a-0.0.1: build
a-0.0.1: copy/register
b-0.0.1: configure (lib)
b-0.0.1: build (lib)
b-0.0.1: copy/register
a-0.0.1: configure (bench)
Configuring a-0.0.1...
a-0.0.1: build (bench)
Preprocessing library a-0.0.1...
Preprocessing benchmark 'a-benches' for a-0.0.1...

And then the shadowing messages. From what I can see, there's nothing we can do in Stack to work around this. It seems like:

• We configure/build library a, and get package ID a-123
• We configure/build library b, and get package ID b-456
• Now we go back and configure a for building benchmarks, and explicitly tell Cabal to only build the benchmark component. However, Cabal builds both the library and benchmark, ends up with a different package ID, and shadowing occurs.

I could be mistaken, perhaps something else it happening. But it looks like the only solution will be switching to Cabal 2.0 and using the componentized configuration, which allows for not rebuilding a library.

[alexkazik]

I have a different problem the shadowing:

• I have 8 packages (within one project with only one stack.yaml for them all but each in a own subdirectory, 7 are only libraries and the 8th lib+exe, no test so far)
• They all have clear dependencies (nothing circular)
• When I compile from scratch it usually works
• If I change some sources it sometimes fails due to shadowing, stack clean usually does the trick
• Sometimes only removing the .stack-work directory helps

EDIT: with nightly-2017-09-03 (ghc 8.2)

[lehins]

@alexkazik I stumble upon exactly the same issue recently, which only happens for me with ghc-8.2 (current nightly snapshot), but not ghc-8.0. Naturally cabal versions used are also different.
The workaround that took care of this for me is touch package-name.cabal.

[lehins]

@snoyberg Just in case, here are the steps using massiv library that reliably replicate the issue:
https://github.com/lehins/massiv/tree/5f3d083a343b22e866656e22036d42d294116d4d

• Run benchmarks first time, which will compile everything and run fine:

~/massiv/massiv-bench$ stack --stack-yaml=stack-ghc-8.2.yaml bench massiv-bench:fold-seq --benchmark-arguments='--match prefix Sum'

• Change a file in the source library, say comment out a function zipWith3 in massiv/massiv/src/Data/Array/Massiv.hs

• Try running benchmarks again:

~/massiv/massiv-bench$ stack --stack-yaml=stack-ghc-8.2.yaml bench massiv-bench:fold-seq --benchmark-arguments='--match prefix Sum'
massiv-0.1.0.0: unregistering (local file changes: src/Data/Array/Massiv.hs)
massiv-bench-0.1.0.0: unregistering (missing dependencies: massiv)
massiv-0.1.0.0: build (lib)
massiv-0.1.0.0: copy/register
massiv-bench-0.1.0.0: build (lib + bench)
Preprocessing library for massiv-bench-0.1.0.0..
Building library for massiv-bench-0.1.0.0..
[3 of 6] Compiling Bench.Massiv.Auto ( src/Bench/Massiv/Auto.hs, .stack-work/dist/x86_64-linux/Cabal-2.0.0.2/build/Bench/Massiv/Auto.o ) [Data.Array.Massiv changed]
Preprocessing benchmark 'fold-seq' for massiv-bench-0.1.0.0..
Building benchmark 'fold-seq' for massiv-bench-0.1.0.0..
<command line>: cannot satisfy -package-id massiv-bench-0.1.0.0-4cUIceH4fry9LCxUPcSNIr: 
    massiv-bench-0.1.0.0-4cUIceH4fry9LCxUPcSNIr is unusable due to shadowed dependencies:
      massiv-0.1.0.0-DkTbpETOx8EEaw3SIP44Bw
    (use -v for more information)
Progress: 2/3
--  While building package massiv-bench-0.1.0.0 using:
      /home/lehins/.stack/setup-exe-cache/x86_64-linux/Cabal-simple_mPHDZzAJ_2.0.0.2_ghc-8.2.1 --builddir=.stack-work/dist/x86_64-linux/Cabal-2.0.0.2 build lib:massiv-bench bench:fold-seq --ghc-options " -ddump-hi -ddump-to-file"
    Process exited with code: ExitFailure 1

• Change cabal file for benchmarks suite:

~/massiv/massiv-bench$ touch massiv-bench.cabal

• Everything works fine again:

~/massiv/massiv-bench$ stack --stack-yaml=stack-ghc-8.2.yaml bench massiv-bench:fold-seq --benchmark-arguments='--match prefix Sum'
massiv-bench-0.1.0.0: configure (lib + bench)
Configuring massiv-bench-0.1.0.0...
massiv-bench-0.1.0.0: build (lib + bench)
Preprocessing library for massiv-bench-0.1.0.0..
Building library for massiv-bench-0.1.0.0..
Preprocessing benchmark 'fold-seq' for massiv-bench-0.1.0.0..
Building benchmark 'fold-seq' for massiv-bench-0.1.0.0..
[1 of 1] Compiling Main             ( bench/FoldSeq.hs, .stack-work/dist/x86_64-linux/Cabal-2.0.0.2/build/fold-seq/fold-seq-tmp/Main.o ) [Data.Array.Massiv changed]
Linking .stack-work/dist/x86_64-linux/Cabal-2.0.0.2/build/fold-seq/fold-seq ...
massiv-bench-0.1.0.0: copy/register
Installing library in /home/lehins/github/massiv/massiv-bench/.stack-work/install/x86_64-linux/nightly-2017-07-31/8.2.1/lib/x86_64-linux-ghc-8.2.1/massiv-bench-0.1.0.0-4cUIceH4fry9LCxUPcSNIr
Registering library for massiv-bench-0.1.0.0..
massiv-bench-0.1.0.0: benchmarks
Running 1 benchmarks...
...

[ndmitchell]

FWIW, I'm now seeing this bug very regularly, with GHC 8.2 on Windows. Things like touching the cabal file sometimes work, but are far from reliable in my experience.

[snoyberg]

I received a similar bug report from a new Yesod users who was using Haskell Platform and cabal-install. I'm concerned that it may be a regression in Cabal 2.0. I'll try to repro this tomorrow. My goal is to get a minimal case and then test it with both cabal-install and plain old Setup.hs. If anyone else beats me to it, let me know.

[snoyberg]

I've created a minimal reproducing case for this:

https://github.com/snoyberg/stack-issue-2781

I'm going to see if I can further reproduce this using just the Setup.hs files.

[ndmitchell]

For info, I've had this happen with benchmarks and test suites disabled, so just running stack build. Not sure if that would make your test case any simpler.

[snoyberg]

I was able to reproduce this with just the Setup.hs files, so this is definitely not a bug in Stack. This bug is already being tracked upstream with haskell/cabal#4728.

[snoyberg]

@ndmitchell I just tried simplifying my repro to not use the benchmark and was unable to reproduce. If you have thoughts on how to simplify it, please do take a crack at it, I'm sure it will help the debugging.

[ndmitchell]

While I see it semi-regularly, I never figured out a pattern, so no suggestions I'm afraid.

[wraithm]

I've definitely started running into this issue quite a bit since upgrading to GHC 8.2

[phadej]

I have a hunch that using allow-newer: true makes things pop-up more often. (and actually using deps which need allow-newer)

[phadej]

... or having snapshot package depending on local package

[phadej]

and specifying --haddock. Weird.

[treblacy]

Regarding the life story of cabal-repro.sh:

1. After the first registering of foo, please do a

   ghc-pkg --package-db=$PKGDB field foo abi
   

   to see its ABI hash. If you're adding this command to cabal-repro.sh, maybe add > abi1.log so you can find it easily later.

2. When configuring bar, the above ABI hash of foo is noted and recorded as part of the dependency.

3. After the second registering of foo, please again do a

   ghc-pkg --package-db=$PKGDB field foo abi
   

   (maybe with > ab12.log). The ABI hash changes because the short code of foo changes and it's so short it's a good candidate for getting inlined so it is part of Foo.hi so it affects the ABI hash.

4. Now you build bar without reconfiguring. First the library component is built, which is fine for now, but it is recorded to depend on the 1st foo ABI hash, not the 2nd.

   Next, there are two cases, but eventually I will pull a disjunction-elimination to draw the same conclusion.

   1. You had --enable-benchmarks. Now the benchmark component is built, which depends on bar, but bar depends on a non-existent foo ABI hash. This is when you get an error.

   2. You didn't have --enable benchmarks. So no one uses bar for now, so no one notices anything wrong. But please press ahead with

       runghc Setup.hs register
       ghc -v -no-user-package-db -package-db=$PKGDB
      

      to see the same message about shadowing (or more accurately non-existence—"shadowing" is a pretty old error message). This means the problem has been created and is ready to bite.

      Also of interest is the output of

       ghc-pkg --package-db=$PKGDB field bar abi-depends
      

   Either way you have/will run into an error when you try to use bar for real.

I don't know what you should make of it. If you configure a package, and change something in the environment, then come back to resume building—but you don't reconfigure—this sounds like obviously risky to me. But maybe you have a need for it, and you could say that Cabal and GHC are being too paranoid. Or you could cherish this basic conservative sanity check and call it "this is not a bug, this is a feature". Or maybe a bit of both---perhaps configure time is too early to record other packages' ABI hashes, perhaps build time is better.

I do not know Stack and I do not know the life story of repro.sh.

[ndmitchell]

Note that the Cabal ticket says that only the full platform is affected, and only on non-Windows. Given I'm getting this issue from Windows with Stack using the raw GHC then either they've misdiagnosed the issue, or there are multiple issues causing the same error message.

[snoyberg]

Thanks for the analysis @treblacy. This is in line with @ezyang's comment here: haskell/cabal#4728 (comment). The problem seems to be that Cabal's otherwise-working auto-reconfigure logic isn't kicking in. And Stack isn't noticing that it should force a reconfigure since the package ID hasn't changed.

(Side note: this is another reason I think the change to semantics for package IDs to be the same as package keys is a really bad idea. AFAICT, it's the root cause of this bug in Cabal.)

Anyway, the workaround is somewhat painless: for each package we build, track if any of its dependencies need to be built as well, and in such a case force a reconfigure, even though none of the flags to Setup.hs configure have changed. I've written up such a patch.

[snoyberg]

Merged to master, closing.

[ndmitchell]

Note this is hitting real commercial users on a daily basis, so might deserve a quick point release.

[crockeea]

@snoyberg does this resolve the issue I reported, or "some other issue with shadowed dependencies"?

[snoyberg]

@crockeea AFAIK, it solves your issue. If it doesn't, please say so.

@ndmitchell We're in the middle of putting together a prerelease announcement for Stack 1.6.0, which will include this fix. It would be great if you'd be able to test that release (or just do a stack upgrade --git).

[ntc2]

The prerelease can be binary installed with

stack upgrade --binary-version 1.6.0.20171022

Would be useful to document this at the top of the prerelease description.

[ghorn]

I'm still getting shadowed dependencies on my build at work. I'm on stack 1.6.3 which I'd thought had the workaround. (Workaround merged Oct 21, stack 1.6.3 released Dec 23.)

[mgsloan]

@ghorn It does have it. While ideally this'd never be necessary, maybe there is some lingering brokenness in your package DBs, perhaps try renaming ~/.stack (can copy or move ~/.stack/programs to avoid redownloading ghc), and also renaming your project's .stack-work. Then can try a rebuild. If that doesn't fix it, then can delete the new dirs and rename the old ones back so that you haven't blown away your package caches. Might be interesting to try stack exec -- ghc-pkg check as well.

[JacquesCarette]

Just want to say that with stack 1.6.5, GHC 8.2.2, cabal 1.24.0.2, on Mac OSX, this is happening to me. No reconfiguration step involved. My overall setup is very much like @alexkazik 's.

---

Edit:

1. if I stack clean the offending packages, it works.
2. It is true, however, that the offending packages changed in a non-trivial way (changed signatures); but I did explicitly change the packages' .cabal files. So that ought to have been enough.