New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Benchmarks fail due to "shadowed dependencies" #2781

Closed
crockeea opened this Issue Nov 15, 2016 · 26 comments

Comments

Projects
None yet
@crockeea

crockeea commented Nov 15, 2016

This problem is described in a Stackage Server issue, but the problem should really be reported here. The problem is this:

library a
library b depends on a
benchmarks for library a depend on a and b

stack bench a results in the error:

> stack bench a
...
Preprocessing library a-0.0.1...
Preprocessing benchmark 'a-benches' for a-0.0.1...
<command line>: cannot satisfy -package-id b-0.0.1-2uppVa4SsB06R8hiHGD1I8: 
    b-0.0.1-2uppVa4SsB06R8hiHGD1I8 is unusable due to shadowed dependencies:
      a-0.0.1-B41UI81aQaCEjW3sk56eaB
    (use -v for more information)
Progress: 3/4
--  While building package a-0.0.1 using:
      /home/eric/.stack/setup-exe-cache/x86_64-linux/setup-Simple-Cabal-1.24.0.0-ghc-8.0.1 --builddir=.stack-work/dist/x86_64-linux/Cabal-1.24.0.0 build bench:a-benches --ghc-options " -ddump-hi -ddump-to-file"
    Process exited with code: ExitFailure 1

Note that the benchmarks compile if a-benches only depends on a or b, but not both. I don't see any reason why the benchmarks should fail to compile when they depend on both a and b; it's clear what the compilation order should be.

As indicated in the Stackage Server issue, this problem affects many packages including

  • text
  • unordered-containers
  • clock
  • options
  • attoparsec
  • wai-logger

A minimal example is in shadowed-deps.tar.gz.

@mgsloan mgsloan added the type: bug label Nov 19, 2016

@mgsloan mgsloan modified the milestones: P2: Should, P1: Must Nov 19, 2016

@snoyberg

This comment has been minimized.

Contributor

snoyberg commented Sep 5, 2017

@lehins pointed this out to me. I'm looking at logs that look like this:

$ stack bench a
Downloaded lts-7.0 build plan.    
a-0.0.1: configure
a-0.0.1: build
a-0.0.1: copy/register
b-0.0.1: configure (lib)
b-0.0.1: build (lib)
b-0.0.1: copy/register
a-0.0.1: configure (bench)
Configuring a-0.0.1...
a-0.0.1: build (bench)
Preprocessing library a-0.0.1...
Preprocessing benchmark 'a-benches' for a-0.0.1...

And then the shadowing messages. From what I can see, there's nothing we can do in Stack to work around this. It seems like:

  • We configure/build library a, and get package ID a-123
  • We configure/build library b, and get package ID b-456
  • Now we go back and configure a for building benchmarks, and explicitly tell Cabal to only build the benchmark component. However, Cabal builds both the library and benchmark, ends up with a different package ID, and shadowing occurs.

I could be mistaken, perhaps something else it happening. But it looks like the only solution will be switching to Cabal 2.0 and using the componentized configuration, which allows for not rebuilding a library.

@alexkazik

This comment has been minimized.

alexkazik commented Sep 5, 2017

I have a different problem the shadowing:

  • I have 8 packages (within one project with only one stack.yaml for them all but each in a own subdirectory, 7 are only libraries and the 8th lib+exe, no test so far)
  • They all have clear dependencies (nothing circular)
  • When I compile from scratch it usually works
  • If I change some sources it sometimes fails due to shadowing, stack clean usually does the trick
  • Sometimes only removing the .stack-work directory helps

EDIT: with nightly-2017-09-03 (ghc 8.2)

@lehins

This comment has been minimized.

Contributor

lehins commented Sep 5, 2017

@alexkazik I stumble upon exactly the same issue recently, which only happens for me with ghc-8.2 (current nightly snapshot), but not ghc-8.0. Naturally cabal versions used are also different.
The workaround that took care of this for me is touch package-name.cabal.

@lehins

This comment has been minimized.

Contributor

lehins commented Sep 5, 2017

@snoyberg Just in case, here are the steps using massiv library that reliably replicate the issue:
https://github.com/lehins/massiv/tree/5f3d083a343b22e866656e22036d42d294116d4d

  • Run benchmarks first time, which will compile everything and run fine:
~/massiv/massiv-bench$ stack --stack-yaml=stack-ghc-8.2.yaml bench massiv-bench:fold-seq --benchmark-arguments='--match prefix Sum'
  • Change a file in the source library, say comment out a function zipWith3 in massiv/massiv/src/Data/Array/Massiv.hs

  • Try running benchmarks again:

~/massiv/massiv-bench$ stack --stack-yaml=stack-ghc-8.2.yaml bench massiv-bench:fold-seq --benchmark-arguments='--match prefix Sum'
massiv-0.1.0.0: unregistering (local file changes: src/Data/Array/Massiv.hs)
massiv-bench-0.1.0.0: unregistering (missing dependencies: massiv)
massiv-0.1.0.0: build (lib)
massiv-0.1.0.0: copy/register
massiv-bench-0.1.0.0: build (lib + bench)
Preprocessing library for massiv-bench-0.1.0.0..
Building library for massiv-bench-0.1.0.0..
[3 of 6] Compiling Bench.Massiv.Auto ( src/Bench/Massiv/Auto.hs, .stack-work/dist/x86_64-linux/Cabal-2.0.0.2/build/Bench/Massiv/Auto.o ) [Data.Array.Massiv changed]
Preprocessing benchmark 'fold-seq' for massiv-bench-0.1.0.0..
Building benchmark 'fold-seq' for massiv-bench-0.1.0.0..
<command line>: cannot satisfy -package-id massiv-bench-0.1.0.0-4cUIceH4fry9LCxUPcSNIr: 
    massiv-bench-0.1.0.0-4cUIceH4fry9LCxUPcSNIr is unusable due to shadowed dependencies:
      massiv-0.1.0.0-DkTbpETOx8EEaw3SIP44Bw
    (use -v for more information)
Progress: 2/3
--  While building package massiv-bench-0.1.0.0 using:
      /home/lehins/.stack/setup-exe-cache/x86_64-linux/Cabal-simple_mPHDZzAJ_2.0.0.2_ghc-8.2.1 --builddir=.stack-work/dist/x86_64-linux/Cabal-2.0.0.2 build lib:massiv-bench bench:fold-seq --ghc-options " -ddump-hi -ddump-to-file"
    Process exited with code: ExitFailure 1
  • Change cabal file for benchmarks suite:
~/massiv/massiv-bench$ touch massiv-bench.cabal
  • Everything works fine again:
~/massiv/massiv-bench$ stack --stack-yaml=stack-ghc-8.2.yaml bench massiv-bench:fold-seq --benchmark-arguments='--match prefix Sum'
massiv-bench-0.1.0.0: configure (lib + bench)
Configuring massiv-bench-0.1.0.0...
massiv-bench-0.1.0.0: build (lib + bench)
Preprocessing library for massiv-bench-0.1.0.0..
Building library for massiv-bench-0.1.0.0..
Preprocessing benchmark 'fold-seq' for massiv-bench-0.1.0.0..
Building benchmark 'fold-seq' for massiv-bench-0.1.0.0..
[1 of 1] Compiling Main             ( bench/FoldSeq.hs, .stack-work/dist/x86_64-linux/Cabal-2.0.0.2/build/fold-seq/fold-seq-tmp/Main.o ) [Data.Array.Massiv changed]
Linking .stack-work/dist/x86_64-linux/Cabal-2.0.0.2/build/fold-seq/fold-seq ...
massiv-bench-0.1.0.0: copy/register
Installing library in /home/lehins/github/massiv/massiv-bench/.stack-work/install/x86_64-linux/nightly-2017-07-31/8.2.1/lib/x86_64-linux-ghc-8.2.1/massiv-bench-0.1.0.0-4cUIceH4fry9LCxUPcSNIr
Registering library for massiv-bench-0.1.0.0..
massiv-bench-0.1.0.0: benchmarks
Running 1 benchmarks...
...
@ndmitchell

This comment has been minimized.

Contributor

ndmitchell commented Sep 6, 2017

FWIW, I'm now seeing this bug very regularly, with GHC 8.2 on Windows. Things like touching the cabal file sometimes work, but are far from reliable in my experience.

@snoyberg

This comment has been minimized.

Contributor

snoyberg commented Sep 6, 2017

I received a similar bug report from a new Yesod users who was using Haskell Platform and cabal-install. I'm concerned that it may be a regression in Cabal 2.0. I'll try to repro this tomorrow. My goal is to get a minimal case and then test it with both cabal-install and plain old Setup.hs. If anyone else beats me to it, let me know.

@snoyberg

This comment has been minimized.

Contributor

snoyberg commented Sep 7, 2017

I've created a minimal reproducing case for this:

https://github.com/snoyberg/stack-issue-2781

I'm going to see if I can further reproduce this using just the Setup.hs files.

@ndmitchell

This comment has been minimized.

Contributor

ndmitchell commented Sep 7, 2017

For info, I've had this happen with benchmarks and test suites disabled, so just running stack build. Not sure if that would make your test case any simpler.

@snoyberg

This comment has been minimized.

Contributor

snoyberg commented Sep 7, 2017

I was able to reproduce this with just the Setup.hs files, so this is definitely not a bug in Stack. This bug is already being tracked upstream with haskell/cabal#4728.

@snoyberg

This comment has been minimized.

Contributor

snoyberg commented Sep 7, 2017

@ndmitchell I just tried simplifying my repro to not use the benchmark and was unable to reproduce. If you have thoughts on how to simplify it, please do take a crack at it, I'm sure it will help the debugging.

@ndmitchell

This comment has been minimized.

Contributor

ndmitchell commented Sep 7, 2017

While I see it semi-regularly, I never figured out a pattern, so no suggestions I'm afraid.

@wraithm

This comment has been minimized.

wraithm commented Sep 21, 2017

I've definitely started running into this issue quite a bit since upgrading to GHC 8.2

@phadej

This comment has been minimized.

Contributor

phadej commented Sep 26, 2017

I have a hunch that using allow-newer: true makes things pop-up more often. (and actually using deps which need allow-newer)

@phadej

This comment has been minimized.

Contributor

phadej commented Sep 26, 2017

... or having snapshot package depending on local package

@phadej

This comment has been minimized.

Contributor

phadej commented Sep 26, 2017

and specifying --haddock. Weird.

@treblacy

This comment has been minimized.

treblacy commented Oct 5, 2017

Regarding the life story of cabal-repro.sh:

  1. After the first registering of foo, please do a

    ghc-pkg --package-db=$PKGDB field foo abi
    

    to see its ABI hash. If you're adding this command to cabal-repro.sh, maybe add > abi1.log so you can find it easily later.

  2. When configuring bar, the above ABI hash of foo is noted and recorded as part of the dependency.

  3. After the second registering of foo, please again do a

    ghc-pkg --package-db=$PKGDB field foo abi
    

    (maybe with > ab12.log). The ABI hash changes because the short code of foo changes and it's so short it's a good candidate for getting inlined so it is part of Foo.hi so it affects the ABI hash.

  4. Now you build bar without reconfiguring. First the library component is built, which is fine for now, but it is recorded to depend on the 1st foo ABI hash, not the 2nd.

    Next, there are two cases, but eventually I will pull a disjunction-elimination to draw the same conclusion.

    1. You had --enable-benchmarks. Now the benchmark component is built, which depends on bar, but bar depends on a non-existent foo ABI hash. This is when you get an error.

    2. You didn't have --enable benchmarks. So no one uses bar for now, so no one notices anything wrong. But please press ahead with

       runghc Setup.hs register
       ghc -v -no-user-package-db -package-db=$PKGDB
      

      to see the same message about shadowing (or more accurately non-existence—"shadowing" is a pretty old error message). This means the problem has been created and is ready to bite.

      Also of interest is the output of

       ghc-pkg --package-db=$PKGDB field bar abi-depends
      

    Either way you have/will run into an error when you try to use bar for real.

I don't know what you should make of it. If you configure a package, and change something in the environment, then come back to resume building—but you don't reconfigure—this sounds like obviously risky to me. But maybe you have a need for it, and you could say that Cabal and GHC are being too paranoid. Or you could cherish this basic conservative sanity check and call it "this is not a bug, this is a feature". Or maybe a bit of both---perhaps configure time is too early to record other packages' ABI hashes, perhaps build time is better.

I do not know Stack and I do not know the life story of repro.sh.

@ndmitchell

This comment has been minimized.

Contributor

ndmitchell commented Oct 7, 2017

Note that the Cabal ticket says that only the full platform is affected, and only on non-Windows. Given I'm getting this issue from Windows with Stack using the raw GHC then either they've misdiagnosed the issue, or there are multiple issues causing the same error message.

snoyberg added a commit that referenced this issue Oct 22, 2017

snoyberg added a commit that referenced this issue Oct 22, 2017

@snoyberg

This comment has been minimized.

Contributor

snoyberg commented Oct 22, 2017

Thanks for the analysis @treblacy. This is in line with @ezyang's comment here: haskell/cabal#4728 (comment). The problem seems to be that Cabal's otherwise-working auto-reconfigure logic isn't kicking in. And Stack isn't noticing that it should force a reconfigure since the package ID hasn't changed.

(Side note: this is another reason I think the change to semantics for package IDs to be the same as package keys is a really bad idea. AFAICT, it's the root cause of this bug in Cabal.)

Anyway, the workaround is somewhat painless: for each package we build, track if any of its dependencies need to be built as well, and in such a case force a reconfigure, even though none of the flags to Setup.hs configure have changed. I've written up such a patch.

@snoyberg snoyberg referenced this issue Oct 22, 2017

Merged

Force reconfigure when deps are rebuilt #2781 #3503

2 of 2 tasks complete

snoyberg added a commit that referenced this issue Oct 22, 2017

Merge pull request #3503 from commercialhaskell/2781-shadow-bug
Force reconfigure when deps are rebuilt #2781
@snoyberg

This comment has been minimized.

Contributor

snoyberg commented Oct 22, 2017

Merged to master, closing.

@snoyberg snoyberg closed this Oct 22, 2017

@ndmitchell

This comment has been minimized.

Contributor

ndmitchell commented Oct 22, 2017

Note this is hitting real commercial users on a daily basis, so might deserve a quick point release.

@crockeea

This comment has been minimized.

crockeea commented Oct 24, 2017

@snoyberg does this resolve the issue I reported, or "some other issue with shadowed dependencies"?

@snoyberg

This comment has been minimized.

Contributor

snoyberg commented Oct 24, 2017

@crockeea AFAIK, it solves your issue. If it doesn't, please say so.

@ndmitchell We're in the middle of putting together a prerelease announcement for Stack 1.6.0, which will include this fix. It would be great if you'd be able to test that release (or just do a stack upgrade --git).

@ntc2

This comment has been minimized.

Contributor

ntc2 commented Oct 27, 2017

The prerelease can be binary installed with

stack upgrade --binary-version 1.6.0.20171022

Would be useful to document this at the top of the prerelease description.

tswelsh added a commit to tswelsh/stack that referenced this issue Nov 7, 2017

tswelsh added a commit to tswelsh/stack that referenced this issue Nov 7, 2017

@ghorn

This comment has been minimized.

ghorn commented Jan 18, 2018

I'm still getting shadowed dependencies on my build at work. I'm on stack 1.6.3 which I'd thought had the workaround. (Workaround merged Oct 21, stack 1.6.3 released Dec 23.)

@mgsloan

This comment has been minimized.

Collaborator

mgsloan commented Jan 18, 2018

@ghorn It does have it. While ideally this'd never be necessary, maybe there is some lingering brokenness in your package DBs, perhaps try renaming ~/.stack (can copy or move ~/.stack/programs to avoid redownloading ghc), and also renaming your project's .stack-work. Then can try a rebuild. If that doesn't fix it, then can delete the new dirs and rename the old ones back so that you haven't blown away your package caches. Might be interesting to try stack exec -- ghc-pkg check as well.

@JacquesCarette

This comment has been minimized.

JacquesCarette commented Aug 10, 2018

Just want to say that with stack 1.6.5, GHC 8.2.2, cabal 1.24.0.2, on Mac OSX, this is happening to me. No reconfiguration step involved. My overall setup is very much like @alexkazik 's.


Edit:

  1. if I stack clean the offending packages, it works.
  2. It is true, however, that the offending packages changed in a non-trivial way (changed signatures); but I did explicitly change the packages' .cabal files. So that ought to have been enough.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment