"Cannot satisfy" errors when copying precompiled packages with custom snapshots

[hesselink]

We have custom snapshots that we treat as immutable and rename when upgrading. Stack sees that packages are the same and copies precompiled versions. However, this seems to lead to errors like this:

> stack build --test --bench
...
aeson-0.9.0.1: copying precompiled package
...
    <command line>: cannot satisfy -package-id aeson-0.9.0.1-b5b185082465176d844de3bc8c6b4b08:
        aeson-0.9.0.1-b5b185082465176d844de3bc8c6b4b08 is unusable due to missing or recursive dependencies:
          vector-0.10.12.3-4a6464d91c95dd62d2822a0831fdfc6b

This makes sense, because the hash for vector in this sandbox is different:

> stack exec -- ghc-pkg describe vector

name: vector
version: 0.10.12.3
id: vector-0.10.12.3-3f8182e905e7fa382a931cb4031c9dd2

It turns out the hash is from the previous snapshot:

<switch back to old code>
> stack exec ghc-pkg describe vector

name: vector
version: 0.10.12.3
id: vector-0.10.12.3-4a6464d91c95dd62d2822a0831fdfc6b

This is after I completely wiped by ~/.stack and all .stack-work directories in the project. I can reproduce by unregistering all the packages stack exec -- ghc-pkg check reports as broken, and trying again: it copies the package again, and gives the same error.

[hesselink]

Forgot to note: the version of things I'm using:

$ stack --version
Version 0.1.5.0 X86_64
$ ghc --version
The Glorious Glasgow Haskell Compilation System, version 7.8.3
$ ghc-pkg list
/Library/Frameworks/GHC.framework/Versions/7.8.3-x86_64/usr/lib/ghc-7.8.3/package.conf.d
   Cabal-1.18.1.3
   array-0.5.0.0
   base-4.7.0.1
   bin-package-db-0.0.0.0
   binary-0.7.1.0
   bytestring-0.10.4.0
   containers-0.5.5.1
   deepseq-1.3.0.2
   directory-1.2.1.0
   filepath-1.3.0.2
   ghc-7.8.3
   ghc-prim-0.3.1.0
   haskeline-0.7.1.2
   haskell2010-1.1.2.0
   haskell98-2.0.0.3
   hoopl-3.10.0.1
   hpc-0.6.0.1
   integer-gmp-0.5.1.0
   mtl-2.1.3.1
   old-locale-1.0.0.6
   old-time-1.1.0.2
   parsec-3.1.9
   pretty-1.1.1.1
   process-1.2.0.0
   rts-1.0
   setenv-0.1.1.3
   tar-0.4.1.0
   template-haskell-2.9.0.0
   terminfo-0.4.0.0
   text-1.2.0.6
   time-1.4.2
   transformers-0.3.0.0
   unix-2.7.0.1
   xhtml-3000.2.1
   zlib-0.5.4.2

[snoyberg]

Can you provide a test case with a minimalist custom snapshot that can trigger this behavior?

[hesselink]

No, sadly I can't seem to. I've tried to reproduce in a test package which depends on aeson, and switching from one snapshot to another, but that just worked (it didn't copy the precompiled thing, since the transitive deps were different). I couldn't even reproduce it when compiling only part of our app, only when compiling the whole thing from scratch (all 360 packages, which takes a long time). What I'm doing is:

• Building everything (with --test --bench) with snapshot A.
• Switching to snapshot B, which upgrades primitive from 0.6 to 0.6.1.0. That's a dependency of vector (not upgraded) which is a dependency of aeson.
• Build everything again (with --test --bench).

In the output I see it rebuilding vector (because of the changed dep) but then copying aeson. Could it be that when checking if it can copy a prebuilt artefact, it only checks the direct dep versions, and not the transitive ones?

[snoyberg]

It's based on installed package ID. This could be a result of a truly annoying Cabal bug that results in duplicated installed package IDs despite different inputs, see:

haskell/cabal#2830

Can you see if you have two identical package IDs for any of the packages involved (likely vector) appearing in different databases with different dependencies?

[hesselink]

Not for vector, this was the first thing I looked at (see above). Primitive has actual different versions, so I doubt that one is the culprit.

I've since wiped the snapshots again and restored the old state, to be able to continue working. Is there any other package I should check? I'll perhaps be able to do more testing tonight.

[snoyberg]

I can't think of anything in particular. If you can repro and then check
for a duplicate ipid, that would at least let us know if my guess might be
correct.

On Tue, Oct 6, 2015, 4:27 PM Erik Hesselink notifications@github.com
wrote:

Not for vector, this was the first thing I looked at (see above).
Primitive has actual different versions, so I doubt that one is the culprit.

I've since wiped the snapshots again and restored the old state, to be
able to continue working. Is there any other package I should check? I'll
perhaps be able to do more testing tonight.

—
Reply to this email directly or view it on GitHub
#1103 (comment)
.

[hesselink]

I've checked all the package ids of aeson and its (transitive) dependencies. All of them are the same, except for primitive and vector.

I still don't quite understand what's going on. I see stack installing primitive with a different version from the previous snapshot and a different package id. Then it installs vector, with the same version but a different package id. Then it has to decide what to do about aeson. Based on the package ids of its direct dependencies, it should decide not to copy it, since vector has a different one. What's going on?

For completeness, here's the package info:

Old snapshot:

id: aeson-0.9.0.1-b5b185082465176d844de3bc8c6b4b08
depends: attoparsec-0.13.0.1-3c63e96327b344cdb033884fc4a71d63
         base-4.7.0.1-c64d224738ec7af4085e89ca9c12c37b
         bytestring-0.10.4.0-18fe2f3ce284617c82da1702e16772cf
         containers-0.5.5.1-23e2a2b94d6e452c773209f31d8672c5
         deepseq-1.3.0.2-733fe43e121f761739636bd6670bea77
         dlist-0.7.1.2-5326dd34961083a4ce3915215d4c3461
         ghc-prim-0.3.1.0-954cb57749cf319beafdc89b3415422c
         hashable-1.2.3.3-7c8f3595877949e4c5bea68de662b88f
         mtl-2.2.1-05c98059bc3926647251ab9e6cb3164e
         old-locale-1.0.0.6-09baf1dbc5be8338e5eba7c5bb515505
         scientific-0.3.3.8-8a756b5ca3d375459bd26f109ff55b4a
         syb-0.5.1-905dff0f71b66bed728ccbc86436488b
         template-haskell-2.9.0.0-310613e57f38a482326cc4ba2989009e
         text-1.2.1.3-2b2666f8f5002465096b6f348f152d84
         time-1.4.2-bf925e935c287d0b75398fe297453c28
         transformers-0.4.3.0-4bf279c7bbcd18fdf67f75df501d1fdf
         unordered-containers-0.2.5.1-68212d710817e37880a51101b20b3308
         vector-0.10.12.3-4a6464d91c95dd62d2822a0831fdfc6b

id: vector-0.10.12.3-4a6464d91c95dd62d2822a0831fdfc6b
depends: base-4.7.0.1-c64d224738ec7af4085e89ca9c12c37b
         deepseq-1.3.0.2-733fe43e121f761739636bd6670bea77
         ghc-prim-0.3.1.0-954cb57749cf319beafdc89b3415422c
         primitive-0.6-0c47dfdc4b370764c79d5a34f7d98a91

id: primitive-0.6-0c47dfdc4b370764c79d5a34f7d98a91
depends: base-4.7.0.1-c64d224738ec7af4085e89ca9c12c37b
         ghc-prim-0.3.1.0-954cb57749cf319beafdc89b3415422c
         transformers-0.4.3.0-4bf279c7bbcd18fdf67f75df501d1fdf

New snapshot:

id: aeson-0.9.0.1-b5b185082465176d844de3bc8c6b4b08
depends: attoparsec-0.13.0.1-3c63e96327b344cdb033884fc4a71d63
         base-4.7.0.1-c64d224738ec7af4085e89ca9c12c37b
         bytestring-0.10.4.0-18fe2f3ce284617c82da1702e16772cf
         containers-0.5.5.1-23e2a2b94d6e452c773209f31d8672c5
         deepseq-1.3.0.2-733fe43e121f761739636bd6670bea77
         dlist-0.7.1.2-5326dd34961083a4ce3915215d4c3461
         ghc-prim-0.3.1.0-954cb57749cf319beafdc89b3415422c
         hashable-1.2.3.3-7c8f3595877949e4c5bea68de662b88f
         mtl-2.2.1-05c98059bc3926647251ab9e6cb3164e
         old-locale-1.0.0.6-09baf1dbc5be8338e5eba7c5bb515505
         scientific-0.3.3.8-8a756b5ca3d375459bd26f109ff55b4a
         syb-0.5.1-905dff0f71b66bed728ccbc86436488b
         template-haskell-2.9.0.0-310613e57f38a482326cc4ba2989009e
         text-1.2.1.3-2b2666f8f5002465096b6f348f152d84
         time-1.4.2-bf925e935c287d0b75398fe297453c28
         transformers-0.4.3.0-4bf279c7bbcd18fdf67f75df501d1fdf
         unordered-containers-0.2.5.1-68212d710817e37880a51101b20b3308
         vector-0.10.12.3-4a6464d91c95dd62d2822a0831fdfc6b

id: vector-0.10.12.3-3f8182e905e7fa382a931cb4031c9dd2
depends: base-4.7.0.1-c64d224738ec7af4085e89ca9c12c37b
         deepseq-1.3.0.2-733fe43e121f761739636bd6670bea77
         ghc-prim-0.3.1.0-954cb57749cf319beafdc89b3415422c
         primitive-0.6.1.0-625f00a7e72512c1e252b23f6931cc40

id: primitive-0.6.1.0-625f00a7e72512c1e252b23f6931cc40
depends: base-4.7.0.1-c64d224738ec7af4085e89ca9c12c37b
         ghc-prim-0.3.1.0-954cb57749cf319beafdc89b3415422c
         transformers-0.4.3.0-4bf279c7bbcd18fdf67f75df501d1fdf

[hesselink]

I've finally got a reproduction! To reproduce:

• Download this file and unpack it somewhere.
• Temporarily move your ~/.stack out of the way, so stack doesn't copy other already installed packages.
• In the unpack location, issue stack build.
• Edit stack.yaml to rename the snapshot and switch the file to snapshot-2.yaml.
• Issue stack build again.

This should fail when installing statistics with the message:

    Configuring statistics-0.13.2.3...
    setup-Simple-Cabal-1.18.1.3-x86_64-osx-ghc-7.8.3: The following installed
    packages are broken because other packages they depend on are missing. These
    broken packages must be rebuilt before they can be used.
    package aeson-0.9.0.1 is broken due to missing package
    vector-0.10.12.3-4a6464d91c95dd62d2822a0831fdfc6b
    package math-functions-0.1.5.2 is broken due to missing package
    vector-0.10.12.3-4a6464d91c95dd62d2822a0831fdfc6b
    package monad-par-0.3.4.7 is broken due to missing package
    mwc-random-0.13.3.2-2b81aa2f75c1529efd0d20df67f69688
    package vector-binary-instances-0.2.1.0 is broken due to missing package
    vector-0.10.12.3-4a6464d91c95dd62d2822a0831fdfc6b
    package vector-th-unbox-0.2.1.2 is broken due to missing package
    vector-0.10.12.3-4a6464d91c95dd62d2822a0831fdfc6b

[snoyberg]

Thanks, the repro triggers the problem for me. The issue seems to be with mwc-random. Still investigating.

[snoyberg]

OK, I've found the problem. It's specifically related to Cabal 1.18; upgrading to Cabal 1.22 would work around it. The issue is that we're checking the configure options for the precompiled cache, but I forgot that - in Cabal 1.18 - the configure options don't give the installed package ID. Fix should be simple, though I'll have to be careful to avoid invalidating caches for Cabal 1.22.

[snoyberg]

OK, I think this should be fixed on master, can you check? And thanks for the repro, I wouldn't have been able to figure this out otherwise.

[hesselink]

Yes, that fixed it for me, both for the repro case and for our actual code base. Thanks!