Stack doesn't find custom snapshot package

[nmattia]

General summary/comments (optional)

I cannot pull git: defined packages in a custom snapshot. The package version is looked up from the custom snapshot correctly, but then stack tries to find it on hackage. Stack does download the source but it is never built.

This is the (failing) lookup, as far as I can tell: https://github.com/commercialhaskell/stack/blob/master/src/Stack/Package.hs#L200

CC @mgsloan

Steps to reproduce

Clone this repo, run stack build reflex. The repo specifies a minimal stack.yaml with a custom snapshot:

stack.yaml

resolver: snapshot.yaml

snapshot.yaml

resolver: lts-9.9
name: Dummy
packages:
- git: https://github.com/reflex-frp/reflex
  commit: d9ef8457dfd140fde32d15a6959a15ddcd7e15e4

Expected

Stack should have tried to build the reflex package.

Actual

stack build reflex
Cloning d9ef8457dfd140fde32d15a6959a15ddcd7e15e4 from http://github.com/reflex-frp/reflex.git
Cloning into '/home/nicolas/projects/stack-custom-test/.stack-work/downloaded/zoCbUP3YjykH'...
remote: Counting objects: 3473, done.
remote: Total 3473 (delta 0), reused 0 (delta 0), pack-reused 3472
Receiving objects: 100% (3473/3473), 910.78 KiB | 325.00 KiB/s, done.
Resolving deltas: 100% (2130/2130), done.
Checking connectivity... done.
Didn't see reflex-0.5.0 in your package indices.
Updating and trying again.
Selected mirror https://s3.amazonaws.com/hackage.fpcomplete.com/                                 
Downloading timestamp                                                                            
No updates to your package index were found                                                      
Update complete                                                                                  
The following package identifiers were not found in your indices: reflex-0.5.0
Possible candidates: reflex-0.2, reflex-0.4.0, reflex-0.0.1, reflex-0.3.2, reflex-0.3, reflex-0.1.1, reflex-0.1.0, reflex-0.3.1.

See the verbose output.

Stack version

Stack built from source at commit 8266a06, although reproducible with Stack 1.6.3

[nmattia]

It turns out that the reflex package gets a bad PackageLocationIndex at some point.

Here's what I was able to gather:

• loadSnapshot is called to prepare the snapshot for a build. The reflex package has the correct package location index.
• calculatePackagePromotion is called. The reflex package is an element of parentPackages0, parentPackages1 and parentPackages2.
• As such, it goes through splitUnmetDeps.
• splitUnmetDeps flags it to have unmet dependencies, and the package goes through globalToSnapshot.
• In globalToSnapshot the package location index is lost and overwritten:

globalToSnapshot name lpi = lpi
    { lpiLocation = PLIndex (PackageIdentifierRevision (PackageIdentifier name (lpiVersion lpi)) CFILatest)
    }

It may well be that the package has unmet dependencies, however I'm surprised that stack goes ahead with the build by trying to refetch the package from Hackage rather than aborting. Moreover I've seen the issue in a project where the incriminated package had all its dependencies either in the parent snapshot (LTS) or in the custom snapshot.

@snoyberg you seem to be the sole contributor to the Stack.Snapshot module. Any thoughts on this?

[snoyberg]

Definitely sounds like a bug, thanks for the thorough investigation here. If it really does have unmet deps, then we should be complaining about that. If it doesn't have unmet deps, then it's a bigger bug. Do you have an example of a case where all of the dependencies were in fact met?

[nmattia]

I'm able to reproduce when all dependencies are met:

stack.yaml

resolver: snapshot.yaml
extra-deps:
- prim-uniq-0.1.0.1

snapshot.yaml

resolver: lts-9.9
name: Dummy
packages:
- prim-uniq-0.1.0.1
- ref-tf-0.4.0.1
- git: https://github.com/reflex-frp/reflex
  commit: d9ef8457dfd140fde32d15a6959a15ddcd7e15e4

$ stack build reflex
Didn't see reflex-0.5.0 in your package indices.
Updating and trying again.
Selected mirror https://s3.amazonaws.com/hackage.fpcomplete.com/                                 
Downloading timestamp                                                                            
No updates to your package index were found                                                      
Update complete                                                                                  
The following package identifiers were not found in your indices: reflex-0.5.0
Possible candidates: reflex-0.2, reflex-0.4.0, reflex-0.0.1, reflex-0.3.2, reflex-0.3, reflex-0.1.1, reflex-0.1.0, reflex-0.3.1.

Note that

• If I move reflex from snapshot.yaml's packages: to stack.yaml's extra-deps: (with syntax change) then stack is able to build reflex
• stack somehow still assumes that dependencies are unmet, here's the content of noLongerParent2 : fromList [(reflex,PLIndex "reflex-0.5.0")]

[nmattia]

In the comment above, I didn't realize I had prim-uniq both in extra-deps and the snapshot's packages. I've removed it and I'm now seeing something strange with the following config:

stack.yaml

resolver: snapshot.yaml

snapshot.yaml

resolver: lts-9.9
name: Dummy
packages:
- prim-uniq-0.1.0.1
- ref-tf-0.4.0.1
- git: https://github.com/reflex-frp/reflex
  commit: d9ef8457dfd140fde32d15a6959a15ddcd7e15e4

Running stack build reflex will work the first time. However it will fail on all subsequent attempts. But if at least one character is changed in snapshot.yaml (like added empty line, trailing whitespace, comment, etc) the build will be successful (on the first try, once again). Two thoughts:

• This sounds like a caching issue, where stack is able to work with a clean (empty) cache but is having issues when a cache is already present
• (Unrelated to the problem at hand) Shouldn't the parsed snapshot.yaml be hashed (the Haskell representation of the content) rather than the file content itself?

[snoyberg]

This is definitely a bug that needs to be resolved, I've marked it as a blocker for the next release of Stack. I'm currently working on a number of other refactorings in the codebase, and won't be able to get to this immediately myself. A great next step (if you're up for the challenge) is to add a minimal reproducing case as an integration test. The minimal repro (something ideally smaller than reflex) will be massively useful for debugging this, and I'd be happy to help convert such a repro into an integration test. You interested in pursuing this?

[nmattia]

I'll try to find the time, I'll update you here when I get started.

[DanBurton]

I believe I've reproduced the same bug with the hspec megarepo. Interestingly, I can build the hspec-core project in a subdir, but not the hspec project in the root dir. In this example, the commit is the latest commit on hspec master.

$ stack --version
Version 1.6.1, Git revision f25811329bbc40b0c21053a8160c56f923e1201b (5435 commits) x86_64 hpack-0.20.0

$ cat stack.yaml
resolver: snapshot.yaml
packages: []

$ cat snapshot.yaml
resolver: nightly-2018-01-04
name: repro-snap
packages:
- git: git@github.com:hspec/hspec
  commit: 65b246ef0840e5b3f4cf3567c741d1edc33debb8
  subdirs:
  - '.'
  - hspec-core

$ stack build hspec-core --dry-run
# blah blah, works as expected

$ stack build hspec --dry-run
...blah blah...
Didn't see hspec-2.5.0 in your package indices.
...blah blah...
The following package identifiers were not found in your indices: hspec-2.5.0
...blah blah...

[DanBurton]

Same bugginess with the unordered-containers repo. When I have

# snapshot.yaml
resolver: nightly-2018-01-04
packages:
- git: git@github.com:tibbe/unordered-containers
  commit: 60ced060304840ed0bf368249ed6eb4e43d4cefc

It will build unordered-containers from Hackage rather than the repo, since the cabal file at that commit specifies a version which has been released. (Expected behavior would be to build from the repo regardless.)

If instead I use my patch which has bumped the version...

# snapshot.yaml
resolver: nightly-2018-01-04
packages:
- git: git@github.com:danburton/unordered-containers
  commit: d1f36251669687d96834df0c2a7bec1d162c1e89

Then it complains with the error message as mentioned above.

The following package identifiers were not found in your indices: unordered-containers-0.2.8.1

(Where, again, expected behavior is to build from the repo.)

[kadoban]

Edit: looks less like the same thing now that the fix below doesn't fix the issue I mentioned.

[snoyberg]

@nmattia I believe I've found the problem. Can you check out the 3714-discarded-package-location branch (PR #3759) and see if it resolves the issue for you?

[nmattia]

Seems to work, at least I can't reproduce the issue on the example above. Thanks a lot!

[DanBurton]

Seems to work for the use cases I mentioned as well.

[snoyberg]

Cool, thanks for confirming