dependency not getting installed in .stack-work with stack 1.3.0 #2849

Closed
wolftune opened this Issue Dec 15, 2016 · 25 comments

Projects

None yet

4 participants

@wolftune
Contributor

related to #2845 this is specifically occurring with http://github.com/snowdriftcoop/snowdrift

Tested on multiple systems (all Linux-based)

With Stack 1.2.0, everything is fine. With Stack 1.3.0, the yesod binary is not found. It's not getting put into .stack-work correctly.

If I build successfully with stack 1.2.0, I can keep the stack-work, then build with stack 1.3.0 and still have things work because the binary built before is still present.

This probably relates to snowdrift using an alternate forked version of yesod-bin with a git address. It seems stack 1.3.0 has a bug where it isn't installing binaries in stack-work correctly when the binary comes from an extra dependency…

@mitchellwrosen
Contributor

Hmm... I filed #2845, but I'm not sure it's related (or even a bug!). I was using 1.2.1, as noted in the ticket.

@mitchellwrosen
Contributor
mitchellwrosen commented Dec 15, 2016 edited

However, I can reproduce this bug using 1.3; here's how:

$ git clone http://github.com/snowdriftcoop/snowdrift
$ cd snowdrift
$ stack build yesod-bin
$ stack exec -- which yesod

Expected output: yesod installed somewhere in .stack-work/
Actual output: nothing

(I also tried stack build yesod-bin:exe:yesod - no dice)

@mitchellwrosen
Contributor

It also doesn't have to do with the forked yesod-bin package, as pointing at the actual repo at the proper tag (as below) exhibits the same behavior.

- location:
    git: https://github.com/yesodweb/yesod
    commit: d1495bad85565f75ce0238545dbec1d4257aba24
  subdirs:
    - yesod-bin
  extra-dep: true
@mitchellwrosen
Contributor
mitchellwrosen commented Dec 15, 2016 edited

Here's a log of stack build -v yesod-bin:exe:yesod --no-time-in-log:

Version 1.3.0 x86_64 hpack-0.15.0
[debug] Checking for project config at: /home/mitchell/snowdrift/stack.yaml
@(Stack/Config.hs:863:9)
[debug] Loading project config file stack.yaml
@(Stack/Config.hs:881:13)
[debug] Trying to decode /home/mitchell/.stack/build-plan-cache/x86_64-linux/lts-6.17.cache
@(Data/Store/VersionTagged.hs:68:5)
[debug] Success decoding /home/mitchell/.stack/build-plan-cache/x86_64-linux/lts-6.17.cache
@(Data/Store/VersionTagged.hs:72:13)
[debug] Run process: /sbin/ldconfig -p
@(System/Process/Read.hs:306:3)
[debug] Process finished in 0ms: /sbin/ldconfig -p
@(System/Process/Read.hs:306:3)
[debug] Run process: /usr/bin/gcc -v
@(System/Process/Read.hs:306:3)
[debug] Process finished in 0ms: /usr/bin/gcc -v
@(System/Process/Read.hs:306:3)
[debug] PIE enabled
@(Stack/Setup.hs:574:17)
[debug] Found shared library libtinfo.so.5 in /usr/lib
@(Stack/Setup.hs:563:38)
[debug] Found shared library libtinfo.so.6 in /usr/lib
@(Stack/Setup.hs:563:38)
[debug] Found shared library libncursesw.so.6 in 'ldconfig -p' output
@(Stack/Setup.hs:550:29)
[debug] Found shared library libgmp.so.10 in 'ldconfig -p' output
@(Stack/Setup.hs:550:29)
[debug] Did not find shared library libgmp.so.3
@(Stack/Setup.hs:564:38)
[debug] Using standard GHC build
@(Stack/Setup.hs:597:9)
[debug] Getting global package database location
@(Stack/GhcPkg.hs:55:5)
[debug] Run process: /home/mitchell/.stack/programs/x86_64-linux/ghc-7.10.3/bin/ghc-pkg --no-user-package-db list --global
@(System/Process/Read.hs:306:3)
[debug] Asking GHC for its version
@(Stack/Setup/Installed.hs:103:13)
[debug] Getting Cabal package version
@(Stack/GhcPkg.hs:170:5)
[debug] Run process: /home/mitchell/.stack/programs/x86_64-linux/ghc-7.10.3/bin/ghc --numeric-version
@(System/Process/Read.hs:306:3)
[debug] Run process: /home/mitchell/.stack/programs/x86_64-linux/ghc-7.10.3/bin/ghc-pkg --no-user-package-db field --simple-output Cabal version
@(System/Process/Read.hs:306:3)
[debug] Process finished in 13ms: /home/mitchell/.stack/programs/x86_64-linux/ghc-7.10.3/bin/ghc-pkg --no-user-package-db list --global
@(System/Process/Read.hs:306:3)
[debug] Process finished in 12ms: /home/mitchell/.stack/programs/x86_64-linux/ghc-7.10.3/bin/ghc-pkg --no-user-package-db field --simple-output Cabal version
@(System/Process/Read.hs:306:3)
[debug] Process finished in 23ms: /home/mitchell/.stack/programs/x86_64-linux/ghc-7.10.3/bin/ghc --numeric-version
@(System/Process/Read.hs:306:3)
[debug] Resolving package entries
@(Stack/Setup.hs:252:5)
[debug] Starting to execute command inside EnvConfig
@(Stack/Runners.hs:163:18)
[debug] Parsing the cabal files of the local packages
@(Stack/Build/Source.hs:298:5)
[debug] Parsing the targets
@(Stack/Build/Source.hs:235:5)
[debug] Exception ignored when attempting to load /home/mitchell/snowdrift/website/.stack-work/dist/x86_64-linux/Cabal-1.22.5.0/stack-build-cache: /home/mitchell/snowdrift/website/.stack-work/dist/x86_64-linux/Cabal-1.22.5.0/stack-build-cache: openBinaryFile: does not exist (No such file or directory)
@(Data/Store/VersionTagged.hs:86:9)
[debug] Start: getPackageFiles /home/mitchell/snowdrift/website/Snowdrift.cabal
@(Stack/Package.hs:250:21)
[debug] Finished in 6ms: getPackageFiles /home/mitchell/snowdrift/website/Snowdrift.cabal
@(Stack/Package.hs:250:21)
[debug] Exception ignored when attempting to load /home/mitchell/snowdrift/crowdmatch/.stack-work/dist/x86_64-linux/Cabal-1.22.5.0/stack-build-cache: /home/mitchell/snowdrift/crowdmatch/.stack-work/dist/x86_64-linux/Cabal-1.22.5.0/stack-build-cache: openBinaryFile: does not exist (No such file or directory)
@(Data/Store/VersionTagged.hs:86:9)
[debug] Start: getPackageFiles /home/mitchell/snowdrift/crowdmatch/crowdmatch.cabal
@(Stack/Package.hs:250:21)
[debug] Finished in 1ms: getPackageFiles /home/mitchell/snowdrift/crowdmatch/crowdmatch.cabal
@(Stack/Package.hs:250:21)
[debug] Exception ignored when attempting to load /home/mitchell/snowdrift/.stack-work/downloaded/6nk1N5bfncwY/persistent/.stack-work/dist/x86_64-linux/Cabal-1.22.5.0/stack-build-cache: /home/mitchell/snowdrift/.stack-work/downloaded/6nk1N5bfncwY/persistent/.stack-work/dist/x86_64-linux/Cabal-1.22.5.0/stack-build-cache: openBinaryFile: does not exist (No such file or directory)
@(Data/Store/VersionTagged.hs:86:9)
[debug] Start: getPackageFiles /home/mitchell/snowdrift/.stack-work/downloaded/6nk1N5bfncwY/persistent/persistent.cabal
@(Stack/Package.hs:250:21)
[debug] Finished in 4ms: getPackageFiles /home/mitchell/snowdrift/.stack-work/downloaded/6nk1N5bfncwY/persistent/persistent.cabal
@(Stack/Package.hs:250:21)
[debug] Exception ignored when attempting to load /home/mitchell/snowdrift/.stack-work/downloaded/Nl5BUGxz7nps/.stack-work/dist/x86_64-linux/Cabal-1.22.5.0/stack-build-cache: /home/mitchell/snowdrift/.stack-work/downloaded/Nl5BUGxz7nps/.stack-work/dist/x86_64-linux/Cabal-1.22.5.0/stack-build-cache: openBinaryFile: does not exist (No such file or directory)
@(Data/Store/VersionTagged.hs:86:9)
[debug] Start: getPackageFiles /home/mitchell/snowdrift/.stack-work/downloaded/Nl5BUGxz7nps/postgresql-simple-migration.cabal
@(Stack/Package.hs:250:21)
[debug] Finished in 4ms: getPackageFiles /home/mitchell/snowdrift/.stack-work/downloaded/Nl5BUGxz7nps/postgresql-simple-migration.cabal
@(Stack/Package.hs:250:21)
[debug] Exception ignored when attempting to load /home/mitchell/snowdrift/run-persist/.stack-work/dist/x86_64-linux/Cabal-1.22.5.0/stack-build-cache: /home/mitchell/snowdrift/run-persist/.stack-work/dist/x86_64-linux/Cabal-1.22.5.0/stack-build-cache: openBinaryFile: does not exist (No such file or directory)
@(Data/Store/VersionTagged.hs:86:9)
[debug] Start: getPackageFiles /home/mitchell/snowdrift/run-persist/run-persist.cabal
@(Stack/Package.hs:250:21)
[debug] Finished in 0ms: getPackageFiles /home/mitchell/snowdrift/run-persist/run-persist.cabal
@(Stack/Package.hs:250:21)
[debug] Exception ignored when attempting to load /home/mitchell/snowdrift/.stack-work/downloaded/bnIHMPumBlP2/yesod-bin/.stack-work/dist/x86_64-linux/Cabal-1.22.5.0/stack-build-cache: /home/mitchell/snowdrift/.stack-work/downloaded/bnIHMPumBlP2/yesod-bin/.stack-work/dist/x86_64-linux/Cabal-1.22.5.0/stack-build-cache: openBinaryFile: does not exist (No such file or directory)
@(Data/Store/VersionTagged.hs:86:9)
[debug] Start: getPackageFiles /home/mitchell/snowdrift/.stack-work/downloaded/bnIHMPumBlP2/yesod-bin/yesod-bin.cabal
@(Stack/Package.hs:250:21)
[debug] Finished in 2ms: getPackageFiles /home/mitchell/snowdrift/.stack-work/downloaded/bnIHMPumBlP2/yesod-bin/yesod-bin.cabal
@(Stack/Package.hs:250:21)
[debug] Finding out which packages are already installed
@(Stack/Build/Installed.hs:68:5)
[debug] Run process: /home/mitchell/.stack/programs/x86_64-linux/ghc-7.10.3/bin/ghc-pkg --global --no-user-package-db dump --expand-pkgroot
@(System/Process/Read.hs:306:3)
[debug] Process finished in 15ms: /home/mitchell/.stack/programs/x86_64-linux/ghc-7.10.3/bin/ghc-pkg --global --no-user-package-db dump --expand-pkgroot
@(System/Process/Read.hs:306:3)
[debug] Ignoring package haskeline due to wanting version 0.7.2.3 instead of 0.7.2.1
@(Stack/Build/Installed.hs:191:5)
[debug] Ignoring package terminfo due to wanting version 0.4.0.2 instead of 0.4.0.1
@(Stack/Build/Installed.hs:191:5)
[debug] Ignoring package Cabal due to wanting version 1.22.8.0 instead of 1.22.5.0
@(Stack/Build/Installed.hs:191:5)
[debug] Run process: /home/mitchell/.stack/programs/x86_64-linux/ghc-7.10.3/bin/ghc-pkg --user --no-user-package-db --package-db /home/mitchell/.stack/snapshots/x86_64-linux/lts-6.17/7.10.3/pkgdb dump --expand-pkgroot
@(System/Process/Read.hs:306:3)
[debug] Process finished in 46ms: /home/mitchell/.stack/programs/x86_64-linux/ghc-7.10.3/bin/ghc-pkg --user --no-user-package-db --package-db /home/mitchell/.stack/snapshots/x86_64-linux/lts-6.17/7.10.3/pkgdb dump --expand-pkgroot
@(System/Process/Read.hs:306:3)
[debug] Run process: /home/mitchell/.stack/programs/x86_64-linux/ghc-7.10.3/bin/ghc-pkg --user --no-user-package-db --package-db /home/mitchell/snowdrift/.stack-work/install/x86_64-linux/lts-6.17/7.10.3/pkgdb dump --expand-pkgroot
@(System/Process/Read.hs:306:3)
[debug] Process finished in 11ms: /home/mitchell/.stack/programs/x86_64-linux/ghc-7.10.3/bin/ghc-pkg --user --no-user-package-db --package-db /home/mitchell/snowdrift/.stack-work/install/x86_64-linux/lts-6.17/7.10.3/pkgdb dump --expand-pkgroot
@(System/Process/Read.hs:306:3)
[debug] Trying to decode /home/mitchell/.stack/indices/Hackage/00-index.cache
@(Data/Store/VersionTagged.hs:68:5)
[debug] Success decoding /home/mitchell/.stack/indices/Hackage/00-index.cache
@(Data/Store/VersionTagged.hs:72:13)
[debug] Constructing the build plan
@(Stack/Build/ConstructPlan.hs:159:5)
[debug] Checking if we are going to build multiple executables with the same name
@(Stack/Build.hs:196:5)
[debug] Executing the build plan
@(Stack/Build/Execute.hs:454:5)
[debug] Getting global package database location
@(Stack/GhcPkg.hs:55:5)
[debug] Run process: /home/mitchell/.stack/programs/x86_64-linux/ghc-7.10.3/bin/ghc-pkg --no-user-package-db list --global
@(System/Process/Read.hs:306:3)
[debug] Process finished in 11ms: /home/mitchell/.stack/programs/x86_64-linux/ghc-7.10.3/bin/ghc-pkg --no-user-package-db list --global
@(System/Process/Read.hs:306:3)
@mitchellwrosen
Contributor

I think this behavior changed in 58d4f42, and it was intentional.

The solution for the snowdrift guys (cc @wolftune) is to simply set extra-dep: false for the yesod-bin local package.

@mgsloan
Collaborator
mgsloan commented Dec 17, 2016

Alternatively, require that stack build yesod-bin is run. Sounds like this has been resolved, thanks for investigating @mitchellwrosen !

@mgsloan mgsloan closed this Dec 17, 2016
@mitchellwrosen
Contributor

@mgsloan No problem, but can you clarify what you mean by that last comment?

@wolftune
Contributor

@mgsloan the source of the issue has been identified, but I'm not sure that we know that the way the behavior happens is indeed the desired result. At any rate, your understanding is not correct because stack build yesod-bin does not succeed in our case in getting past this issue!

I'm testing right now to see if workaround from @mitchellwrosen succeeds here. Again, not sure that it's the correct answer or a workaround for something that isn't right in stack.

@mitchellwrosen
Contributor

@wolftune It's not a workaround; as I said the change to not treat packages with extra-dep: true as targets was intentional =)

@wolftune
Contributor

Confirmed resolution from @mitchellwrosen — but I don't myself understand the meaning here of extra-dep: false versus extra-dep: true and why it would (or wouldn't?) make sense to include a location in stack.yaml that isn't an extra-dep. Clearly, I did not follow the recent changes to stack and the .yaml syntax.

I agree this is thus solved, as far as I can tell. Perhaps the erroneous suggestion from @mgsloan is just a red herring.

@mitchellwrosen
Contributor

Well, looking over the commit message I don't actually know for sure if @snoyberg realized he was changing the behavior of stack build <local dep with executable component>.

@wolftune as far as I know, you have a forked version of yesod-bin for the sole purpose of manually typing stack build yesod-bin, no? It's a little odd that it can't be marked as extra-dep: true so that it does't (for example) squelch the build output of the package we really care about... the very reason the commit was made in the first place!

@wolftune wolftune added a commit to snowdriftcoop/snowdrift that referenced this issue Dec 17, 2016
@wolftune wolftune extra-dep: false now needed for yesod-bin
As of stack 1.2.1 the yesod binary won't get installed in .stack-work
unless we make this change. Everything works fine as far as I can tell
with this resolution.

See commercialhaskell/stack#2849
31c6588
@wolftune
Contributor
wolftune commented Dec 17, 2016 edited

@mitchellwrosen no, that wasn't a reason at all (as far as I thought / know). The reason for the fork was to work with some new TH we added.

See:
https://git.snowdrift.coop/sd/yesod/commit/fa15b526dcc4acfdb4ec4536d5edadbcfcd6391d
and
https://git.snowdrift.coop/sd/yesod/commit/e6af409c68d217b8e33ef9f4e267935e07b25f65

@wolftune
Contributor

And, in conjunction with that, I don't find it transparent (to me at least) whether the extra-dep true or false variable would be a concern for this case (or similar cases)

@mitchellwrosen
Contributor

Err, I worded that wrong, but I meant you forked it to change it, "it" being an executable component and not a library.

@wolftune
Contributor

That sounds right… since we use yesod-bin to run essentially yesod-devel, we need it to process our code correctly. I think our separate special build script runs the install of yesod-bin perhaps. At any rate, with our setup now, nobody manually runs stack build yesod-bin (although that was tried unsuccessfully in the testing of this issue).

@mitchellwrosen
Contributor
mitchellwrosen commented Dec 17, 2016 edited

And re: your last second-to-last comment, right, I think I agree, and I'm not sure that the fix was necessarily the right one (for reasons I tried to outline above but will reiterate): it's now not possible to have a forked version of an executable as part of the definition of a stack project, unless the package containing the executable is a "top-level" one. My terminology is off, but I hope I'm making sense.

@wolftune
Contributor

it's now not possible to have a forked version of an executable as part of the definition of a stack project…

Sounds like it could be a real problem with stack that should be at least considered and reflected upon by stack devs to figure out what to do or if this limitation is acceptable…

@wolftune
Contributor

@mgsloan perhaps you can now reflect on the situation enough to determine the right course of action… maybe a new issue should be opened that specifically clarifies this problem as @mitchellwrosen is describing it? I'm certainly not sure of my understanding at this point.

@snoyberg
Contributor

This does look like a bug to me, mea culpa. Reopening. Does anyone have a recommendation on the best behavior here? I think the original issue I was dealing with is still a real one, but perhaps the logic needs to be closer to "only treat it as a target if it's explicitly spelled out."

@snoyberg snoyberg reopened this Dec 19, 2016
@snoyberg
Contributor

OK, after a bit more research into this:

  • I was originally trying to solve #2545
  • I first wrote the patch referenced above (58d4f42), which as noted here is incorrect
  • I then immediately wrote the patch e3a0502, which properly disables test suites and benchmarks for extra-dep packages
  • It appears that with the second patch, the first patch is no longer needed, and can just be backed out

I'll open up a PR now, I'd appreciate if someone could test this use case against that PR once available.

@snoyberg snoyberg referenced this issue Dec 20, 2016
Merged

Local extra-dep packages can be targets #2849 #2863

2 of 2 tasks complete
@wolftune
Contributor

@snoyberg can you point me quickly to how to build locally from the particular git branch?

@snoyberg
Contributor
$ git clone --depth=1 https://github.com/commercialhaskell/stack --branch 2849-extra-dep-target && cd stack && stack install
@wolftune
Contributor

right, duh. stack install is all I need, I had the branch already. I only ever did stack install x not just stack install of a repo

@wolftune wolftune added a commit to snowdriftcoop/snowdrift that referenced this issue Dec 20, 2016
@wolftune wolftune Revert "extra-dep: false now needed for yesod-bin"
This reverts commit 31c6588.

It turns out it was a stack bug after all, see
commercialhaskell/stack#2849
45bbdc0
@wolftune
Contributor

@snoyberg I was able to build and run as expected using the stack from that PR, so confirmed fixed here

@snoyberg
Contributor

Since the fix has been confirmed, closing.

@snoyberg snoyberg closed this Dec 27, 2016
@snoyberg snoyberg self-assigned this Dec 27, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment