Build-tools not detected when not using Stackage snapshot

[conklech]

This shows up with the gtk libraries; unsure how general it is.

With a dummy project depending on cairo, a normal stack.yaml depending on a snapshot correctly (somehow?) infers that gtk2hs-buildtools must be installed to build cairo:

$ cat stack.yaml
resolver: lts-2.18
packages:
- '.'
$ stack build --dry-run
Nothing to unregister

Would build:
alex-3.1.4: database=snapshot, source=package index
buildtools-test-0.1.0.0: database=local, source=D:\Projects\buildtools-test\, after: cairo-0.13.1.0
cairo-0.13.1.0: database=snapshot, source=package index, after: gtk2hs-buildtools-0.13.0.4,text-1.2.0.6
gtk2hs-buildtools-0.13.0.4: database=snapshot, source=package index, after: alex-3.1.4,hashtables-1.2.0.2
[..]

But with resolver: ghc-7.10 and manually specified dependencies, it it doesn't figure that out:

$ cat stack.yaml
resolver: ghc-7.10
flags:
  text:
    integer-simple: false
extra-deps: 
  - cairo-0.13.1.0
  - mtl-2.2.1
  - text-1.2.1.1
  - utf8-string-1
packages:
- '.'
$ stack build --dry-run
Nothing to unregister

Would build:
buildtools-test-0.1.0.0: database=local, source=D:\Projects\buildtools-test\, after: cairo-0.13.1.0
cairo-0.13.1.0: database=local, source=package index, after: mtl-2.2.1,text-1.2.1.1,utf8-string-1
[..]

That build would fail. Adding gtk2hs-buildtools-0.13.0.4 to the extra-deps field does not solve the problem.

How does stack determine that gtk2hs-buildtools is required, and why doesn't that work with the ghc-* resolver?

[snoyberg]

The build tool discovery is fragile at best, because of the way Cabal specifies these things. Currently, the tool map is only capable of finding things in the snapshot build plan. I can't think of an efficient way of making this work well, but if you (or someone else) is interested in taking a stab at this, I can work through the issue with you.

grep the codebase for getToolMap to get started with this.

[conklech]

It's documented in Stack.Build.ConstructPlan that only the snapshot is used to find build tools.

This is so, if I understand correctly, basically for circularity reasons. The mapping from tool names to providing packages is needed in the Ctx structure that's the Reader component of the RWST monad used in plan construction. But when Ctx is constructed, the non-snapshot packages (extraToBuild0) are just a set of package names; they're added to the build plan as the monad runs. In other words, snapshot packages start off in the build plan and are available to satisfy build-tool dependencies; extra packages are not.

It looks like our behavior is to silently accept the inability to satisfy a built-tool requirement.

The general solution would be to make two passes over the build plan, directly or indirectly, so that extra packages are available in the second pass.

To work around this problem, you can manually stack build the required packages (alex, happy, gtk2hs-buildtools). Doing so requires manually adding those packages' dependencies to stack.yaml. stack install is not necessary and would pollute the global $PATH.

This problem is particularly pernicuous because a package maintainer may have the required tools available on the global $PATH and therefore never encounter this problem.

[borsboom]

Another use case described in #1427.

[mgsloan]

Considering that another instance of this has come up, bumping to P2.

[mgsloan]

Another detail of this is that if you have a dependency of a build-tool in extra-deps, then it will not be considered as a candidate tool. Argh! This really ought to get fixed.

In 4fb0b06, I added some warnings that link to this issue. Once the issue is fixed, the suggested cause should be removed.

[mgsloan]

This will be resolved as a consequence of #1265

[m4lvin]

I got »No packages found in snapshot which provide a "hsc2hs" executable, which is a build-tool dependency of« and was pointed here. However, I am using a snapshot (lts-5.18). Anything I am doing wrong?

[mgsloan]

hsc2hs should be included with ghc-7.10.3. Are you using a system installed ghc? If not, perhaps put system-ghc: false in your ~/.stack/config.yaml and use stack setup to install it?

[m4lvin]

Thanks for the suggestion, but the error seems to be independent of where ghc comes from. I took a fresh minimal Debian Jessie system, only installed stack and cloned my project. After "stack setup" I have ~/.stack/programs/x86_64-linux/ghc-7.10.3/bin/hsc2hs available. But "stack build" still gives me the same warning.
Here is an example on travis: https://travis-ci.org/m4lvin/cudd/jobs/133326886#L360

[blitzcode]

I also just hit this error message, all I did was update a project from lts-5.4 to lts-6.0, no changes to the global stack config or system GHC or anything like this.

stack build
Downloaded lts-6.0 build plan.
Caching build plan
No packages found in snapshot which provide a "hsc2hs" executable, which is a build-tool dependency of "bindings-GLFW"
Missing build-tools may be caused by dependencies of the build-tool being overridden by extra-deps.
This should be fixed soon - see this issue https://github.com/commercialhaskell/stack/issues/595

This is with stack v1.1 on OS X 10.10.

[ghost]

Just got this same error also on lts-2.22 with freshly install stack on ubuntu 14.04

[acowley]

I have the same hsc2hs message with lts-6.2 on darwin.

[lukexi]

I get this here too with bindings-GLFW on nightly-2016-02-02 on Win64, stack Version 1.1.3, Git revision 006044, using system GHC 7.10.3.

$ which hsc2hs
/usr/local/ghc/bin/hsc2hs

[mantkiew]

I got this message No packages found in snapshot which provide a "ghc" executable, which is a build-tool dependency of "..." when building with nightly-2016-06-21. In my cabal file, I only have

build-tools:        ghc >= 7.8.3

[prabhakar97]

Got this while trying to build yesod. System hsc2hs is in my path. Did a stack build hsc2hs too, but in vain. Can anybody tell how to get unblocked at least temporarily?

No packages found in snapshot which provide a "hsc2hs" executable, which is a build-tool dependency of "skein"
Missing build-tools may be caused by dependencies of the build-tool being overridden by extra-deps.
This should be fixed soon - see this issue https://github.com/commercialhaskell/stack/issues/595

While constructing the BuildPlan the following exceptions were encountered:

--  Failure when adding dependencies:    
      yesod: needed (>=1.2), couldn't resolve its dependencies
    needed for package classy-prelude-yesod-0.12.8

--  Failure when adding dependencies:    
      classy-prelude-yesod: needed (>=0.10.2), couldn't resolve its dependencies
    needed for package yesod-0.0.0

[sjakobi]

@mantkiew:

build-tools: ghc >= 7.8.3

I've never seen ghc declared as a build-tool (which doesn't necessarily mean that it's wrong).

If you want to have a constraint on the compiler version used, you'd usually do that via a constraint on the base package which is (still) tightly linked to the ghc version.

For ghc >= 7.8.3 that would be base >= 4.7.0.1.

[sjakobi]

@prabhakar97:

There's no need to worry about the build tools warning as long as you actually have that build tool on your PATH.

The dependency resolution errors are more worrying. You could try to relax the bounds that you define yourself or scrutinize your dependencies for overly strict bounds. This is not the right place to discuss this though.

[m4lvin]

Something I still don't understand: Why do I get a warning sending me here even if I am using a resolver? The problem seems to be that hsc2hs is not on stackage. Which makes sense because it comes with ghc already. So maybe the actual bug is that stack still thinks hsc2hs needs to be installed? (Okay, so this is tracked in #2235 already, sorry for the noise.)

[alx741]

For me this issue is making Yesod completely unusable: stack build produces:

No packages found in snapshot which provide a "hsc2hs" executable, which is a build-tool dependency of "skein"
Missing build-tools may be caused by dependencies of the build-tool being overridden by extra-deps.
This should be fixed soon - see this issue #595
While constructing the build plan, the following exceptions were encountered:
Plan construction failed.

And that's it... No Yesod for me 😞

Edit: yesod is not a good name to use for the project when creating it with stack: yesodweb/yesod#1299

[NorfairKing]

+1 Having this problem with pandoc and embed-data-files as well.

[codygman]

+1 Having this problem with Idris.

[mitchellwrosen]

It is not high priority, because usage for big projects without using a resolver doesn't seem to be a common usecase

@mgsloan Can you clarify what you mean by this? I read through the thread and I don't quite understand the resolver/no resolver distinction.

I have a repo with a nightly resolver that fails to build on travis because haskell-src-exts is a transitive dependency that requires happy, which stack isn't installing for some reason. My workaround is the same as was made in https://github.com/chrisdone/hindent/pull/345 - just explicitly stack install happy.

[mgsloan]

What I really meant there is that usually folks use a snapshot. If the required build-tool is in the snapshot, then it will be installed.

[OlivierNicole]

@mgsloan In my case alex is in the snapshot, but as you noted it won't work because one of its dependencies is in extra-deps. (Well I'm guessing that's the cause.) So it can be quite a bother.

[mboes]

Hadrian also exhibits this problem. The stack.yaml in that case is pretty simple:

resolver: lts-8.21
packages:
- '.'
- '../libraries/Cabal/Cabal'

I'm not sure extra-deps are at fault here though, because a) the set of extra-deps is empty in this case, and b) the two build-tools that are not found are alex and happy, neither of which transitively depend on the Cabal library AFAIK.

[snoyberg]

See #3249

[tanzeelrana]

input-output-hk/cardano-sl#1921

stack build --ghc-options="-DGITREV=747e091ba72f81d2f0c73650a4f89fe93b09fc84 -Wwarn +RTS -A256m -n2m -RTS" --test --no-haddock-deps --bench --jobs=4 --flag cardano-sl:with-wallet --no-run-tests --no-run-benchmarks --fast --allow-different-user cardano-sl-core

No packages found in snapshot which provide a "cpphs" executable, which is a build-tool dependency of "cardano-sl-core"
Missing build-tools may be caused by dependencies of the build-tool being overridden by extra-deps.
This should be fixed soon - see this issue #595

cardano-sl-core-1.0.3: configure (lib)
Configuring cardano-sl-core-1.0.3...
Cabal-simple_mPHDZzAJ_1.24.2.0_ghc-8.0.2: The program 'cpphs' version >=1.19
is required but it could not be found.

-- While building package cardano-sl-core-1.0.3 using:

[snoyberg]

This issue has been resolved in Stack 1.6, which is currently in prerelease.