New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cabal-install doesn't consider build-tools as dependencies #220

Closed
bos opened this Issue May 24, 2012 · 16 comments

Comments

Projects
None yet
9 participants
@bos
Contributor

bos commented May 24, 2012

Summary by @ezyang. When you write build-tools: alex, Cabal only checks that an executable of this name is installed; cabal-install does not know that if there is no such executable, alex needs to be installed. One complication is that build-tools does not necessarily correspond to a package, so it is not a priori obvious to cabal-install what should be installed to get an executable.

Here is Proposal A: Build-tools only

  • We introduce a new type of qualified goal, a build-tools goal, similar to how setup qualified goals work.
  • We modify the solver to interpret each field of build-tools as a package name, and we solve for each build-tool as a separate build-tools dependency (so we don't do something goofy like try to solve for happy and alex's dependency sets consistently when they really couldn't care less.)
  • Some build-tools fields don't correspond to actual packages. We give two escape hooks in this situation. First, Cabal has an internal mapping of build tools to Maybe PackageName. If Nothing, this indicates that the build-tool is not managed by Cabal and we shouldn't try to solve for it. If Just pn, then it means that installing pn will also install the requested executable. This mapping will be configurable using the build-tool-packages field in a cabal-project and also modifiable by command line argument. (Names can be bikeshedded of course.) Our second escape hook is, if a build-tool refers to an unknown package name, we print a warning but continue to solve, simply ignoring that package. A user can suppress the warning by disabling this using the aforementioned flag.

Consider these examples:

  • build-tools: mytoolpkg: cabal-install will install all executables defined by a package named mytoolpkg, and ensure they are available prior to building this package.
  • build-tools: gtk2hsc2hs: cabal-install is hard-coded to know that the gtk2hsc2hs executable is provided by the gtk2hs-buildtools, so it ensures that it builds and installs all executables from the package before building this package.
  • build-tools: someprog, plus a flag --provides-build-tool someprog=somepkg, cabal-install sees someprog (which has no package), but using the flag mapping it knows that somepkg is the package that provides this build tool.

Here is Proposal B: tool-depends

  • We introduce a new type of qualified goal, a build-tools goal, similar to how setup qualified goals work. (Like before)
  • We introduce a new tool-depends field, which consists of pkg:exename version-constraints specifications, e.g., tool-depends: gtk2hs-buildtools:gtk2hs2hs < 2.0. This means that we must build the gtk2hsc2hs executable from gtk2hs-buildtools before building this package. In principle cabal-install need only build the gtk2hsc2hs executable, although today's setup interface doesn't allow one to selectively enable executables, so we'd probably just build all executables. If qualification is omitted it is assumed that you need ALL executables from the package.
  • build-tools refers exclusively to non-packaged executables, e.g. provided by the system. However, for backwards compatibility, we introduce an internal mapping hard-coded into GHC which maps some build-tools fields to package names. The intended interpretation is that build-tools: happy > 0.4 actually elaborates to tool-depends: happy:happy > 0.4. This mapping will be configurable using the build-tool-packages field in a cabal-project and also modifiable by command line argument. (Names can be bikeshedded of course.)

Examples:

  • tool-depends: mytoolpkg: cabal-install will install all executables defined by a package named mytoolpkg, and ensure they are available prior to building this package.
  • tool-depends: mytoolpkg:mytool: cabal-install will install the executable mytool from mytoolpkg before building this package
  • build-tools: happy: this is equivalent to writing tool-depends: happy:happy.
  • build-tools: someprog: as before this does not effect solver behavior; there just simply needs to be a someprog in the path.

(Imported from Trac #227, reported by guest on 2008-02-01)

On #haskell, nelhage mentioned that one of the bugs he ran into while trying to install Yi-0.3 through cabal-install was an error which looked like this:

Installing: /home/gwern/.cabal/lib/regex-posix-0.72.0.2/ghc-6.8.2/regex-posix-0.72.0.2/ghc-6.8.2
Registering regex-posix-0.72.0.2...
Reading package info from "dist/installed-pkg-config" ... done.
Saving old package config file... done.
Writing new package config file... done.
'yi-0.3' is cached.
[1 of 1] Compiling Main             ( Setup.hs, dist/setup/Main.o )
Linking dist/setup/setup ...
Configuring yi-0.3...
setup: alex version >=2.0.1&&<3 is required but it could not be found.

That is, the dependency on Alex (a version which satisfied >=2.0.1&&<3) was not met. Perfectly sensible as he did not have Alex installed. Puzzled, he did a 'cabal install alex', which worked, and then re-started. (I guess he had his GHC installed locally?).

He and I were puzzled, because cabal-install is supposed/usually does track down dependencies. Is build-tools excluded for a reason, or was it just accidentally omitted from dependency-tracking? If the latter, I think this is a bit of a bug.

(I use Cabal-1.2.3.0, btw, and the error message above was my reproduction of Nelhage's problem. I had to uninstall Alex from my system, but that did the trick, and reinstalling bypassed the issue just fine - which is why I hadn't seen it before.)

gwern

@ghost ghost assigned dcoutts May 24, 2012

@bos

This comment has been minimized.

Show comment
Hide comment
@bos

bos May 24, 2012

Contributor

(Imported comment by @dcoutts on 2008-02-01)

One problem is that not all build-tools correspond to haskell packages. Some do some don't. We have a hard coded list of them at the moment (which can be extended in Setup.hs files) so we could extend that with what haskell package if any the tools correspond to. Any better suggestions to make it a tad more generic?

Contributor

bos commented May 24, 2012

(Imported comment by @dcoutts on 2008-02-01)

One problem is that not all build-tools correspond to haskell packages. Some do some don't. We have a hard coded list of them at the moment (which can be extended in Setup.hs files) so we could extend that with what haskell package if any the tools correspond to. Any better suggestions to make it a tad more generic?

@bos

This comment has been minimized.

Show comment
Hide comment
@bos

bos May 24, 2012

Contributor

(Imported comment by @dcoutts on 2008-02-28)

This limitation clearly confuses people.

  • [http://www.haskell.org/pipermail/cabal-devel/2008-June/003220.html](http://www.haskell.org/pipermail/cabal-devel/2008-June/003220.html)
  • [http://www.haskell.org/pipermail/haskell-cafe/2008-June/044222.html](http://www.haskell.org/pipermail/haskell-cafe/2008-June/044222.html)
Contributor

bos commented May 24, 2012

(Imported comment by @dcoutts on 2008-02-28)

This limitation clearly confuses people.

  • [http://www.haskell.org/pipermail/cabal-devel/2008-June/003220.html](http://www.haskell.org/pipermail/cabal-devel/2008-June/003220.html)
  • [http://www.haskell.org/pipermail/haskell-cafe/2008-June/044222.html](http://www.haskell.org/pipermail/haskell-cafe/2008-June/044222.html)
@bos

This comment has been minimized.

Show comment
Hide comment
@bos

bos May 24, 2012

Contributor

(Imported comment by darrint on 2008-06-16)

Seems like what you need is to be able to determine which packages provide which executables. Alex, for instance, declares that it provides an executable by that name. For the next release of cabal-install, maybe change the package repository metadata format to include more (all?) the cabal info instead of a file list. That will allow you to solve more situations and do a full install plan up front.

Would that work?

Contributor

bos commented May 24, 2012

(Imported comment by darrint on 2008-06-16)

Seems like what you need is to be able to determine which packages provide which executables. Alex, for instance, declares that it provides an executable by that name. For the next release of cabal-install, maybe change the package repository metadata format to include more (all?) the cabal info instead of a file list. That will allow you to solve more situations and do a full install plan up front.

Would that work?

@bos

This comment has been minimized.

Show comment
Hide comment
@bos

bos May 24, 2012

Contributor

(Imported comment by @dcoutts on 2008-06-16)

See also #342

Contributor

bos commented May 24, 2012

(Imported comment by @dcoutts on 2008-06-16)

See also #342

@bos

This comment has been minimized.

Show comment
Hide comment
@bos

bos May 24, 2012

Contributor

(Imported comment by @mcandre on 2009-01-05)

I have the same problem with ncurses and c2hs. I'd really like dependencies to be handled automatically whenever possible.

Contributor

bos commented May 24, 2012

(Imported comment by @mcandre on 2009-01-05)

I have the same problem with ncurses and c2hs. I'd really like dependencies to be handled automatically whenever possible.

@mietek

This comment has been minimized.

Show comment
Hide comment
@mietek

mietek May 29, 2014

Contributor

Looking forward to this being fixed.

Contributor

mietek commented May 29, 2014

Looking forward to this being fixed.

@tibbe

This comment has been minimized.

Show comment
Hide comment
@tibbe

tibbe May 29, 2014

Member

@mietek right now this is blocked on no one having the time to work on this. If you're interested perhaps you could take a look? If you have any questions you can write to cabal-devel@haskell.org.

Member

tibbe commented May 29, 2014

@mietek right now this is blocked on no one having the time to work on this. If you're interested perhaps you could take a look? If you have any questions you can write to cabal-devel@haskell.org.

@mietek

This comment has been minimized.

Show comment
Hide comment
@mietek

mietek Jun 2, 2014

Contributor

@tibbe: I can't right now, but I am interested in solving this, in one way or another.

Contributor

mietek commented Jun 2, 2014

@tibbe: I can't right now, but I am interested in solving this, in one way or another.

@TheunisKotze

This comment has been minimized.

Show comment
Hide comment

👍

@mietek

This comment has been minimized.

Show comment
Hide comment
@mietek

mietek Nov 26, 2014

Contributor

Halcyon supports declaring and automatically installing build-tools and other Haskell apps for use at build-time with the HALCYON_SANDBOX_EXTRA_APPS option. Version constraints for these apps can be declared with the HALCYON_SANDBOX_EXTRA_APPS_CONSTRAINTS option.

Similarly, you can use HALCYON_EXTRA_APPS and HALCYON_EXTRA_APPS_CONSTRAINTS to declare additional apps to be installed for use at run-time.

See Haskell Language for an example of declaring alex and happy as sandbox extra apps.

Contributor

mietek commented Nov 26, 2014

Halcyon supports declaring and automatically installing build-tools and other Haskell apps for use at build-time with the HALCYON_SANDBOX_EXTRA_APPS option. Version constraints for these apps can be declared with the HALCYON_SANDBOX_EXTRA_APPS_CONSTRAINTS option.

Similarly, you can use HALCYON_EXTRA_APPS and HALCYON_EXTRA_APPS_CONSTRAINTS to declare additional apps to be installed for use at run-time.

See Haskell Language for an example of declaring alex and happy as sandbox extra apps.

@danse

This comment has been minimized.

Show comment
Hide comment
@danse

danse Jul 13, 2015

This is quite a big flaw. I am an Haskell newbie, but i will try to put my two cents in order to keep the discussion alive

One problem is that not all build-tools correspond to haskell packages. Some do some don't. We have a hard coded list of them at the moment (which can be extended in Setup.hs files) so we could extend that with what haskell package if any the tools correspond to. Any better suggestions to make it a tad more generic?

What is the purpose of build-tools if not to list items to be installed? Correct me if i am wrong: if a build tool is declared as a build dependency in build-depends, it will be correctly installed, but if it is listed within build-tools it might be ignored?

danse commented Jul 13, 2015

This is quite a big flaw. I am an Haskell newbie, but i will try to put my two cents in order to keep the discussion alive

One problem is that not all build-tools correspond to haskell packages. Some do some don't. We have a hard coded list of them at the moment (which can be extended in Setup.hs files) so we could extend that with what haskell package if any the tools correspond to. Any better suggestions to make it a tad more generic?

What is the purpose of build-tools if not to list items to be installed? Correct me if i am wrong: if a build tool is declared as a build dependency in build-depends, it will be correctly installed, but if it is listed within build-tools it might be ignored?

@oneeyedziggy

This comment has been minimized.

Show comment
Hide comment
@oneeyedziggy

oneeyedziggy Jul 20, 2015

"Every little bit helps," said the gnat and it pissed in the sea.
here's my little bit... I'm also a first timer like danse above (who i suspect is going to OsCon 2015), this kind of puts a sour taste in the mouth of anyone trying to pick up haskell...

"Every little bit helps," said the gnat and it pissed in the sea.
here's my little bit... I'm also a first timer like danse above (who i suspect is going to OsCon 2015), this kind of puts a sour taste in the mouth of anyone trying to pick up haskell...

@hvr

This comment has been minimized.

Show comment
Hide comment
@hvr

hvr Jul 27, 2015

Member

IMHO to keep this KISS-y, build-tools should be taken into account when solving for an install-plan, and try to find an install-plan that is possible with the currently available tools. I.e. consider build-tools a static (non-upgradeable/installable) property of the build-environment (like e.g. the GHC version -- which ultimately is a build-tool as well).

If, at some point in the future, we want to support auto-installing build-tools, we'd have to massively extend cabal-installs support for tracking package executables.

/cc @kosmikus

Member

hvr commented Jul 27, 2015

IMHO to keep this KISS-y, build-tools should be taken into account when solving for an install-plan, and try to find an install-plan that is possible with the currently available tools. I.e. consider build-tools a static (non-upgradeable/installable) property of the build-environment (like e.g. the GHC version -- which ultimately is a build-tool as well).

If, at some point in the future, we want to support auto-installing build-tools, we'd have to massively extend cabal-installs support for tracking package executables.

/cc @kosmikus

@mietek

This comment has been minimized.

Show comment
Hide comment
@mietek

mietek Jul 27, 2015

Contributor

@hvr’s suggestion would certainly make the experience less frustrating.

As it is, you’ll only find out you’re missing a build-tool halfway through building your dependencies — and that’s if you’re paying attention, because #1924 means unless you manually abort the build, you’ll continue burning CPU cycles on something which has no chance of succeeding.

Contributor

mietek commented Jul 27, 2015

@hvr’s suggestion would certainly make the experience less frustrating.

As it is, you’ll only find out you’re missing a build-tool halfway through building your dependencies — and that’s if you’re paying attention, because #1924 means unless you manually abort the build, you’ll continue burning CPU cycles on something which has no chance of succeeding.

ezyang added a commit to ezyang/cabal that referenced this issue Mar 31, 2016

Have Cabal build a Setup executable and test with it.
I recently discovered that build-tools can be used to specify
dependencies on internal executables (#220).  This means that
we can make a nice improvement to the package-tests test suite:
instead of manually building a Setup.hs script ourselves,
we can add one to the Cabal file for Cabal, and have the
test suite have a build-tools dependency on it.

Perhaps the only objection to this is that the Cabal *library*
now has an executable cabal-setup.  Maybe with some buildable
shenanigans we can make it so that we never build this executable
unless the user asks for it, or the test suite is enabled.
Alternately, cabal-setup is a generally handy executable to have
around.

In fact, we can use this idea to completely kill the LBI hack.
But I'm going to leave that for another patchset, assuming we
like this one.

Signed-off-by: Edward Z. Yang <ezyang@cs.stanford.edu>

ezyang added a commit to ezyang/cabal that referenced this issue Mar 31, 2016

Have Cabal build a Setup executable and test with it.
I recently discovered that build-tools can be used to specify
dependencies on internal executables (#220).  This means that
we can make a nice improvement to the package-tests test suite:
instead of manually building a Setup.hs script ourselves,
we can add one to the Cabal file for Cabal, and have the
test suite have a build-tools dependency on it.

Perhaps the only objection to this is that the Cabal *library*
now has an executable cabal-setup.  Maybe with some buildable
shenanigans we can make it so that we never build this executable
unless the user asks for it, or the test suite is enabled.
Alternately, cabal-setup is a generally handy executable to have
around.

Signed-off-by: Edward Z. Yang <ezyang@cs.stanford.edu>

@ezyang ezyang added the meta: ezyang label Mar 31, 2016

ezyang added a commit to ezyang/cabal that referenced this issue Mar 31, 2016

Have Cabal build a Setup executable and test with it.
I recently discovered that build-tools can be used to specify
dependencies on internal executables (#220).  This means that
we can make a nice improvement to the package-tests test suite:
instead of manually building a Setup.hs script ourselves,
we can add one to the Cabal file for Cabal, and have the
test suite have a build-tools dependency on it.

Perhaps the only objection to this is that the Cabal *library*
now has an executable cabal-setup.  Maybe with some buildable
shenanigans we can make it so that we never build this executable
unless the user asks for it, or the test suite is enabled.
Alternately, cabal-setup is a generally handy executable to have
around.

Signed-off-by: Edward Z. Yang <ezyang@cs.stanford.edu>

ezyang added a commit to ezyang/cabal that referenced this issue Mar 31, 2016

Have Cabal build a Setup executable and test with it.
I recently discovered that build-tools can be used to specify
dependencies on internal executables (#220).  This means that
we can make a nice improvement to the package-tests test suite:
instead of manually building a Setup.hs script ourselves,
we can add one to the Cabal file for Cabal, and have the
test suite have a build-tools dependency on it.

Perhaps the only objection to this is that the Cabal *library*
now has an executable cabal-setup.  Maybe with some buildable
shenanigans we can make it so that we never build this executable
unless the user asks for it, or the test suite is enabled.
Alternately, cabal-setup is a generally handy executable to have
around.

Signed-off-by: Edward Z. Yang <ezyang@cs.stanford.edu>

ezyang added a commit to ezyang/cabal that referenced this issue Apr 1, 2016

Have Cabal build a Setup executable and test with it.
I recently discovered that build-tools can be used to specify
dependencies on internal executables (#220).  This means that
we can make a nice improvement to the package-tests test suite:
instead of manually building a Setup.hs script ourselves,
we can add one to the Cabal file for Cabal, and have the
test suite have a build-tools dependency on it.

Perhaps the only objection to this is that the Cabal *library*
now has an executable cabal-setup.  Maybe with some buildable
shenanigans we can make it so that we never build this executable
unless the user asks for it, or the test suite is enabled.
Alternately, cabal-setup is a generally handy executable to have
around.

Signed-off-by: Edward Z. Yang <ezyang@cs.stanford.edu>

khumba added a commit to khumba/qtah that referenced this issue Jul 5, 2016

Improve the dependencies on qtah-generator.
This changes the dependencies on qtah-generator from build-tools to
build-depends, to work with version dependencies and also to ensure
that the generator gets installed, since build-tools doesn't trigger
installation currently (haskell/cabal#220).
@ezyang

This comment has been minimized.

Show comment
Hide comment
@ezyang

ezyang Aug 20, 2016

Contributor

CC @hvr

I implemented this in #3662 and I realized that we'll need to change build-tools in a bit more major way than I originally suggested in proposal A. So, in no particular order, here are things I observed:

  1. Today, most names you put in build-tools are rejected by Cabal, saying "Unknown build tool blah". The reason for this is Cabal has a ProgramDb which exhaustively enumerates all of the build-tools that Cabal actually knows about. Today, the only way to extend this database is to write a Custom setup script that sets hookedPrograms to add some extra programs to the database (this is dumb, by the way). So, what I decided to is this: IF an executable in a build-tools is not already in the program database, we attempt to configure it as a SimpleProgram, instead of immediately giving up. (But maybe this is wrong and counter to the intent of the people who wrote this interface originally.) One problem with doing it this way is that the program database is not serialized to disk (to my knowledge), so the program won't be known in subsequent invocations. This is not too much of a problem at the moment because you never actually try to invoke the program from Cabal; it's always done through a PATH invocation, e.g., a GHC flag (perhaps another indication that doing ProgramDb is the wrong thing.)
  2. OK, so that's nice, but now there's a second problem: the executable you want from a package may NEVER coincide with the package name. An example is cabal, from the cabal-install package. So you could say build-tools: cabal-install which will make the dep-solver happy, but then Cabal will say that it can't find any executable named cabal-install. And if you say cabal, well, someone needs to say what package it comes from, and clearly (except for legacy cases) this information should be in the Cabal file, not some user configuration.) So there really needs to be syntax as flexible as tool-depends (from proposal B) to finger the specific executable you want.
  3. Given that you can finger the correct executable, I don't see any reason for the "legacy mapping" to be configurable outside the Cabal file: putting it in the Cabal file is basically always better. It only poses a problem for backwards compatibility, but the way to handle this is to NOT extend the build-tools syntax, and instead provide package name to executable mappings out of band (in a field that old versions of Cabal can ignore). So, something like tool-depends, but just specifying a bunch of packages which you should look at to find the executables you actually want. Perhaps you could even have thinning and renaming Backpack style.
Contributor

ezyang commented Aug 20, 2016

CC @hvr

I implemented this in #3662 and I realized that we'll need to change build-tools in a bit more major way than I originally suggested in proposal A. So, in no particular order, here are things I observed:

  1. Today, most names you put in build-tools are rejected by Cabal, saying "Unknown build tool blah". The reason for this is Cabal has a ProgramDb which exhaustively enumerates all of the build-tools that Cabal actually knows about. Today, the only way to extend this database is to write a Custom setup script that sets hookedPrograms to add some extra programs to the database (this is dumb, by the way). So, what I decided to is this: IF an executable in a build-tools is not already in the program database, we attempt to configure it as a SimpleProgram, instead of immediately giving up. (But maybe this is wrong and counter to the intent of the people who wrote this interface originally.) One problem with doing it this way is that the program database is not serialized to disk (to my knowledge), so the program won't be known in subsequent invocations. This is not too much of a problem at the moment because you never actually try to invoke the program from Cabal; it's always done through a PATH invocation, e.g., a GHC flag (perhaps another indication that doing ProgramDb is the wrong thing.)
  2. OK, so that's nice, but now there's a second problem: the executable you want from a package may NEVER coincide with the package name. An example is cabal, from the cabal-install package. So you could say build-tools: cabal-install which will make the dep-solver happy, but then Cabal will say that it can't find any executable named cabal-install. And if you say cabal, well, someone needs to say what package it comes from, and clearly (except for legacy cases) this information should be in the Cabal file, not some user configuration.) So there really needs to be syntax as flexible as tool-depends (from proposal B) to finger the specific executable you want.
  3. Given that you can finger the correct executable, I don't see any reason for the "legacy mapping" to be configurable outside the Cabal file: putting it in the Cabal file is basically always better. It only poses a problem for backwards compatibility, but the way to handle this is to NOT extend the build-tools syntax, and instead provide package name to executable mappings out of band (in a field that old versions of Cabal can ignore). So, something like tool-depends, but just specifying a bunch of packages which you should look at to find the executables you actually want. Perhaps you could even have thinning and renaming Backpack style.
@hvr

This comment has been minimized.

Show comment
Hide comment
@hvr

hvr Aug 20, 2016

Member

as we discussed on IRC, it turns out that "proposal B" is starting to look a lot more attractive given the pointed out issues, as requires less confusing magic/fallback tricks

Member

hvr commented Aug 20, 2016

as we discussed on IRC, it turns out that "proposal B" is starting to look a lot more attractive given the pointed out issues, as requires less confusing magic/fallback tricks

ezyang added a commit to ezyang/cabal that referenced this issue Aug 21, 2016

Solve for, build, and add to path build-tools dependencies.
This fixes #220: new-build now builds, installs and adds executables to
PATH automatically if they show up in 'build-tools'.  However, there is
still more that could be done: the new behavior only applies to a
specific list of 'build-tools' (alex, happy, etc) which Cabal recognizes
out of the box.  The plan is to introduce a new 'tool-depends' field to
allow dependencies on other executables as well.

Signed-off-by: Edward Z. Yang <ezyang@cs.stanford.edu>

@ezyang ezyang closed this in c0a4860 Aug 21, 2016

TomMD added a commit to TomMD/liquidhaskell-cabal that referenced this issue Feb 21, 2017

Revert "Attempt to add `liquid` as a legitimate build tool."
This reverts commit c62d623.

It appears adding a build tool is not so simple, requiring changes to
PackageDB.  This task has seen recent activity [1] but the process is
currently unclear to me.

[1] haskell/cabal#220

@yav yav referenced this issue Oct 27, 2017

Closed

Dependancy Missing #22

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment