Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

build-tool-depends executables don't have a .conf file #8434

Open
parsonsmatt opened this issue Aug 29, 2022 · 29 comments
Open

build-tool-depends executables don't have a .conf file #8434

parsonsmatt opened this issue Aug 29, 2022 · 29 comments
Labels
cabal-install: nix integration re: build-tool Concerning `build-tools` and `build-tool-depends`

Comments

@parsonsmatt
Copy link

This is a problem that manifests when trying to use haskell.nix. If it sees the build-tool-depends entries, it does try to build them, but then cabal build is unable to see or execute them.

Cf input-output-hk/haskell.nix#839 (comment)

Note that this really isn't specific shellFor, but specific to v2 cabal. You can always use v1-* cabal commands to get the old behavior (see SolveExecutables in cabal code). I think a reason this is coming up recently is that cabal 3.4.0.0 made v2 the default.

The reason this is really difficult is that both Nixpkgs & haskell.nix use the Setup.hs method of configure / install, which doesn't install a .conf file for executable components. I'm not sure if it makes more sense to

See Cabal comment for reference:

build-tools dependencies
NB: Only include these dependencies if SolveExecutables
is True. It might be false in the legacy solver
codepath, in which case there won't be any record of
an executable we need.

I'm unclear whether or not this is a feasible or acceptable approach, and would appreciate guidance on this. @matthewbauer says that we can either

  1. "Patch Cabal so Setup.hs method installs executables correctly,"
  2. "or try to switch to the new install method." (what is the new install method to use?)
@Mikolaj
Copy link
Member

Mikolaj commented Aug 29, 2022

I'm unfortunately very ignorant of Nix. What is your situation? Are you committed to Nix or does the "new install method to use" include something without Nix? Alternatively, I seem to remember Nix people that use the Nix shell to just normally call cabal, presumably including v2- commands. Is that an option for you? Is haskell.nix a must, or only Nix?

@parsonsmatt
Copy link
Author

What is your situation?

We currently are running nixpkgs for our main and are trying to transition to haskell.nix as this will allow us to use cabal.project as a source-of-truth.

In the nixpkgs approach, we specify all dependencies in nix, and then cabal build pulls from the nix store for these dependencies.

In the hakell.nix approach, we specify all dependencies in cabal.project or .cabal, and haskell.nix creates a nix-shell from these specifications that builds what we need. This is good because it allows us to have a cabal.project that is always up-to-date with all the version info you need, making nix optional, and for those that want to use nix, they can still have an easier time editing the package files rather than complex nix derivations.

I'd frankly prefer we not use nix at all, but it's a necessary part of our build infrastructure at the moment, and there are no plans on replacing it.

Is that an option for you? Is haskell.nix a must, or only Nix?

The value-prop of haskell.nix is that it creates the nix derivations based on the cabal specifications. SO we can still have a nix-shell build environment with all necessary system dependencies, but we don't need to use nix to manage Haskell dependencies.

As far as I can understand it, the real question is:

Can we make Setup.hs method installs executables with a .conf file?

@Mikolaj
Copy link
Member

Mikolaj commented Aug 29, 2022

Can we make Setup.hs method install executables with a .conf file?

So let's focus on this one. Please correct me as I spell this out in my own words. I see the files in my project:

./dist-newstyle/packagedb/ghc-9.0.2/horde-ad-0.1.0.0-inplace-testLibrary.conf
./dist-newstyle/packagedb/ghc-9.0.2/horde-ad-0.1.0.0-inplace.conf

These have been generated by cabal v2-build. From the above I understand Setup.hs neither generates them nor uses them. But v2 both generates them and uses them at least in case of build-tool-depends executables?

So the idea is that haskell.nix would generate the executables with Setup.hs, make the executables and the new .conf files somehow available to the user and then the user would run v2- commands, which need the .conf files?

I suppose, in the best case, it's just the matter of adding a flag to Setup.hs that calls the already existing code from v2- to create the .conf file. In the worst case, the Setup.hs code is deep within the Cabal library and would need to call some code deep within the cabal-install package and would lack tools to create the arguments the cabal-install code needs. Does anybody know which it is?

Could haskell.nix use v2- to create the executables, instead? Why hop between Setup.hs and v2-? They are not supposed to be interoperating and the direction the calls go is always from v2- (and v1- for that matter) to Setup.hs, never back.

@matthewbauer
Copy link
Collaborator

Right, the main issue seems to be using ./Setup install on executable dependencies of a v2 project. While it obviously comes up when using Nix, I don't think it necessarily requires using Nix.

From the above I understand Setup.hs neither generates them nor uses them

I don't think that's true. Setup.hs will generate them with ./Setup install, which calls the ./Setup register command:

https://cabal.readthedocs.io/en/3.4/setup-commands.html#runhaskell-setup-hs-register

I think the issue originates from these lines:

when (hasLibs pkg_descr) $ register pkg_descr localbuildinfo registerFlags

if hasLibs pkg_descr

Since with v2- we do want to register even if there are no library components.

Could haskell.nix use v2- to create the executables, instead? Why hop between Setup.hs and v2-?

That's possible. Might be worth asking haskell.nix if that would work. I actually didn't realize there was a v2-install until today. I wonder if any other package managers are using v2-install to package haskell.

@Mikolaj
Copy link
Member

Mikolaj commented Aug 29, 2022

Since with v2- we do want to register even if there are no library components.

Oh, right. If Setup.hs thinks exes don't need to be registered, that could be fixed and there should be no harm in doing so. I wonder if Setup.hs knows how to register exes (is that only about build-tool-depends exes?). Have you tried changing these conditions by adding an alternative with hasExes and building a package with such a version?

I wonder if any other package managers are using v2-install to package haskell.

That's quite possible for package managers (e.g., the HomeBrew one) that don't package Haskell libraries but just build exes from Hackage and package those, which is a very sane thing to do. Otherwise, v2-install is problematic, because in v2, generally, libraries are not supposed to be installed globally. There are methods and workarounds for that, but they are second class. However, Setup.hs is third class. :) Or rather very low level.

@ParetoOptimalDev
Copy link

ParetoOptimalDev commented Aug 29, 2022

Could haskell.nix use v2- to create the executables, instead? Why hop between Setup.hs and v2-?

Because if I have a haskell.nix project using a build tool like hspec-discover I would rather nix do the building.

I'd rather nix do the building because it will likely skip building entirely and download the prebuilt cached hspec-discover binary from cache.nixos.org or my cachix binary cache.

The problem is cabal v2 won't trust the nix built hspec-discover as a build tool.

From my perspective it seems either:

  • cabal needs to give a way to trust executables it didn't build
  • haskell.nix has to keep hacking around this and having bad UX and interoperability with cabal for build tools

I hope it's not the second, because then cabal.project can't be a source of truth for Haskell.nix.

@ParetoOptimalDev
Copy link

ParetoOptimalDev commented Aug 29, 2022

Something that may clear up confusion:

  • bootstrapping Haskell toolchain and installing haskell packages are done by nix
  • development is done in nix-shell with the nix bootstrapped environment and cabal v2 commands
  • cabal should install no haskell packages because Nix should have already bootstraped them

This issue is about that last point being true for buildTools.

@Mikolaj
Copy link
Member

Mikolaj commented Aug 30, 2022

I see, so it's nix that installs packages globally, and to v2 they look like just a bigger set of packages bundled with GHC, so it knows how to handle that. Except that the bundled packages (libraries) are never used as build-tool-depends (exes). BTW, when you are currently generating the build-tool-depends exes with Setup.hs, are they placed on the path (by Setup.hs or by haskell.nix)? In fact, I don't remember if the registering done by Setup.hs applies to executables at all. IIRC, v1 (which is close to Setup.hs) just copies exes to bin/ and does not enter them into any packagedb.

@gbaz: you have an intimate knowledge of these things; how hard would it be to teach v2 to use any random executables in path for build-tool-depends (preferring the just built/registered exes, if v2 can tell at all)? Alternatively, could Setup.hs do any registering of the exes, if that's needed (.conf files, etc.)? Is that a security issue (a trojan in path posing as a build tool) or just a tweak needed in the code?

@ParetoOptimalDev
Copy link

ParetoOptimalDev commented Sep 14, 2022

@Mikolaj Is there a deprecation plan for cabal-v1?

The current best work-around for using Nix correctly in the workflow where Nix provides packages and build-tools is to use cabal-v1.

haskell-language-server is also affected here because a project that uses something like hspec-discover will cause cabal to go and install the hspec-discover binary even though Nix has already installed it.

I suppose I could send a patch to haskell-language-server to follow cabal-v1 when it comes to build tools but that feels very much like going backwards.

I feel uncomfortable basing my workflows on cabal-v1 and fear in the future I'll be forced away from using Nix with Cabal.

I saw an interesting proposal in #4646 about getting Nix and Cabal to cache dependencies in a way where each is compatible with the other:

One goal in that regard is that Nix and Cabal should be able to hash files in the same way. I think the best thing to do is teach them both git tree hashes. Once we reach the point where hackage builds, patched hackage builds, and local builds are all cached with comparable keys, things will start to move smoothly. - @Ericson2314

That sounds quite ideal, maybe that's worth opening an issue on for wider discussion?

Edit: I also see a proposal for Build Drivers that could help this issues, but I don't fully understand everything yet: #7906

In any case, thanks for your help 🙂

@Mikolaj
Copy link
Member

Mikolaj commented Sep 14, 2022

@Mikolaj Is there a deprecation plan for cabal-v1?

The only plan I know of is shutting down the lights in a major version release. Which is why we are not keen on doing that until somehow the world sorts itself out (people's workflows update, v2 supports more of v1, etc.). Semi-maintaining v1 costs something, but so far not so much as justify harsh solutions IMHO.

haskell-language-server is also affected here because a project that uses something like hspec-discover will cause cabal to go and install the hspec-discover binary even though Nix has already installed it.

Is that a real problem? Afterwards cabal knows about the binary and will not (often) install again.

I saw an interesting proposal in #4646 about getting Nix and Cabal to cache dependencies in a way where each is compatible with the other:

Yes, that looks fine and the other tickets you linked, as well. Unfortunately, lately we have quite many valuable public discussions and comparably few actual implementations. If there was a party committed to implementing a particular feature then the relevant discussions would be a much better use of our scarce resources, I'm very much open to imperfect designs and implementations that scratch a real itch and are maintained by users, but not so much to public debates of universal solutions. We just don't have the resources.

@ParetoOptimalDev
Copy link

Is that a real problem? Afterwards cabal knows about the binary and will not (often) install again.

Arguably no, but I'll list a few issues:

  • I can't just forget about the "provide haskell toolchain" aspect of things
  • I have to worry implications of and differentiate between Nix installed Haskell dependencies and Cabal installed Haskell dependencies (maybe I could just not worry? 😅
  • If I delete my cabal store for whatever reason, I can't quickly get back to work like binary caching from Nix would allow me to
  • I've had these build-tools take 30-60m before, and I never know when that may happen so it feels unpredictable since my focus is on making sure Nix has already provided haskell dependencies.

I'm very much open to imperfect designs and implementations that scratch a real itch and are maintained by users, but not so much to public debates of universal solutions. We just don't have the resources.

I very much agree with about imperfect designs and implementations being more valuable than public debates of universal solutions.

I want to pull the trigger and try either directly patching cabal-v2 to allow Nix to provide it build-tools or perhaps, trying to implement the likely more ambitious shared caching between Nix and Cabal.

My fear, and perhaps the fears of others, is putting work into a patch and then the contribution not being found acceptable.

Is there a best practice for avoiding this? Should I create some issue like "RFC: Allow cabal-v2 to trust build-tools from Nix" and state my intentions to work on that if no one disagrees?

My other looming fear I haven't touched on is that it feels like there is some resistance to integrating better with Nix. Challenging that feeling though:

  • nix integration was worked on and merged before
  • cabal-v2 itself takes inspiration from nix with nix-local builds

So perhaps my fears are unfounded and I would even be okay with just trying to hack on Cabal to allow it to trust build-tools?

In the spirit of respecting scarce resources I'll say that I would have made this response shorter if I had the time 🙂

@Mikolaj
Copy link
Member

Mikolaj commented Sep 14, 2022

but I'll list a few issues

Fair enough. At the very least, for reproducibility and curbing chaos, it should be clearly delineated which parts of the "haskell toolchain" cabal takes care of and which Nix does.

I want to pull the trigger and try either directly patching cabal-v2 to allow Nix to provide it build-tools or perhaps, trying to implement the likely more ambitious shared caching between Nix and Cabal.

Go for it!

My fear, and perhaps the fears of others, is putting work into a patch and then the contribution not being found acceptable.

Some, though not all, features can be maintained as separate tools (see https://github.com/phadej/cabal-extras or cabal-plan). This is actually a very good way of starting the development process, regardless if and when they get merged into cabal proper or advertised in cabal manual or release notes (we should do that more often).

Should I create some issue like "RFC: Allow cabal-v2 to trust build-tools from Nix" and state my intentions to work on that if no one disagrees?

Yes, please do, even if that's going to be an external tool (no idea if that makes sense).

resistance to integrating better with Nix

I think the fight has gone out of us. ;D Expect no fierce resistance.

Seriously, though, there is certainly some duplication of functionality there, with the accompanying maintenance effort whenever the functionality in questions changes in GHC and/or cabal. So the extent to which we duplicate needs a pragmatic justification (not just noble causes, like "every binary should be cached" or Debian's "every library should be shared"; we see where this leads) and a plan for future maintenance. I'm sure we never had a problem with Nix, only with the natural scenario of abandoned volatile features we don't understand, but we eventually need to (un)maintain. I'm sure we can sort it out in a ticket.

@parsonsmatt
Copy link
Author

My fear, and perhaps the fears of others, is putting work into a patch and then the contribution not being found acceptable.

This is absolutely a concern of ours at Mercury, as well. We're happy to put effort/cash towards fixing this issue, but we don't to waste those resources on a design that wouldn't be accepted.

@Mikolaj
Copy link
Member

Mikolaj commented Sep 14, 2022

That's understandable. Let's fail fast, already at initial design phase or at planing phase, e.g., if we see that's going to be too expensive for any of the parties.

However, if an external tool that uses some of the four cabal libraries (that's why we split them off) is an acceptable fit, perhaps with some extra effort required to make things less monolithic, then the risk of rejection diminishes. The tool could also use some of the already existing external tools, e.g., for parsing cabal files. In general, it would be ideal if we had a thriving ecosystem of external cabal tools/plugins, with only as much changes as needed done within cabal proper. Only the tools that "everybody uses" and/or that require minimal maintenance would be merged with cabal at some point, if the author is fine with the drawn out release cadence.

@gbaz
Copy link
Collaborator

gbaz commented Sep 14, 2022

My understanding from talking to past cabal developers and maintainers is that this is something we absolutely should seek to fix. For libraries, cabal builds and leverages a proper packagedb. For executables, the executabledb was built in a somewhat ad-hoc way and never fully specified for external interaction. We should pin down the spec of the executabledb so that nix or any other package management system can properly register packages with it.

If the mercury team (or anyone!) were to come up with a proposal for a proper executabledb that nix and others could leverage, I think that would be a good step consistent with the design of v2-build and in accord with what the intent always was. Of course I'd need to see the details of such a proposal before stating I support it :-)

I do very much think that the solution provided should not be specific to nix, but rather something more general, as I believe the discussion in this thread has already pointed to.

In particular, if Setup.hs installing executables with a conf file really suffices, then this is a very small change which could potentially have a very nice result. (However, I think we would want to carefully consider what knock-on impact, if any, this may have -- in particular, what directory would the conf files go in, how would cabal-the-executable know to look in that directory, etc).

@ParetoOptimalDev
Copy link

FYI I'm at least getting my feet wet with the Cabal codebase and starting to understand the control flow now that I've gotten it building locally. I know where checks in a executable database would go in the repo at least, so progress 😄

Today I'll try to understand the library package database code so I can start imagining what an executable database might look like.

@ParetoOptimalDev
Copy link

ParetoOptimalDev commented Feb 11, 2023

My attempt of making progress on this issue via modeling the library package database behavior won't work because from a Nix perspective even source-repository packages aren't skipped if the package is already provided by the package-database.

Given cabal.project:

packages: .

source-repository-package
    type: git
    location: https://github.com/well-typed/cborg
    tag: 4bdc818a1f0b35f38bc118a87944630043b58384
    subdir: cborg

myproj.cabal:


executable example-source-repository-project
    -- ... snip ...
    build-depends:    base ^>=4.16.4.0

And having cborg in the package database whether from Nix or otherwise so that this it's listed by ghc-pkg list:

$ nix develop -c ghc-pkg list | grep cbor
cborg-0.2.8.0

Cabal will still go to clone the cborg library.

I guess that makes sense becausethere isn't a good way for cabal to tell that the library in the package database fulfills what's asked for by the source-repository-package though. Or really... any way without including the git sha256 commit in the cabal file or something.

This is just the tension of both flake.nix and cabal.project wanting to be the source of truth I guess. Even Nix approaches that try to use the cabal.project as the source of truth won't work though without a way to tell cabal "trust that I provide the correct cborg source repository".

Which means that hybrid Nix and cabal project can't exist without hacky solutions like a separate cabal.project which solutions like IOHK's wrappedCabal try to automate.

Another recent workaround for this recommended as a best practice here is building your own internal hackage with foliage and eschewing source-package-repository entirely.

Maintaining compatibility with the typical cabal.project workflow is paramount though as someone introducing new teams to Nix on a trial basis though. "Replace source-package-repository with your own internal hackage" would likely be laughed away and make the entire endeavor a non-starter.

The closest issue I can find to this is #5444.

So I'm diverting my attention from this issue and to #5444, specifically starting with reflecting on this comment where a cabal maintainer states:

I'd welcome a feature where tar.gz local and remote packages, and source-repository-packages were treated like override package-repositories (i.e. like a single version "hackage"-like repositories) rather than a must have packages.

hololeap added a commit to hololeap/hackport that referenced this issue May 27, 2023
build-tool-depends doesn't play nice with Setup.hs installs, and tries
to rebuild cabal-install/doctest locally. This has recently caused some
compilation errors so there is now a 'ci-depends' flag to toggle between
build-tool-depends (for CI systems) and build-depends (for users with
the needed tools installed to their system).

This reverts commit 51bc8c1.

See: gentoo-haskell/gentoo-haskell#1074
See: haskell/cabal#8434
Signed-off-by: hololeap <hololeap@users.noreply.github.com>
hololeap added a commit to hololeap/hackport that referenced this issue May 27, 2023
build-tool-depends doesn't play nice with Setup.hs installs, and tries
to rebuild cabal-install/doctest locally. This has recently caused some
compilation errors when it tries to build cabal-install, so we'll just
remove it from build-tool-depends on the assumption that whoever is
running doctest-v2 probably has /usr/bin/cabal on their system.

This reverts commit 51bc8c1.

See: gentoo-haskell/gentoo-haskell#1074
See: haskell/cabal#8434
Signed-off-by: hololeap <hololeap@users.noreply.github.com>
hololeap added a commit to hololeap/hackport that referenced this issue May 27, 2023
build-tool-depends doesn't play nice with Setup.hs installs, and tries
to rebuild cabal-install/doctest locally. This has recently caused some
compilation errors when it tries to build cabal-install, so we'll just
remove it from build-tool-depends on the assumption that whoever is
running doctest-v2 probably has /usr/bin/cabal on their system.

Stop using all but Cabal-3.8 on GitHub tests. The others fail during
dependency resolution complaining that they cannot build ghc.

This reverts commit 51bc8c1.

See: gentoo-haskell/gentoo-haskell#1074
See: haskell/cabal#8434
Signed-off-by: hololeap <hololeap@users.noreply.github.com>
hololeap added a commit to hololeap/hackport that referenced this issue May 27, 2023
build-tool-depends doesn't play nice with Setup.hs installs, and tries
to rebuild cabal-install/doctest locally. This has recently caused some
compilation errors when it tries to build cabal-install, so we'll just
remove it from build-tool-depends on the assumption that whoever is
running doctest-v2 probably has /usr/bin/cabal on their system.

All Cabal versions except 3.8 fail on GitHub tests complaining that they
cannot build ghc:

```
cabal: Failed to build ghc-8.10.7 (which is required by exe:doctest from
doctest-0.21.1)
```

This reverts commit 51bc8c1.

See: gentoo-haskell/gentoo-haskell#1074
See: haskell/cabal#8434
Signed-off-by: hololeap <hololeap@users.noreply.github.com>
hololeap added a commit to hololeap/hackport that referenced this issue Jun 2, 2023
build-tool-depends doesn't play nice with Setup.hs installs, and tries
to rebuild cabal-install/doctest locally. This has recently caused some
compilation errors when it tries to build cabal-install, so we'll just
remove it from build-tool-depends on the assumption that whoever is
running doctest-v2 probably has /usr/bin/cabal on their system.

Switch to `cabal-version: 'latest'` instead of a matrix of cabal
versions for github action.

Use template from haskell/actions/setup homepage where possible:
https://github.com/haskell/actions/tree/main/setup#model-cabal-workflow-with-caching

This reverts commit 51bc8c1.

See: gentoo-haskell/gentoo-haskell#1074
See: haskell/cabal#8434
Signed-off-by: hololeap <hololeap@users.noreply.github.com>
hololeap added a commit to hololeap/hackport that referenced this issue Jun 2, 2023
build-tool-depends doesn't play nice with Setup.hs installs, and tries
to rebuild cabal-install/doctest locally. This has recently caused some
compilation errors when it tries to build cabal-install, so we'll just
remove it from build-tool-depends on the assumption that whoever is
running doctest-v2 probably has /usr/bin/cabal on their system.

Switch to `cabal-version: 'latest'` instead of a matrix of cabal
versions for github action.

Use template from haskell/actions/setup homepage where possible:
https://github.com/haskell/actions/tree/main/setup#model-cabal-workflow-with-caching

This reverts commit 51bc8c1.

See: gentoo-haskell/gentoo-haskell#1074
See: haskell/cabal#8434
Signed-off-by: hololeap <hololeap@users.noreply.github.com>
hololeap added a commit to hololeap/hackport that referenced this issue Jun 3, 2023
build-tool-depends doesn't play nice with Setup.hs installs, and tries
to rebuild cabal-install/doctest locally. This has recently caused some
compilation errors when it tries to build cabal-install, so we'll just
remove it from build-tool-depends on the assumption that whoever is
running doctest-v2 probably has /usr/bin/cabal on their system.

Switch to `cabal-version: 'latest'` instead of a matrix of cabal
versions for github action.

Use template from haskell/actions/setup homepage where possible:
https://github.com/haskell/actions/tree/main/setup#model-cabal-workflow-with-caching

This reverts commit 51bc8c1.

See: gentoo-haskell/gentoo-haskell#1074
See: haskell/cabal#8434
Signed-off-by: hololeap <hololeap@users.noreply.github.com>
hololeap added a commit to hololeap/hackport that referenced this issue Jun 3, 2023
build-tool-depends doesn't play nice with Setup.hs installs, and tries
to rebuild cabal-install/doctest locally. This has recently caused some
compilation errors when it tries to build cabal-install, so we'll just
remove it from build-tool-depends on the assumption that whoever is
running doctest-v2 probably has /usr/bin/cabal on their system.

Switch to `cabal-version: 'latest'` instead of a matrix of cabal
versions for github action.

Use template from haskell/actions/setup homepage where possible:
https://github.com/haskell/actions/tree/main/setup#model-cabal-workflow-with-caching

This reverts commit 51bc8c1.

See: gentoo-haskell/gentoo-haskell#1074
See: haskell/cabal#8434
Signed-off-by: hololeap <hololeap@users.noreply.github.com>
@andreabedini
Copy link
Collaborator

Re-reading this after a long time (and maybe missing some detail, it's a bit of a long thread across different repos).

From a cabal POV, it's not clear what value would bring to put exe components in a packagedb. We have packagedb to tell GHC where things are and GHC is never going to load an executable component (AFAIK).

In terms of allowing other tools to understand what build-tool-depends are required for which package component, this information is already included in plan.json

    {
      "type": "configured",
      "id": "tasty-discover-5.0.0-e-tasty-discover-93f8385e9e6e53f6adee25f304045702ba0cd8203e4f3ed512a68290e801e537",
      ...
      "bin-file": "/home/andrea/.local/state/cabal/store/ghc-9.4.5/tasty-discover-5.0.0-e-tasty-discover-93f8385e9e6e53f6adee25f304045702ba0cd8203e4f3ed512a68290e801e537/bin/tasty-discover"
    },
    ...
    {
      "type": "configured",
      "id": "tmp-build-depends-0.1.0.0-inplace",
      ...
      "exe-depends": [
        "tasty-discover-5.0.0-e-tasty-discover-93f8385e9e6e53f6adee25f304045702ba0cd8203e4f3ed512a68290e801e537"
      ],
      ...
    },

When this level of detail is not enough, one can always pick inside cabal-install's brain using cabal-install:lib, which has been available since cabal-install-3.8.

From a haskell.nix POV, I would say this is a short-coming of how shellFor is implemented and we should have all the information to make it work (of course the devil is in the detail but I will allow myself some optimism here 😄 ...). I'll comment some more on input-output-hk/haskell.nix#839.

@andreabedini
Copy link
Collaborator

Another aspect is cabal rebuilding build-tool-depends, even if they are already "available" somewhere. I believe that problem is complex and might not even be entirely solvable (I guess this balances the optimism in the previous comment :P).

cabal-install does use packagedb to pick pre-installed libraries when appropriate and maybe this is where the idea of .conf files for executables come from. Unfortunately, this mechanism has already plenty of issues (e.g. #6039, #8702).

Taking a step back, if the problem is that cabal will recompile tools already available, maybe a simple --dont-recompile-tools option can solve the problem? Or --with-build-tool tasty-discover=? or maybe even --with-build-tool my-package:tasty-discover:exe:tasty-discover= to get the solver scope right?

@ScottFreeCode
Copy link

ScottFreeCode commented Jul 26, 2023

@parsonsmatt @ParetoOptimalDev You actually don't need haskell.nix to have Nix treat Cabal as the source of truth. Just an underdocumented function in Nix called callCabal2nix. Examples here: https://gist.github.com/ScottFreeCode/9b0ea3e52fa56c93ea8ca14791e55bb4

(EDITED to add: This still runs into the same issue about v2 not working inside nix shells.)

@ScottFreeCode
Copy link

ScottFreeCode commented Jul 26, 2023

I have a straightforward reproduction without iohk's/iog's haskell.nix, here: https://gist.github.com/ScottFreeCode/c1adb5681c5b61c61373615f63a4a7d2

I've collected GitHub issues relating to this problem here: https://gist.github.com/ScottFreeCode/ef9f254e2dd91544bba4a068852fc81f

@andreabedini

@andreabedini
Copy link
Collaborator

@ScottFreeCode Thanks. I won't promise I will finish reading all this anytime soon but I appreciate your notes! They are valuable.

@andreabedini andreabedini added re: build-tool Concerning `build-tools` and `build-tool-depends` cabal-install: nix integration labels Jul 26, 2023
@ParetoOptimalDev
Copy link

ParetoOptimalDev commented Jul 26, 2023

You actually don't need haskell.nix to have Nix treat Cabal as the source of truth. Just an underdocumented function in Nix called callCabal2nix.

@ScottFreeCode Treating cabal as the source of truth requires also parsing cabal.project. To my knowledge callCabal2nix still doesn't do that.

There is a project that experiments with parsing the cabal.project in Nix I know of however. See Any way to interoperate with cabal/cabal.project only users in haskell-template?.

Right now in a Nix project that uses callCabal2Nix any source-repository-package entry in cabal.project causes an error in the development shell because cabal wants to download that package itself rather than using one provided by Nix.

One workaround that exists for making sure Nix packages are used is to remove all source-repository-package entries in cabal.project before building the Haskell package with nix as described here.

To clarify my use case is "Nix provides ghc, cabal, and all haskell packages" and then cabal is just used as a build tool without network access.

@gbaz
Copy link
Collaborator

gbaz commented Jul 26, 2023

@andreabedini I don't understand your comments. You seem to agree why a conf file would be useful -- so that cabal could use an existing executable that's been registered rather than compiling its own and duplicating work, just as it can already use an existing globally-registered library rather than compiling its own. However, you seem to go out of your way to propose a lot of other solutions that may address certain cases. What objection, if any, do you have to just having the executable db -- for use of cabal -- as this ticket proposes?

@ScottFreeCode
Copy link

@ParetoOptimalDev I'm not as familiar with source-repository-package; I did try to set up some cabal.project/multi-package builds in the "monorepo" gists and it seemed like they're working. (Aside from this issue of course, were Cabal v2 isn't able to use nix-shell executables; but either a straight nix-build or a v1 workaround are successful. Plus v2 works in the version without other executables needed in the build.) Is there something I missed that makes those examples not generally realistic/adequate?

That said/asked, I'm assuming there may still be relatively more complex cases that haskell.nix makes easier; but I know I only tried it because I couldn't find an explanation of how to make Nix piggyback on the package.cabal definition in general – which turned out to be possible (easier to set up and more efficient than haskell.nix, even!) but just rare to find examples or documentation to work off of. So I wanted to put it out there, not that there's never a need for haskell.nix, but that it's still possible to use that Cabal-first approach with "vanilla" Nix to start and switch to haskell.nix only when some additional requirement cannot be easily met with callCabal2nix, whatever those requirements may end up being.

@andreabedini
Copy link
Collaborator

(Assuming you refer to #8434 (comment))

What objection, if any, do you have to just having the executable db -- for use of cabal -- as this ticket proposes?

My only (soft) objection is that there are already many things that packagedb + solver cannot do at this moment. I agree with you (in #8434 (comment)) that this needs to be solved; but I am still not sure of the design.

(To be better discussed elsewhere but my question I get stuck trying to answer is: who owns packagedb and what is that for? if GHC owns it, then why does cabal-install writes cabal-install-specific build information in it? If Cabal owns it, would that be any different? (Cabal /= cabal-install, stack uses Cabal). Note GHC already throws away most of the content of packagedb entries and uses its own internal database (Which it calls "cache"))

@gbaz
Copy link
Collaborator

gbaz commented Jul 27, 2023

I don't think this would add any complexity to the solver. We just check if something suitable is already registered, and if not, solve for it. The solving for it part we do already -- so this comes before the solver, and eases its burden if anything. Nor would this necessarily be the same packagedb already in use -- just something conceptually similar to it.

So I understand your reservations about putting more load on those two items, but this proposal doesn't seem to do either.

That said, I would say on the packagedb that "ghc owns the packagedb jointly with Cabal-the-library" just as a matter of the engineering fact that I think they need to jointly agree on any changes to the datastructure.

@andreabedini
Copy link
Collaborator

andreabedini commented Jul 27, 2023

We just check if something suitable is already registered, and if not, solve for it.
...
Nor would this necessarily be the same packagedb already in use -- just something conceptually similar to it.

This is significantly different from what I had immagined i.e. having pre-installed exe units/components that the solver could choose to use rather than their corresponding source package.

In this case the difference with what I was proposing in #8434 (comment) is just a matter of input format: we can either register build-tool-depends in a file somewhere or list the available ones on the command line.

I am still not sure how this (either version) is going to work though.

We already have some logic to skip solving for them (as @matthewbauer observed a while ago), so we can easily surface that flag to project config and/or command line and skip solving for exe dependencies (or, with a little change, only some of them). This would roughly reproduce the bahaviour of v1 builds and installs w.r.t exe depends.

But how do we maintain purity in the cabal store? Could we use the exe binary hash instead of its installed-unit-it? The store is only used to avoid recompilation, it does not influence solving, so the externally provided tools won't have to match those hashes.

@ulidtko
Copy link

ulidtko commented Jun 28, 2024

We already have some logic to skip solving for them (as @matthewbauer observed input-output-hk/haskell.nix#839 (comment))

Thanks for the pointer & reminder. cabal v1-install saved my day once again!

The issue I had: trying to install a library, needed for a compiler bug repro case, with GHC HEAD compiler built in only the vanilla way.

No Nix, no flake, no cabal-project, no nothing — just direct usage of cabal-install & ghc-pkg. Bug & WTF counters overflow.


To elaborate... briefly.

cabal install -w $MYGHC --disable-executable-dynamic --lib rel8 — insistently wants to recompile hsc2hs as a build-tool dependency of whatever intermediate package. Fails, because the dynamic way is not enabled in this GHC — and, somehow the --disable-executable-dynamic flag gets ignored in this invocation.

Another bug?.. Does --lib silently ignore the executable-dynamic cli flag?

Without --lib, I can separately cabal install -w $MYGHC --disable-executable-dynamic hsc2hs just fine. The flag registers here, and it links with vanilla, as expected... But that's of no use! Because despite the freshly-built hsc2hs executable existing in the cabal-store, and even getting symlinked into ~/.cabal/bin — v2-install --lib still wants to build it again. And fails again, in exact same fashion.

Trying to add --constraint "any.hsc2hs installed" has led to a deep nowhere. I suddenly found myself resolving dependencies by hand, and just had to stop myself and step back.


... meanwhile: cabal v1-install -w $MYGHC --user --disable-shared --allow-newer rel8 — worked on 1st try 🎉

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cabal-install: nix integration re: build-tool Concerning `build-tools` and `build-tool-depends`
Projects
None yet
Development

No branches or pull requests

8 participants