source-repository-package pins and shellFor

[dhess]

We've just added a couple of source-repository-package pins in our cabal.project to work around a few upstream issues, as I suspect many of you may have done recently:

-- https://github.com/vincenthz/hs-memory/issues/86
-- https://github.com/haskell-infra/hackage-trustees/issues/330#issuecomment-1019507291
source-repository-package
  type: git
  location: https://github.com/tfausak/hs-memory
  tag: 3cf661a
  --sha256: 06y4xyigajhdcd59dgwasv49yr95869q3p74a441yls5vb86v28c

-- https://github.com/haskell-foundation/foundation/pull/555
source-repository-package
    type: git
    location: https://github.com/TomMD/foundation
    tag: 0bb195e
    subdir: foundation basement
    --sha256: 1rwy3qxrf8zxycfksidwrffqbazacchqx19w5q1rvdvh7ab6wd75

Our Nix builds work fine with this, but we noticed that our project's shellFor doesn't include these dependencies, which leads to two issues:

1. Cabal ends up building a lot of stuff locally.
2. If the developer hasn't run cabal update (which we otherwise don't need to do), the shell build fails outright.

It turns out I had pointed this out 2 years ago when I asked about using Nix for overrides: #510 I just forgot about it in the interim 😂

Anyway, is there something I can add to our project's haskell.nix config so that shellFor includes these dependencies? They're obviously being built by our nix-build builds as those are working as expected, and we'd obviously like to leverage our binary cache, especially considering how many packages are affected by these particular pins.

[michaelpj]

Hmm, I thought there was an issue for this but I can't find it.

Unfortunately, this is a cabal issue. haskell.nix puts the built packages in the package-db, but cabal insists on building source-repository-packages itself, regardless.

AFAIK the only workaround is to delete them from your cabal.project after entering the shell, which is pretty bad.

[dhess]

Bummer. Thanks for the quick response, in any case. Shall I keep this open, or would you prefer to close it given that it sounds like it's out of haskell.nix's hands?

[georgefst]

Unfortunately, this is a cabal issue. haskell.nix puts the built packages in the package-db, but cabal insists on building source-repository-packages itself, regardless.

@michaelpj Is there a particular issue on cabal's tracker that you have in mind, which would solve this?

These look possibly relevant:

• source-repository-package not optional haskell/cabal#5444 (includes various proposals around treating source-repository-packages more like non-local dependencies)
• Add package-db flag for v2 commands haskell/cabal#7676 (proper package-db support for v2- commands)

[michaelpj]

Yeah, I think haskell/cabal#5444 is the clearest place where other people bump up against this behaviour.

[codygman]

It looks like a Cabal maintainer said just 24 days ago:

I'd welcome a feature where tar.gz local and remote packages, and source-repository-packages were treated like override package-repositories (i.e. like a single version "hackage"-like repositories) rather than a must have packages.
i.e. if package in source-repository-package is needed, than it's build, but cabal build all won't match on it.

But indicates they won't be the one to make the change:

That is a somewhat radical change (i.e. only truly local unpacked packages were "project" / "local" packages), which needs some dedicated person to push through. I won't be one.

So this can be fixed but will require that change.

Though it also seems that conditionals in cabal.project can replace the hack of deleting things from cabal.project if you use cabal-install 3.8+ right now.

[L-as]

You can use https://github.com/mlabs-haskell/haskell-nix-extra-hackage to solve this, but it will not work for non-Nix users obviously.

[peterstorm]

You can use https://github.com/mlabs-haskell/haskell-nix-extra-hackage to solve this, but it will not work for non-Nix users obviously.

Could you describe how it works, if possible?

[michaelpj]

IOG now has a alternative solution for building your own internal Hackage that's a bit simpler than that (IMO), and can avoid the use of soruce-repository-packages and get good caching behaviour:

• https://github.com/andreabedini/foliage
• Example repository here with docs: https://github.com/input-output-hk/cardano-haskell-packages

[TeofilC]

@andreabedini this is the issue about haskell.nix not dealing well with source-repository-packages