How to programmatically interact with Cabal Store?

[swamp-agr]

Dear Cabal Team,

• I find myself in very difficult position and create this issue as a question.
• I will try to describe it as simple as possible and provide questions for further clarifications.

Situation

• Gentoo package maintainers kindly asked me to relax fay boundaries for aeson and two more packages.
• After that I was unable to release package on Hackage with HTTP 400 errors.
• Thus, I decided to upgrade fay to cabal-install 3.2.0.0.
• Once all hints from cabal check were applied, I tried to update Travis CI pipeline with haskell-ci utility.
• It switched installation to v2- nix-style builds.
• And it seems that v2- approach conflicts with current fay implementation.
  • fay is invoking ghc-pkg under the hood to access local packages separately (because fay-base conflicts with base through ambiguous modules names (Prelude, Data.Maybe, ...)).
  • v2-build put package in $HOME/.cabal/store.
• fay-base has dependent packages. They are also affected.

Questions

I am wondering, is it possible to access cabal-install internals, e.g. Distribution.Client.Store for interacting with Cabal Store (without replicating its modules)? This particular situation has many points of failures, thus, I kindly ask you to share the roadmap for cabal store:

• is it going to be stable?
• is it going to fully replace ghc-pkg mechanics? (if so, what is the estimated deprecation period?)
• Or maybe integrated with global package database?

Mitigation way

Since required modules are inside executable component cabal of cabal-install, I am not sure I could easily depend on them.

cabal v2-build all
Resolving dependencies...
cabal: Could not resolve dependencies:
[__0] trying: fay-0.24.1.1 (user goal)
[__1] next goal: cabal-install (dependency of fay)
[__1] rejecting: cabal-install-3.2.0.0 (does not contain library, which is
required by fay)

In this way, I have to replicate subset of cabal-install as internal library in order to unlock described way. I understand that it could be expensive and produce discrepancy with existing codebase in future.

I spent several hours in bug tracker and have not found similar issue with same questions. If I created duplicate for well-known issue, please refer to it and close this one as duplicate.

Best Regards,
@swamp-agr

[phadej]

Cabal store is just another GHC package database. Using e.g. https://hackage.haskell.org/package/cabal-install-parsers-0.4/docs/Cabal-Config.html you can get its root location.

Then you have to be somehow handle multiple package versions there, (e.g. you can have dozens of different hashable versions).

if fay somehow worked with sandboxes, it could work with store, if you can work out multiple package versions.
If the need to find just fay-base of a particular (known) version, then it could work.

Another alternative is to rely on package environments which cabal implicitly generates (though that is turned off by default). However there isn't a UI to manage them well. If there were, you could get very sandbox-like end result which uses a single global store.

• It is likely that configuration parsing will be part of "official" cabal-install library (rather then in my very tiny cabal-install-parsers lib).
• It is unlikely that store handling functions are what you need, and I'm not sure what their value would be in general (thus probably won't be in the library)
• There is no concrete plans for a library. It would be nice to have, but as I mention in other similar issues (e.g. Expose cabal.project.freeze parsing in the Cabal library #7224), the parsing code is not in state it could be extracted from rest of cabal-install into usable library. Someone have to do (a lot of) refactoring first, to separate that code.

[swamp-agr]

@phadej, thank you for detailed response!
Let me clarify my use-case and show why I have to explicitly ask about reading/writing to store.

use case explanation

• The fay "ecosystem", AFAIK, was previously built on re-using GHC package database.
• E.g. user was able to:
  • create the project,
  • include batteries (such fay-base, fay-dom, fay-jquery) as project dependencies,
  • write code for fay,
  • and then run fay as a compiler to create JS asset.
• fay during execution (either on its own or via user-supplied args) checks multiple package-dbs and resolves all dependencies.....

from user perspective:

• I don't want to manually check Cabal store or somewhere else.
• I usually want fay to automatically proceed everywhere and gather all information about given dependencies.
• With v1 approach and cabal (and even stack) it was possible to run something like:

cabal v1-exec -- fay --include $PROJECT_HOME/shared,$PROJECT_HOME/fay \
  --package fay-dom,fay-websockets \
  -o static/bundle.js $PROJECT_HOME/fay/Client.hs

Moreover, it was even possible to trigger dependent fay compilation via its API and extra TH (from Haskell code).

from developer perspective:

• Current approach was to rely on separate ghc-pkg process (!) to obtain info from global, local, custom (sandbox) package databases.
• It lacks of accessing Cabal Store (v2 nix-style builds are a bit problematic right now).

Next steps

• Instead of working with multiple package databases, I am going to invest further in package environments (--package-env) as an approach to create stack of package databases and combine them into single one with all necessary dependencies. It looks very promising.
• I will try to produce some tools for reading, locating required packages and resolving them.
• I will share my findings with you, once ensure that package environments fit described use-case.

Regarding my initial issue

• I will temporarily revert Travis CI to v1 style to unblock test failure analysis (that comes from boundary relaxation described previously) and Gentoo maintainers with blocked packaging activities.

[fwcd]

I am curious about this too. The Curry language compiler "KiCS2", which targets Haskell, relies on precompiling and storing its runtime in a custom package db, which seems to work fine with the v1-* commands, but fails with v2-install.

[andreabedini]

FWIW all parsing (including project parsing) is now exposed in Cabal-syntax and cabal-install libraries.