Improve Cabal version and remote-repo handling

[mietek]

Halcyon was originally built with the assumption that the version number of cabal-install and the contents of its package database don’t need to be treated as app dependencies.

Unfortunately, build directories created with cabal-install 1.20.0.3 are not usable with cabal-install 1.22.0.0 (haskell/cabal#2320). Additionally, recent Hackage issues (haskell/hackage-server#305) and the slow adoption of constraints files put the original assumption further into question. Therefore, I propose to:

• Ensure that sandbox directories, build directories, and install directories are built with the right version of cabal-install, and that changes to Cabal magic files are reflected in the magic hash.
• Allow setting HALCYON_CABAL_VERSION with a .halcyon/cabal-version magic file.
• Allow setting HALCYON_CABAL_REMOTE_REPO with a .halcyon/cabal-remote-repo magic file.
• Avoid needless warnings and cabal update runs when using Stackage as the Cabal remote-repo.

This will have the interesting benefit of making it practical for install archives of popular Haskell apps to be included in Halcyon public storage, built using Stackage and not explicit version constraints, and for Halcyon to be suggested as a simple installation method for these apps.

The work is already done and awaits further testing. The remaining questions are:

• Should changing the Cabal remote-repo trigger a sandbox rebuild, just like changing the Cabal version? If yes, all Halcyon public archives would have to be built using the default Halcyon Cabal remote-repo.
• Therefore, should Stackage LTS be the default Cabal repository for Halcyon?

CC @begriffs @dbp @jspahrsummers @ozataman @pbrisbin @rehno-lindeque @snoyberg

[jspahrsummers]

Should changing the Cabal remote-repo trigger a sandbox rebuild, just like changing the Cabal version?

👍 I think this makes sense no matter what.

Therefore, should Stackage LTS be the default Cabal repository for Halcyon?

I don't think I have a good sense of what tradeoffs would be involved here. The positives seem self-evident—what are the potential downsides?

[pbrisbin]

Should changing the Cabal remote-repo trigger a sandbox rebuild

I understand erring on the side that decreases the likelihood of edge-case errors, but in practice I don't think this should be required. Practically speaking, we're talking about Hackage and Stackage. Stackage is a subset of Hackage so switching repos (but not changing dependency versions) won't be changing the code you're installing and therefore shouldn't require a pre-emptive rebuild. Only if you're using a remote-repo that has the same named and versioned packages, but with changes in their source, would that be the case -- and I don't know how valuable it is to support that scenario. Even if I'm setting up a custom repo with forked packages, I should be changing version numbers in those forks.

should Stackage LTS be the default Cabal repository for Halcyon?

The only downside I know of is out of date or missing packages. There is a compromise for this scenario though:

You can still use Hackage, but with a constraints file that is the package set from Stackage. Stackage even provides these files for download (albeit in cabal freeze format currently). In theory, this set of dependencies has been vetted, so stability should be increased. If however, you need something new or not present on Stackage, you can just add it. Your main base of packages is fixed at stable versions, but you can install something additional without moving completely away from the "stable" approach. I don't know if this is worth considering for Halcyon, but thought I'd throw it out there.

[mietek]

I don't think this should be required. … Only if you're using a remote-repo that has the same named and versioned packages, but with changes in their source…

Generally, I agree.

Note, however, that “Package maintainers and Hackage trustees are allowed to edit certain bits of package metadata after a release, without uploading a new tarball.” For example, a bug was found in random-1.0.1.3, and so this particular version was made uninstallable (by having its base dependency changed to >=5 && <5). The recent Hackage issues I mentioned (haskell/hackage-server#305) were caused by a bug in hackage-server, which temporarily reversed the order of these metadata revisions.

That being said, I don’t think it’s practical to try and prevent issues caused by silent metadata changes, as it would require Halcyon to perform quite a bit of additional work.

[mietek]

You can still use Hackage, but with a constraints file that is the package set from Stackage. Stackage even provides these files for download…

Currently, Halcyon expects constraints files (either .halcyon/constraints or cabal.config) which don’t include any extraneous constraints. This should be made clear in the documentation, and is now tracked as mietek/halcyon-website#1.

I’ve been looking into ways to support using the Stackage LTS cabal.config files (which include a constraint for each and every package in Stackage), but I don’t have a good answer yet.

[pbrisbin]

Halcyon expects constraints files ... which don’t include any extraneous constraints

Is this something to do with cabal or is Halcyon enforcing that itself? I don't see the benefit to this extra restriction, and (as in this case) it is limiting some possible use-cases.

[mietek]

The question is, what is a constraints file supposed to be?

Currently, Halcyon wants a constraints file for a particular app to list all Cabal packages which are the app’s direct or indirect dependencies. This matches the output of cabal freeze, and indeed, compatibility with cabal.config files generated by cabal freeze is the main reason for the current state of things, as Halcyon uses cabal freeze internally.

It’s likely that switching from cabal freeze to cabal install --dry-run could open up more possibilities (#52).

[mietek]

With 43ae61b, the items mentioned in the first comment are now live.

Be aware these changes will invalidate your existing sandbox directory and build directory archives, but shouldn’t cause any build failures.

When restoring an archive which doesn’t include a Cabal version number, Halcyon will emit a warning, and fall back to building the required items:

   *** WARNING: Failed to restore sandbox directory

   *** WARNING: Failed to restore build directory