IMPORTANT: If you're here because you need to publish a new version of your package, you probably want to read the section on adding a package from GitHub.
This is a Cabal package repository ("CHaP") whose purpose is to contain all the Haskell packages used by the Cardano open-source project which are not on Hackage.
The package repository itself is available here.
It is built from a git repository which
contains the metadata specifying all the package versions. The package repository is built using
foliage.
A package repository is essentially a mapping from package name and version to the source distribution for the package. Most Haskell programmers will be familiar with the package repository hosted on Hackage, which is enabled by default in Cabal.
However, Cabal supports the use of additional package repositories. This is convenient for users who can't or don't want to put their packages on Hackage.
Using source-repository-package stanzas is another common way of getting dependencies
that are not on Hackage. Both have their place: CHaP gives us proper versioning
and simpler setup, source-repository-packages are useful for ad-hoc use of
patched or pre-release versions.
Crucially, additional Cabal package repositories like CHaP and source-repository-package
stanzas are compatible and source-repository-packages always win. That is,
they interact in the same way as Hackage and source-repository-packages do. This gives us
behaviour that we want: ad-hoc source-repository-package stanzas will override
packages from Hackage or CHaP.
To use CHaP from cabal, add the following lines to your
cabal.project file:
repository cardano-haskell-packages
url: https://input-output-hk.github.io/cardano-haskell-packages
secure: True
root-keys:
3e0cce471cf09815f930210f7827266fd09045445d65923e6d0238a6cd15126f
443abb7fb497a134c343faf52f0b659bd7999bc06b7f63fa76dc99d631f9bea1
a86a1f6ce86c449c46666bda44268677abf29b5b2d2eb5ec7af903ec2f117a82
bcec67e8e99cabfa7764d75ad9b158d72bfacf70ca1d0ec8bc6b4406d1bf8413
c00aae8461a256275598500ea0e187588c35a5d5d7454fb57eac18d9edb86a56
d4a35cd3121aa00d18544bb0ac01c3e1691d618f462c46129271bccf39f7e8ee
The package repository will be understood by cabal, and can be updated with cabal update.
The index-state for the package repository can also be pinned as usual.
You can either just use a single index-state for both Hackage and CHaP:
index-state: 2022-08-25T00:00:00Z
or you can specify a different index-state for each repository:
index-state:
, hackage.haskell.org 2022-12-31T00:00:00Z
, cardano-haskell-packages 2022-08-25T00:00:00Z
Note that a second index-state stanza completely ovverides the first, so
index-state: 2022-12-31T00:00:00Z
index-state: cardano-haskell-packages 2022-08-25T00:00:00Z
would override the index-state for Hackage to HEAD (since HEAD is the default).
To use CHaP with haskell.nix, do the following:
- Add the package repository to your
cabal.projectas above. - Setup a fetcher for the package repository. The easiest way is to use a flake input, such as:
inputs.CHaP = {
url = "github:input-output-hk/cardano-haskell-packages?ref=repo";
flake = false;
};
- Tell haskell-nix to map the CHaP url to the appropriate nix store path using the
inputMapargument of one of haskell.nix project functions. UsingcabalProjectthis would look like the following:
cabalProject {
...
inputMap = { "https://input-output-hk.github.io/cardano-haskell-packages" = CHaP; };
}
When you want to update the state of CHaP, you can simply update the flake input
(in the example above you would run nix flake lock --update-input CHaP).
If you have CHaP configured correctly, then when you run cabal build from inside a haskell.nix
shell, you should not see any of the packages in CHaP being built by cabal.
The exception is if you have a source-repository-package stanza which overrides a dependency of one
of the packages in CHaP. Then cabal will rebuild them both. If this becomes a problem,
you can consider adding the patched package to CHaP itself,
see below.
Warning: Haskell.nix cannot parse the index-state syntax with multiple
repositories (see haskell.nix#1869).
You can use the following workaround to appease both Haskell.nix and cabal.
-- haskell.nix will parse this (https://github.com/input-output-hk/haskell.nix/issues/1869)
index-state: 2022-12-31T00:00:00Z
-- cabal will overwrite the above with this
index-state:
, hackage.haskell.org 2022-12-31T00:00:00Z
, cardano-haskell-packages 2022-08-25T00:00:00Z
When adding a package, it is important to use a timestamp (see below)
that is greater than any other timestamp in the index. Indeed, cabal users rely on
the changes to the repository index to be append-only. A non append-only
change to the package index would change the repository index state as
pinned by index-state, breaking reproducibility.
Using the current date and time (e.g. date --utc +%Y-%m-%dT%H:%M:%SZ)
works alright but if you are sending a PR you need to consider the
possibility that another developer has inserted a new (greater) timestamp
before your PR got merged. We have CI check that prevents this from
happening, and we enforce FF-only merges.
When downstream users pull a package from CHaP, cabal will build it based only on the
information in the cabal file. This means that if your package needs any additional configuration
to build, then it will simply be broken for downstream users unless they replicate that
configuration.
Typical examples of this are anything that you add in cabal.project:
constraintsallow-newersource-repository-package
Try to avoid adding packages to CHaP that need extra configuration in this way. This is not a hard rule, but please bear in mind that doing so requires all downstream consumers to replicate that configuration, making the package much harder to use.
At some point we may start checking this, e.g. by trying to build each added package in isolation.
Package versions are defined using metadata files _sources/$pkg_name/$pkg_version/meta.toml,
which you can create directly. The metadata files have the following format:
# REQUIRED timestamp at which the package appears in the index
timestamp = 2022-03-29T06:19:50+00:00
# REQUIRED URL pointing to the source code tarball (not necessarily a sdist)
url = 'https://github.com/input-output-hk/ouroboros-network/tarball/fa10cb4eef1e7d3e095cec3c2bb1210774b7e5fa'
# OPTIONAL subdirectory inside the tarball where the package is located
subdir = 'typed-protocols'There is a convenience script ./scripts/add-from-github.sh to simplify
adding a package from a GitHub repository.
$ ./scripts/add-from-github.sh
Usage add-from-github.sh [-r REVISION] [-v VERSION] REPO_URL REV [SUBDIRS...]
-r REVISION adds .0.0.0.0.REVISION to the package version
-v VERSION uses VERSION as the package versionThe script will:
- Find the cabal files in the repo (either at the root or in the specified subdirectories)
- Obtain package names and versions from the cabal files
- Create the corresponding
meta.tomlfiles - Commit the changes to the git repository
You can tell the script to override the package version either by passing the version explicitly or by adding a "revision number" (see below).
CHaP should mostly contain versions of packages which are not on Hackage.
If you need to patch a version of a package on Hackage, then there are two options:
- For short lived forks, use a
source-repository-packagestanza by preference. - For long-lived forks (because e.g. the maintainer is unresponsive or the patch is large and will take time to upstream), then we can consider releasing a patched version in CHaP.
The main constraint when adding a patched version to CHaP is to be sure that we use a version number that won't ever conflict with a release made by upstream on Hackage. There are two approaches to doing this:
- Release the package in CHaP under a different name (for the fork). This is very safe, but may not be possible if the dependency is incurred via a package we don't control, as then we can't force it to depend on the renamed package.
- Release the package under a version that is very unlikely to be used by upstream.
The scheme that we typically use is to take the existing version number, add four zero components and then a patch version, e.g.
1.2.3.4.0.0.0.0.1.
IMPORTANT: if you release a patched package to CHaP, make sure to open an issue about it so we can keep track of which patched packages we have. Ideally, include the conditions under which we can deprecate it, e.g. "can deprecate either when it's fixed upstream or when package X removes their dependency on it".
Sometimes it is useful to test in advance how a new package or a cabal file revision affects things.
The first steps are always the same, you need a built version of your modified CHaP locally:
- Make a local checkout of CHaP and make the intended changes
- Build the repository with
nix develop -c foliage build
For the rest of this section we will assume the built repository is in
/home/user/cardano-haskell-packages/_repo.
You can test a locally built CHaP with a small test project consisting of just a
cabal.project file:
-- Give it a different name to avoid cabal confusing it with the
-- real CHaP
repository cardano-haskell-packages-local
-- Point this to the *built* repository
url: file:/home/user/cardano-haskell-packages/_repo
secure: True
-- You can skip the root-keys field
-- Add all the packages you want to try building
extra-packages:
, cardano-prelude-0.1.0.0
You need to tell cabal about the new repository with cabal update (you might need to
clear out ~/.cabal/packages/cardano-haskell-packages-local if you've been
editing your repository destructively).
Then you can build whatever package version you want with cabal:
$ cabal build cardano-preludeYou can troubleshoot a failed build plan using the cabal flags --constraint, --allow-newer and --allow-older. Once you have obtained a working build plan, you should revise you cabal file with appropriate constraints.
You can build packages from CHaP using Nix like this:
nix build
--override-input CHaP /home/user/cardano-haskell-packages/_repo
.#haskellPackages.x86_64-linux.ghc8107.plutus-core."1.1.0.0"
This will build all the components of that package version that CHaP cares about, namely libraries and executables (test suites and benchmarks are not built).
We need to use --override-input because the CHaP flake relies on a built repository.
By default it points to a built repository on the main CHaP repo branch.
But if you have just produced your own built repository (see above) then you want to
use that instead, and --override-input will let you do that.
If you want to test a locally built CHaP against a project that uses CHaP via haskell.nix, you can build the project while overriding CHaP with your local version.
$ nix build --override-input CHaP path:/home/user/cardano-haskell-packages/_repoNote that you will need to change the index-state for cardano-haskell-packages
to be newer than the repository you just built, otherwise cabal will ignore your
new package versions!
Also, you you can examine the build plan without completing the build:
$ nix build .#project.plan-nix.json \
--out-link plan.json \
--override-input CHaP path:/home/user/cardano-haskell-packages/_repoThis is useful if you jsut want to see whether cabal is able to successfully resolve dependencies and see what versions it picked.
It's totally fine to release a package in CHaP to Hackage.
The thing to avoid is to have the same package version in both repositories.
The simplest solution is to just make sure to use a higher major version number when you start releasing to Hackage, even if this looks a bit odd.
For example, if CHaP contains X-1.0 and X-1.1, then the first Hackage release should be X-1.2 or X-2.0.
Since packages are released to CHaP simply by making PRs, CHaP uses CODEOWNERS to determine whose approval is needed to release a package.
The general rules are:
- If a package is clearly owned by a particular team, then set that team as the CODEOWNER.
- Prefer to use GitHub teams over individual accounts wherever possible.
- In the case of patched packages, the owner should be whichever team owns the package that causes the dependency on the package that needs patching.
Generally, use your judgement about what's appropriate.
The CI for CHaP does the following things:
- Checks that the timestamps in the git repository are monotonically increasing through commits. Along with requiring linear history, this ensures that package repository that we build is always an extension of the previous one.
- Builds the package repository from the metadata using
foliage. - Deploys the package repository to the
repobranch, along with some static web content. - Builds a small set of packages using the newly built repository, to flush out any build issues.
- We build with all the major GHC versions we expect to be in use.
- At the moment we don't build all the packages in the repository, only the latest versions of a fixed set.
If you just want or test changes to CHaP, you should make a fork. If you want to replicate the setup from scratch you can clone this template.
If you have trouble, open an issue, or contact the maintainers:
- Andrea Bedini (andrea.bedini@iohk.io)
- Michael Peyton Jones (michael.peyton-jones@iohk.io)