Merged
Conversation
For consistency (other options with 'off') and simplicity (avoid double negatives), switch --no-X flags to on/off switches. For instance, instead of '--no-cache', we have '--cache (on | off)' that defaults to 'on'. Other flags get the same treatment e.g. '--group-fail-fast (on | off)' rather than '--group-fail-fast'. The one exception is '--print-package-set' -- which remains a flag -- though that is because it is really an alternative command, not a switch.
Adds the build-batch job to CI which actually builds the entire snapshot. It does this by dividing the snapshot into batch groups per --batch, then having a different job build each group via github action's matrix strategy and new clc-stackage --batch-index param. This allows us to build the entire snapshot such that overall CI time is reasonable i.e. ~ 1 hour.
Normally, each package in a given package set is used in two ways:
- It is written to generated.cabal's build-depends, hence built.
- Its exact version is written to cabal.project.local as a
constraint, for reproducibility of transitive dependencies.
We used excluded_pkgs.jsonc to opt out of one or both of these for a
given package e.g.
- Totally exclude cuda from the build.
- Exclude boot libs like text from having their exact versions
written, as these libs will be used transitively anyway, and exact
pins complicate building with multiple GHCs.
We since stumbled onto a third scenario: packages that we do not want
in build-depends yet benefit from being pinned. For example,
we do not want happy in build-depends (since it is an exe), yet we
would like to use stackage's pins, so that cabal does not choose a
version that might not work.
We therefore modify excluded_pkgs.jsonc to package_index.jsonc, which
separates these categories into different json keys:
- excluded: Same as before, these packages are filtered out from the
given package set.
- excluded_pinned: Packages that are filtered from the package set
yet we want their pins written anyway e.g. happy.
- unpinned: Packages to be built but should not be pinned e.g.
boot libs.
Collaborator
|
Wow, building entire Stackage as a few GitHub Action jobs is quite impressive :) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Besides some minor improvements and cli param shuffling (that hopefully doesn't affect anyone), there are two major things here:
Adds a new CI job
build-batchthat builds the entire package set on CI, which is actually possible now. The idea is, we have a bunch of jobs that all runwhere
Nis fixed and each job passes in a differentk. As long as we have enough jobs, this covers the entire set. Interestingly, I have recently had success building the entire set in one job i.e. a normal run. Previously this wasn't possible as it took too long, but maybe GHC has sped up just enough to get under the limit. Still, it's pretty close to the limit (~5.5 hours vs. free limit of 6 hours) whereas splitting the batches across jobs has each job take about an hour, so I think the index splitting is worth keeping, regardless. It's quick enough (total CI ~ 1.5 hrs) that I think it can enabled unconditionally.When testing the above, I ran into a build error where
haskell-src-exts's setup was not compatible with the newly releasedhappy-2.2.1. Stackage doesn't have this problem because it currently pinshappy-2.1.7, but because we currently havehappyin theexcluded_pkgs.json-- it needs to be excluded since it is an exe -- it does not get pinned, so cabal ends up taking the latest.In other words, we are getting bitten by
excluded_pkgscurrently pulling double-duty i.e. being used to exclude packages frombuild-dependsand also avoid writing exact version pins.To address this, I have split
excluded_pkgs(renamed topackage_index) to have three separate fields:excluded: As before, these packages are completely filtered from the package set i.e. are not written tobuild-dependsand are not pinned.excluded_pinned: Packages that are excluded frombuild-dependsthat we do want pinned e.g.happy.unpinned: Packages that we want built but not pinned e.g. boot packages liketext. We had these inexcludedbefore, and it was fine, since all boot libs are going to be in the build plan anyway. But since we're modifying the schema anyway, seems better to be clear about our demands.