Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CI: {caching,test,bench}: mk cache aware of package dep versions #2532

Conversation

Anton-Latukha
Copy link
Collaborator

@Anton-Latukha Anton-Latukha commented Dec 23, 2021

  1. CI caches 3rd patry dependency binaries.

  2. That is why we do not want to trigger on all .cabal changes. We want to
    cache only when changes in .cabal descriptions lead to changes in versions of
    deps used, or in their compilation flags. freeze files:

For example the bits of the resulting .freeze file:

active-repositories: hackage.haskell.org:merge
constraints: any.Boolean ==0.2.4,
             ...
             any.HsYAML ==0.2.1.0,
             HsYAML -exe,
             any.HsYAML-aeson ==0.2.0.1,
             HsYAML-aeson -exe,
             any.JuicyPixels ==3.3.6,
             JuicyPixels -mmap,
             any.MemoTrie ==0.6.10,
             MemoTrie -examples,
             ...
             any.QuickCheck ==2.14.2,
             QuickCheck -old-random +templatehaskell,
             aeson -bytestring-builder -cffi -developer -fast,
             ...
             any.aeson-pretty ==0.8.9,
             aeson-pretty -lib-only,
             bifunctors +semigroups +tagged,
             any.blaze-textual ==0.2.2.1,
             blaze-textual -developer -integer-simple +native,
             ...
             any.brittany ==0.13.1.2,
             brittany -brittany-dev-lib -brittany-test-perf,
             ...
             any.clock ==0.8.2,
             clock -llvm,
             any.cmdargs ==0.10.21,
             cmdargs +quotation -testprog,
             ...
             any.comonad ==5.0.8,
             comonad +containers +distributive +indexed-traversable,
             ...
             any.hashtables ==1.2.4.2,
             hashtables -bounds-checking -debug -detailed-profiling -portable -sse42 +unsafe-tricks,
             haskell-language-server -alternatenumberformat +brittany +callhierarchy +class +eval +floskell +fourmolu +haddockcomments +hlint -ignore-plugins-ghc-bounds +importlens +modulename +ormolu -pedantic +pragmas +qualifyimportednames +refineimports -rename +retrie +splice +stylishhaskell +tactic,
             ...
             any.hlint ==3.2.7,
             hlint -ghc-lib +gpl -hsyaml +threaded,
             hls-eval-plugin -pedantic,
             hls-graph -embed-files -pedantic -stm-stats,
             hls-hlint-plugin -ghc-lib -hlint33 -pedantic,
             ...
             any.integer-logarithms ==1.0.3.1,
             integer-logarithms -check-bounds +integer-gmp,
             ...
             any.lens ==5.0.1,
             lens -benchmark-uniplate -dump-splices +inlining -j +test-hunit +test-properties +test-templates +trustworthy,
             ...
             any.scientific ==0.3.7.0,
             scientific -bytestring-builder -integer-simple,
             any.semigroupoids ==5.3.6,
             semigroupoids +comonad +containers +contravariant +distributive +tagged +unordered-containers,
             any.semigroups ==0.19.2,
             semigroups +binary +bytestring -bytestring-builder +containers +deepseq +hashable +tagged +template-haskell +text +transformers +unordered-containers,
             ...
             any.transformers-compat ==0.6.6,
             transformers-compat -five +five-three -four +generic-deriving +mtl -three -two,
             ...
             any.vector ==0.12.3.1,
             vector +boundschecks -internalchecks -unsafechecks -wall,
             any.vector-algorithms ==0.8.0.4,
             vector-algorithms +bench +boundschecks -internalchecks -llvm +properties -unsafechecks,
             ...
index-state: hackage.haskell.org 2021-11-29T12:28:17Z

You got the idea. freeze collects stuff for package compilation & all dependencies.

& this file satisfies the caching for dependencies agenda.

I also checked - it solves & collects all package configurations across all
subprojects in this one file.

1. CI caches 3rd patry dependency binaries.

2. That is why we do not want to trigger on all `.cabal` changes. We want to
cache only when changes in `.cabal` descriptions lead to changes in versions of
deps used, or in their compilation flags. `freeze` files:

For example the bits of the resulting `.freeze` file:

active-repositories: hackage.haskell.org:merge
constraints: any.Boolean ==0.2.4,
             ...
             any.HsYAML ==0.2.1.0,
             HsYAML -exe,
             any.HsYAML-aeson ==0.2.0.1,
             HsYAML-aeson -exe,
             any.JuicyPixels ==3.3.6,
             JuicyPixels -mmap,
             any.MemoTrie ==0.6.10,
             MemoTrie -examples,
             ...
             any.QuickCheck ==2.14.2,
             QuickCheck -old-random +templatehaskell,
             aeson -bytestring-builder -cffi -developer -fast,
             ...
             any.aeson-pretty ==0.8.9,
             aeson-pretty -lib-only,
             bifunctors +semigroups +tagged,
             any.blaze-textual ==0.2.2.1,
             blaze-textual -developer -integer-simple +native,
             ...
             any.brittany ==0.13.1.2,
             brittany -brittany-dev-lib -brittany-test-perf,
             ...
             any.clock ==0.8.2,
             clock -llvm,
             any.cmdargs ==0.10.21,
             cmdargs +quotation -testprog,
             ...
             any.comonad ==5.0.8,
             comonad +containers +distributive +indexed-traversable,
             ...
             any.hashtables ==1.2.4.2,
             hashtables -bounds-checking -debug -detailed-profiling -portable -sse42 +unsafe-tricks,
             haskell-language-server -alternatenumberformat +brittany +callhierarchy +class +eval +floskell +fourmolu +haddockcomments +hlint -ignore-plugins-ghc-bounds +importlens +modulename +ormolu -pedantic +pragmas +qualifyimportednames +refineimports -rename +retrie +splice +stylishhaskell +tactic,
             ...
             any.hlint ==3.2.7,
             hlint -ghc-lib +gpl -hsyaml +threaded,
             hls-eval-plugin -pedantic,
             hls-graph -embed-files -pedantic -stm-stats,
             hls-hlint-plugin -ghc-lib -hlint33 -pedantic,
             ...
             any.integer-logarithms ==1.0.3.1,
             integer-logarithms -check-bounds +integer-gmp,
             ...
             any.lens ==5.0.1,
             lens -benchmark-uniplate -dump-splices +inlining -j +test-hunit +test-properties +test-templates +trustworthy,
             ...
             any.scientific ==0.3.7.0,
             scientific -bytestring-builder -integer-simple,
             any.semigroupoids ==5.3.6,
             semigroupoids +comonad +containers +contravariant +distributive +tagged +unordered-containers,
             any.semigroups ==0.19.2,
             semigroups +binary +bytestring -bytestring-builder +containers +deepseq +hashable +tagged +template-haskell +text +transformers +unordered-containers,
             ...
             any.transformers-compat ==0.6.6,
             transformers-compat -five +five-three -four +generic-deriving +mtl -three -two,
             ...
             any.vector ==0.12.3.1,
             vector +boundschecks -internalchecks -unsafechecks -wall,
             any.vector-algorithms ==0.8.0.4,
             vector-algorithms +bench +boundschecks -internalchecks -llvm +properties -unsafechecks,
             ...
index-state: hackage.haskell.org 2021-11-29T12:28:17Z
```

You got the idea. `freeze` collects stuff for dependencies.

I also checked - it solves & collects all package configuration across all
subprojects in one file.
Before we not had this at all.

Now - most cases would use it.

But for example, all HLS subprojects can not solve single version of deps for
GHC 9.0 at once.

And depending on `.cabal` is not the right way, as it means to account of only 1
configuration of many in the project.
@Anton-Latukha Anton-Latukha force-pushed the 2021-12-23-ci-make-caches-depend-on-package-vers branch from 0d520c6 to bed87e8 Compare December 23, 2021 13:17
@Anton-Latukha
Copy link
Collaborator Author

Would note that caching .cabal - caches only 1 of (including some test samples) 53 .cabal files in monorepo (& has other implications).

@Anton-Latukha
Copy link
Collaborator Author

Anton-Latukha commented Dec 23, 2021

So.

When freeze is not possible - it essentially does what it was doing before this PR (navigates on Hackage index). But if freeze is possible - it caches what Cabal chooses.

@Anton-Latukha Anton-Latukha marked this pull request as ready for review December 23, 2021 16:42
@Anton-Latukha
Copy link
Collaborator Author

Anton-Latukha commented Dec 23, 2021

I've got another idea how to make caching even more fault tolerant. (So caching would be able to run & cache even inside #2503 environment, where dependency versions for all targets don't add up). (opened #2534).

.github/workflows/bench.yml Outdated Show resolved Hide resolved
@Anton-Latukha
Copy link
Collaborator Author

Anton-Latukha commented Dec 23, 2021

Added a workaround to GitHub CI in 0c521cc.

@Anton-Latukha Anton-Latukha force-pushed the 2021-12-23-ci-make-caches-depend-on-package-vers branch from 5209a5c to 0c521cc Compare December 23, 2021 22:48
Copy link
Member

@jneira jneira left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks!

@jneira jneira merged commit 2aaa5b8 into haskell:master Dec 24, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants