CI: {caching,test,bench}: mk cache aware of package dep versions (#2532)

* CI: {caching,test,bench}: mk cache aware of package dep versions 1. CI caches 3rd patry dependency binaries. 2. That is why we do not want to trigger on all `.cabal` changes. We want to cache only when changes in `.cabal` descriptions lead to changes in versions of deps used, or in their compilation flags. `freeze` files: For example the bits of the resulting `.freeze` file: active-repositories: hackage.haskell.org:merge constraints: any.Boolean ==0.2.4, ... any.HsYAML ==0.2.1.0, HsYAML -exe, any.HsYAML-aeson ==0.2.0.1, HsYAML-aeson -exe, any.JuicyPixels ==3.3.6, JuicyPixels -mmap, any.MemoTrie ==0.6.10, MemoTrie -examples, ... any.QuickCheck ==2.14.2, QuickCheck -old-random +templatehaskell, aeson -bytestring-builder -cffi -developer -fast, ... any.aeson-pretty ==0.8.9, aeson-pretty -lib-only, bifunctors +semigroups +tagged, any.blaze-textual ==0.2.2.1, blaze-textual -developer -integer-simple +native, ... any.brittany ==0.13.1.2, brittany -brittany-dev-lib -brittany-test-perf, ... any.clock ==0.8.2, clock -llvm, any.cmdargs ==0.10.21, cmdargs +quotation -testprog, ... any.comonad ==5.0.8, comonad +containers +distributive +indexed-traversable, ... any.hashtables ==1.2.4.2, hashtables -bounds-checking -debug -detailed-profiling -portable -sse42 +unsafe-tricks, haskell-language-server -alternatenumberformat +brittany +callhierarchy +class +eval +floskell +fourmolu +haddockcomments +hlint -ignore-plugins-ghc-bounds +importlens +modulename +ormolu -pedantic +pragmas +qualifyimportednames +refineimports -rename +retrie +splice +stylishhaskell +tactic, ... any.hlint ==3.2.7, hlint -ghc-lib +gpl -hsyaml +threaded, hls-eval-plugin -pedantic, hls-graph -embed-files -pedantic -stm-stats, hls-hlint-plugin -ghc-lib -hlint33 -pedantic, ... any.integer-logarithms ==1.0.3.1, integer-logarithms -check-bounds +integer-gmp, ... any.lens ==5.0.1, lens -benchmark-uniplate -dump-splices +inlining -j +test-hunit +test-properties +test-templates +trustworthy, ... any.scientific ==0.3.7.0, scientific -bytestring-builder -integer-simple, any.semigroupoids ==5.3.6, semigroupoids +comonad +containers +contravariant +distributive +tagged +unordered-containers, any.semigroups ==0.19.2, semigroups +binary +bytestring -bytestring-builder +containers +deepseq +hashable +tagged +template-haskell +text +transformers +unordered-containers, ... any.transformers-compat ==0.6.6, transformers-compat -five +five-three -four +generic-deriving +mtl -three -two, ... any.vector ==0.12.3.1, vector +boundschecks -internalchecks -unsafechecks -wall, any.vector-algorithms ==0.8.0.4, vector-algorithms +bench +boundschecks -internalchecks -llvm +properties -unsafechecks, ... index-state: hackage.haskell.org 2021-11-29T12:28:17Z ``` You got the idea. `freeze` collects stuff for dependencies. I also checked - it solves & collects all package configuration across all subprojects in one file. * CI: {caching,test,bench}: freeze: allow to fail Before we not had this at all. Now - most cases would use it. But for example, all HLS subprojects can not solve single version of deps for GHC 9.0 at once. And depending on `.cabal` is not the right way, as it means to account of only 1 configuration of many in the project. * CI: {caching,test,bench}: fx cache keys * CI: {caching,test,bench}: freeze: output result into log * CI: caching: add workaround to allow last step to fail * CI: {caching,test,bench}: use only freeze file hash
haskell · Dec 24, 2021 · 2aaa5b8 · 2aaa5b8
1 parent ab8c52b
commit 2aaa5b8
Show file tree

Hide file tree

Showing 3 changed files with 42 additions and 6 deletions.
diff --git a/.github/workflows/bench.yml b/.github/workflows/bench.yml
@@ -74,6 +74,15 @@ jobs:
         INDEX_STATE1=$(echo "$INDEX_STATE_ENTRY" | cut -d' ' -f2 | tr ':' '-')
         echo "INDEX_STATE=$INDEX_STATE1" >> $GITHUB_ENV
 
+    - name: Form the package list ('cabal.project.freeze')
+      continue-on-error: true
+      run: |
+        cabal v2-freeze
+        echo ''
+        echo 'Output:'
+        echo ''
+        cat 'cabal.project.freeze'
+
     - name: Hackage sources cache
       uses: actions/cache@v2
       env:
@@ -84,12 +93,13 @@ jobs:
         restore-keys: ${{ env.cache-name }}-
 
     - name: Compiled deps cache
+      id: compiled-deps
       uses: actions/cache@v2
       env:
         cache-name: compiled-deps
       with:
         path: ${{ steps.HaskEnvSetup.outputs.cabal-store }}
-        key:  ${{ env.cache-name }}-${{ runner.os }}-${{ matrix.ghc }}-${{ env.INDEX_STATE }}-${{ hashFiles('cabal.project') }}
+        key:  ${{ env.cache-name }}-${{ runner.os }}-${{ matrix.ghc }}-${{ env.INDEX_STATE }}-${{ hashFiles('cabal.project.freeze') }}
         restore-keys: |
               ${{ env.cache-name }}-${{ runner.os }}-${{ matrix.ghc }}-${{ env.INDEX_STATE }}-
               ${{ env.cache-name }}-${{ runner.os }}-${{ matrix.ghc }}-

diff --git a/.github/workflows/caching.yml b/.github/workflows/caching.yml
@@ -125,6 +125,15 @@ jobs:
           INDEX_STATE1=$(echo "$INDEX_STATE_ENTRY" | cut -d' ' -f2 | tr ':' '-')
           echo "INDEX_STATE=$INDEX_STATE1" >> $GITHUB_ENV
 
+      - name: Form the package list ('cabal.project.freeze')
+        continue-on-error: true
+        run: |
+          cabal v2-freeze
+          echo ''
+          echo 'Output:'
+          echo ''
+          cat 'cabal.project.freeze'
+
       # 2021-12-02: NOTE: Cabal Hackage source tree storage does not depend on OS or GHC really,
       # but can depend on `base`.
       # But this caching is happens only inside `master` for `master` purposes of compiling the deps
@@ -135,9 +144,11 @@ jobs:
         env:
           cache-name: hackage-sources
         with:
-          path: ${{ env.CABAL_PKGS_DIR }}
-          key:          ${{ env.cache-name }}-${{ env.INDEX_STATE }}
-          restore-keys: ${{ env.cache-name }}-
+          path:  ${{ env.CABAL_PKGS_DIR }}
+          key:   ${{ env.cache-name }}-${{ env.INDEX_STATE }}-${{ hashFiles('cabal.project.freeze') }}
+          restore-keys: |
+                 ${{ env.cache-name }}-${{ env.INDEX_STATE }}-
+                 ${{ env.cache-name }}-
 
       - name: Compiled deps cache
         id: compiled-deps
@@ -146,7 +157,7 @@ jobs:
           cache-name: compiled-deps
         with:
           path: ${{ steps.HaskEnvSetup.outputs.cabal-store }}
-          key:  ${{ env.cache-name }}-${{ runner.os }}-${{ matrix.ghc }}-${{ env.INDEX_STATE }}-${{ hashFiles('cabal.project') }}
+          key:  ${{ env.cache-name }}-${{ runner.os }}-${{ matrix.ghc }}-${{ env.INDEX_STATE }}-${{ hashFiles('cabal.project.freeze') }}
           restore-keys: |
                 ${{ env.cache-name }}-${{ runner.os }}-${{ matrix.ghc }}-${{ env.INDEX_STATE }}-
                 ${{ env.cache-name }}-${{ runner.os }}-${{ matrix.ghc }}-
@@ -171,3 +182,8 @@ jobs:
         continue-on-error: true
         run: |
           cabal $cabalBuild || cabal $cabalBuild || cabal $cabalBuild
+
+      # Despite the `continue-on-error: true` directive - CI does not ignore the return code of the last step
+      - name: Workaround to CI platform
+        run: |
+          true
diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml
@@ -123,6 +123,15 @@ jobs:
           INDEX_STATE1=$(echo "$INDEX_STATE_ENTRY" | cut -d' ' -f2 | tr ':' '-')
           echo "INDEX_STATE=$INDEX_STATE1" >> $GITHUB_ENV
 
+      - name: Form the package list ('cabal.project.freeze')
+        continue-on-error: true
+        run: |
+          cabal v2-freeze
+          echo ''
+          echo 'Output:'
+          echo ''
+          cat 'cabal.project.freeze'
+
       - name: Hackage sources cache
         uses: actions/cache@v2
         env:
@@ -133,12 +142,13 @@ jobs:
           restore-keys: ${{ env.cache-name }}-
 
       - name: Compiled deps cache
+        id: compiled-deps
         uses: actions/cache@v2
         env:
           cache-name: compiled-deps
         with:
           path: ${{ steps.HaskEnvSetup.outputs.cabal-store }}
-          key:  ${{ env.cache-name }}-${{ runner.os }}-${{ matrix.ghc }}-${{ env.INDEX_STATE }}-${{ hashFiles('cabal.project') }}
+          key:  ${{ env.cache-name }}-${{ runner.os }}-${{ matrix.ghc }}-${{ env.INDEX_STATE }}-${{ hashFiles('cabal.project.freeze') }}
           restore-keys: |
                 ${{ env.cache-name }}-${{ runner.os }}-${{ matrix.ghc }}-${{ env.INDEX_STATE }}-
                 ${{ env.cache-name }}-${{ runner.os }}-${{ matrix.ghc }}-