Cache - excluding files or folders with ! not working #713

dhadka · 2021-02-08T15:30:07Z

Describe the bug

Excluding a file or folder does not work as expected with caching. For example,

~/cache
!~/cache/subfolder

is matching:

../../../cache

I think this is because caching uses implicitDescendants: false. Since we're not traversing any descendants, the negation pattern does not match anything.

One option is to just set implicitDescendants: true, but this may add inefficiencies as we will need to enumerate all the files and generate a large "includes file" for tar. We essentially need it to scan the descendants but stop whenever the entire folder is included. For example, the following files:

~/cache/foo.txt
~/cache/bar.txt
~/cache/src/main.c
~/cache/src/lib/baz.dll
~/cache/bin/main.exe

with the glob patterns of ~/cache and !~/cache/bin should produce the list:

~/cache/foo.txt
~/cache/bar.txt
~/cache/src

To Reproduce

See https://github.com/dhadka/cache-test-exclusion

Expected behavior

Negated files / folders are excluded from the cache.

The text was updated successfully, but these errors were encountered:

mustaphazorgati · 2021-05-28T09:27:04Z

Any updates here?

lhotari · 2021-05-28T10:29:44Z

There is a workaround. Excluding works if you have all patterns in the same "level", for example

          path: |
            ~/.m2/repository/*/*/*
            !~/.m2/repository/org/apache/pulsar

this is what we use in apache/pulsar, full example here:

https://github.com/apache/pulsar/blob/master/.github/workflows/ci-integration-process.yaml#L55-L65

swarajpeppermint · 2021-07-01T07:44:05Z

There is a workaround. Excluding works if you have all patterns in the same "level", for example
          path: |
            ~/.m2/repository/*/*/*
            !~/.m2/repository/org/apache/pulsar
this is what we use in apache/pulsar, full example here:

https://github.com/apache/pulsar/blob/master/.github/workflows/ci-integration-process.yaml#L55-L65

Was trying to cache apt, using

path: |
    /var/cache/apt/archives/*
    !/var/cache/apt/archives/lock
    !/var/cache/apt/archives/partial

But it still gives me a permission error for trying to access the partial folder in the Post Cache Dependencies Step:
EACCES: permission denied, scandir '/var/cache/apt/archives/partial'

actions/toolkit#713

motss · 2022-02-05T15:19:08Z

Thanks for the workaround in #713 (comment). It works like a charm.

Here are my findings:

# 1. This will cache everything recursively inside `/a` excluding `/a/b`
path: |
    /a/*
    !/a/b

# 2. This will cache everything recursively inside `/a` including `/a/b`
path: |
    /a
    !/a/b

# 3. This will cache everything recursively inside `/a` including `/a/b`
#    with an unexpectedly huge total cache size (~6x, your mileage might vary)
#    and I don't see any visible differences when compared to 1. and 2.
path: |
    /a/**
    !/a/b

# Conclusion
# Just use 1. for the best of both worlds until the glob issue is fixed.
# Worth mentioning that modifying the value of `path` will affect the caches with the exact same `key`,
# so each cache is unique by its own `key` and `path`.

robinp · 2022-02-05T20:22:24Z

Hi guys. Was hitting this as well. Some digging in the code about the likely reason:

toolkit/packages/cache/src/internal/cacheUtils.ts

Line 42 in 15e2399

export async function resolvePaths(patterns: string[]): Promise<string[]> {

is actions/cache generating the list of paths to pass to tar for inclusion.
notice that in the options

toolkit/packages/glob/src/internal-glob-options.ts

Line 28 in 15e2399

* @default true

machDirectories defaults to true.
this is the globGenerator that generates the matches
if a directory was matched, it will be output for. So tar will include it (regardless that later some subdirectories are would be excluded... the parent dir is already passed to tar).

(Note: can observe this by enabling step debug logging, see https://docs.github.com/en/actions/monitoring-and-troubleshooting-workflows/enabling-debug-logging)

Other observation: the negative patterns must come after the positive one, otherwise a last positive match will always win (see

toolkit/packages/glob/src/internal-pattern-helper.ts

Line 65 in 15e2399

for (const pattern of patterns) {

).

Fix suggestion: pass matchDirectories as false in

toolkit/packages/cache/src/internal/cacheUtils.ts

Line 46 in 15e2399

implicitDescendants: false

- worst downside is that empty directories won't be tarred up, which sounds fine for caching use-case.

In the mean time, the workaround of expanding the paths with /* to the level of the excludes is doable as others greatly suggested. But is inconvenient and has some hard to cover edge cases.

actions/toolkit#713

Trying to exclude pdfs actions/toolkit#713

actions/toolkit#713 (comment)

kamzil · 2024-10-03T10:07:45Z

Just wasted way too much time trying to figure this out. GitHub should really pay more attention to the inconsistencies of glob patterns between different parts of Actions.

jonkoops · 2024-10-16T14:27:00Z

Note that none of the fixes here work if the levels are deeper and there are multiple paths that need to be matched, for example:

# Works
path: |
    /a/*
    !/a/b

# Doesn't work
path: |
    /a/*
    !/a/b/c
    
# Works
path: |
    /a/*/*
    !/a/b/c
    
path: |
    /a/*/*
    !/a/b/c
    !/a/b/c/d # No longer works

…ional test logs as artifacts 3461c14 ci: tentatively drop multiprocess and tsan functional tests (Kittywhiskers Van Gogh) 5db8fa0 ci: cache previous releases if running `linux64` variant (Kittywhiskers Van Gogh) cca0d89 ci: add functional tests for linux builds (Kittywhiskers Van Gogh) fc2efb6 ci: upload functional test logs as artifacts (Kittywhiskers Van Gogh) b25e846 ci: handle space exhaustion by deleting files we don't need (Kittywhiskers Van Gogh) 0a1e635 ci: add reusable workflow for running functional tests (Kittywhiskers Van Gogh) 57cf278 ci: use helper script to bundle artifacts (Kittywhiskers Van Gogh) Pull request description: ## Additional Information * [`actions/cache`](https://github.com/marketplace/actions/cache) allows specification of directories that want to be cached using glob expressions, which extend further into defining exclusions, needed due keep cache sizes maintainable ([source](https://github.com/dashpay/dash/blob/bb469687d3d936f82fd8e8fbe0934eec5e17df5e/.gitlab-ci.yml#L128-L138)). * Unfortunately, the implementation of globbing with respect to exclusions is more-or-less broken (see [actions/toolkit#713](actions/toolkit#713 (comment))) with the requirement that the inclusion depth should match the depth of the exclusion. Attempting to play by these rules more or less fails ([build](https://github.com/kwvg/dash/actions/runs/13344612118/job/37273624710#step:5:4634)). * Attempting to use third party actions like [`tj-actions/glob`](https://github.com/marketplace/actions/glob-match) provide for a much more saner experience but they enumerate individual files that match patterns, not folders. This means that when we pass them to `actions/cache`, we breach the arguments length limit ([build](https://github.com/kwvg/dash/actions/runs/13343953711/job/37272121153#step:9:4409)). * Modifying `ulimit` to get around this isn't very feasible due to odd behavior surrounding it (see [actions/runner#3421](actions/runner#3421)) and the general consensus is to save to a file and have the next action read from file ([source](https://stackoverflow.com/a/71349472)). [`tj-actions/glob`](https://github.com/marketplace/actions/glob-match) graciously does this with the `paths-output-file` output but it takes two to play and [`actions/cache`](https://github.com/marketplace/actions/cache) does not accept files (`path` must be a newline-delimited string). The path of least resistance, it seems, is to use a script to bundle our cache into a neat input and leave [`actions/cache`](https://github.com/marketplace/actions/cache) out of it entirely, this is the approach taken. * As we aren't using self-hosted runners, we are subject to GitHub's limits for everything, runner space, total artifact storage budget, total cache storage budget. * Caches that not **accessed** in 7 days are evicted and there's a 10 GB budget for all caches ([source](https://docs.github.com/en/actions/writing-workflows/choosing-what-your-workflow-does/caching-dependencies-to-speed-up-workflows#usage-limits-and-eviction-policy)) and GitHub will evict oldest (presumably by **creation**?) caches to make sure that limit is adhered to. * What makes this limit troubling is the immutable nature of caches as unlike GitLab, which is more conductive to shared caches ([source](https://github.com/dashpay/dash/blob/bb469687d3d936f82fd8e8fbe0934eec5e17df5e/.gitlab-ci.yml#L55-L69)), GitHub insists on its immutability (see [actions/toolkit#505](actions/toolkit#505)) and the only way to "update" a cache is to structure your cache key to allow the updated content to reflect in the key itself or delete the cache and create a new one, which brings race condition concerns ([comment](#6406 (review))). Sidenote, overwriting contents are allowed for artifacts ([source](https://github.com/actions/upload-artifact/blob/65c4c4a1ddee5b72f698fdd19549f0f0fb45cf08/README.md#overwriting-an-artifact)), just not caches. * This means we need to be proactive in getting rid of caches with a short shelf life to avoid more long lasting caches (like `depends-sources`) from being evicted due to old age as we breach the 10 GB limit. We cannot just set a short retention period as GitHub doesn't offer you to do that with caches like they do with artifacts ([source](https://github.com/actions/upload-artifact/blob/65c4c4a1ddee5b72f698fdd19549f0f0fb45cf08/README.md#retention-period)). * ~~While doing this properly would require us to implement a cache reaper workflow, this still needed to be addressed now as the contents of `build-ci` need to be passed onto the functional test workflow and this creates an ephemeral cache with a short lifespan that threatens longer-living (but older) caches.~~ ~~This is currently approached by deleting `build-ci` (output) caches when the functional test runner is successful, we let the cache stick around if the build fails to allow for rerunning failed instances.~~ ~~If for whatever reason a successful build has to be rerun, the build workflow would need to be rerun (though the `ccache` cache will speed this up significantly) to generate the output cache again for the test workflow to succeed. Failing to do this will result in a cache miss and run failure.~~ **Edit:** Switched to using artifacts to mitigate cache thrashing concerns. Auto-expiration is a huge plus, too. * Runners are limited to 14 GB of **addressable** storage space ([source](https://docs.github.com/en/actions/using-github-hosted-runners/using-github-hosted-runners/about-github-hosted-runners#standard-github-hosted-runners-for-public-repositories)) and breaching this limit will cause the runners to fail. * Our TSan ([build](https://github.com/kwvg/dash/actions/runs/13355816205/job/37298587344#step:5:1178)) and multiprocess ([build](https://github.com/kwvg/dash/actions/runs/13355816205/job/37298658464#step:5:1190)) test variants breach this limit when collecting logs and deleting the `build-ci` (see 2153b0b) cache doesn't make enough of a dent to help ([build](https://github.com/kwvg/dash/actions/runs/13356474530)). * Omitting the logs from successful runs would be a regression in content for our `test_logs` artifacts and therefore wasn't considered. * While third-party actions like [`AdityaGarg8/remove-unwanted-software`](https://github.com/marketplace/actions/maximize-build-disk-space-only-remove-unwanted-software) can bring significant space savings ([build](https://github.com/kwvg/dash/actions/runs/13357806504/job/37302970610#step:2:150)), they cannot be run in jobs that utilize the [`container`](https://docs.github.com/en/actions/writing-workflows/choosing-where-your-workflow-runs/running-jobs-in-a-container) context (so, all jobs after the container creation workflow) as they'd be executed inside the container when we want to affect the runner underneath ([build](https://github.com/kwvg/dash/actions/runs/13357260369/job/37301757225#step:3:29), notice the step being run after "Initialize containers"). * There are no plans to implement definable "pre-steps" (see [actions/runner#812](actions/runner#812)) and the only way to implement "before" and "after" steps is by self-hosting ([source](https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners/running-scripts-before-or-after-a-job)). This has been sidestepped tentatively by omitting TSan and multiprocess builds as any attempted fixes would require precision gutting to get borderline space savings which could easily be nullified by future code changes that occupy more space and such measures are better reserved for future PRs. * Artifacts share their storage quota with GitHub Packages ([source](https://docs.github.com/en/billing/managing-billing-for-your-products/managing-billing-for-github-packages/about-billing-for-github-packages#about-billing-for-github-packages)) and artifacts by default linger around for 90 days ([source](https://docs.github.com/en/actions/writing-workflows/choosing-what-your-workflow-does/storing-and-sharing-data-from-a-workflow#about-workflow-artifacts)) and considering that each run can generate multi-gigabyte total artifacts, testing the upper limit runs the risk of being very expensive. **Edit:** It appears pricing is reflective of artifacts in private repos, public repos don't seem to run this risk. * ~~All artifacts generated have an expiry of one day (compared to GitLab's three days, [source](https://github.com/dashpay/dash/blob/bb469687d3d936f82fd8e8fbe0934eec5e17df5e/.gitlab-ci.yml#L165), but they are self-hosted).~~ **Edit:** Artifacts now have an expiry of three days, matching GitLab. * Artifacts are compressed as ZIP archives and there is no way around that as of now (see [actions/upload-artifact#109](actions/upload-artifact#109 (comment))) and the permissions loss it entails is acknowledged by GitHub and their solution is... to put them in a tarball ([source](https://github.com/actions/upload-artifact/blob/65c4c4a1ddee5b72f698fdd19549f0f0fb45cf08/README.md#permission-loss)). To keep size under control, artifacts contain `zstd-5` compressed tarballs (compression level determined by benchmarks, see below) generated by bundling scripts alongside their checksum. <details> <summary>Benchmarks:</summary> ``` $ zstd -b1 -e22 -T0 artifacts-linux64_multiprocess.tar 1#_multiprocess.tar :1586411520 -> 537552492 (x2.951), 2396.2 MB/s, 1367.8 MB/s 2#_multiprocess.tar :1586411520 -> 499098623 (x3.179), 2131.8 MB/s, 1306.6 MB/s 3#_multiprocess.tar :1586411520 -> 474452284 (x3.344), 1371.6 MB/s, 1245.6 MB/s 4#_multiprocess.tar :1586411520 -> 470931621 (x3.369), 620.3 MB/s, 1239.1 MB/s 5#_multiprocess.tar :1586411520 -> 459075785 (x3.456), 457.2 MB/s, 1230.1 MB/s 6#_multiprocess.tar :1586411520 -> 449594612 (x3.529), 415.3 MB/s, 1289.7 MB/s 7#_multiprocess.tar :1586411520 -> 446208421 (x3.555), 282.6 MB/s, 1296.3 MB/s 8#_multiprocess.tar :1586411520 -> 442797797 (x3.583), 254.3 MB/s, 1338.4 MB/s 9#_multiprocess.tar :1586411520 -> 438690318 (x3.616), 210.8 MB/s, 1331.5 MB/s 10#_multiprocess.tar :1586411520 -> 437195147 (x3.629), 164.1 MB/s, 1337.4 MB/s 11#_multiprocess.tar :1586411520 -> 436501141 (x3.634), 108.2 MB/s, 1342.5 MB/s 12#_multiprocess.tar :1586411520 -> 436405679 (x3.635), 102.7 MB/s, 1344.0 MB/s 13#_multiprocess.tar :1586411520 -> 436340981 (x3.636), 65.9 MB/s, 1344.0 MB/s 14#_multiprocess.tar :1586411520 -> 435626720 (x3.642), 61.5 MB/s, 1346.9 MB/s 15#_multiprocess.tar :1586411520 -> 434882716 (x3.648), 49.4 MB/s, 1352.9 MB/s 16#_multiprocess.tar :1586411520 -> 411221852 (x3.858), 33.6 MB/s, 1049.2 MB/s 17#_multiprocess.tar :1586411520 -> 399523001 (x3.971), 26.0 MB/s, 1003.7 MB/s 18#_multiprocess.tar :1586411520 -> 379278765 (x4.183), 21.0 MB/s, 897.5 MB/s 19#_multiprocess.tar :1586411520 -> 378022246 (x4.197), 14.7 MB/s, 896.0 MB/s 20#_multiprocess.tar :1586411520 -> 375741653 (x4.222), 14.0 MB/s, 877.6 MB/s 21#_multiprocess.tar :1586411520 -> 373303486 (x4.250), 11.9 MB/s, 866.8 MB/s 22#_multiprocess.tar :1586411520 -> 358172556 (x4.429), 6.09 MB/s, 884.9 MB/s ``` </details> > **Note:** As mentioned above, we use similar bundling scripts for the outputs cache but unlike artifacts, we cannot disable their compression routines or even adjust compression levels (see [actions/tookit#544](actions/toolkit#544)) ## Notes * ~~If we add or remove binaries in terms of compile output, `bundle-build.sh` needs to be updated. Without updating it, it will fail if it cannot find the files it was looking for and it will not include files that it wasn't told to include.~~ No longer applicable. ## Breaking Changes None expected. ## Checklist - [x] I have performed a self-review of my own code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have added or updated relevant unit/integration/functional/e2e tests **(note: N/A)** - [x] I have made corresponding changes to the documentation **(note: N/A)** - [x] I have assigned this pull request to a milestone _(for repository code-owners and collaborators only)_ ACKs for top commit: PastaPastaPasta: utACK 3461c14 UdjinM6: ACK 3461c14 Tree-SHA512: ef55bc10902c57673ffd9bee6562b362a87658e4c51e543b8553bf48c41544a302d6acad7c5a30395fbfcfd085354251a07327c3e78c93c750585496926be9f6

dhadka added the bug label Feb 8, 2021

dhadka mentioned this issue Feb 8, 2021

Excluded sub directory not working actions/cache#494

Closed

This was referenced Mar 18, 2021

Add usethis as Config/Needs/website dependency r-lib/pkgdown#1572

Merged

pak should not be cached r-lib/actions#264

Closed

thboop added the cache label Apr 30, 2021

Seo-4d696b75 mentioned this issue Nov 1, 2021

修正：cache path Seo-4d696b75/android-paging#35

Merged

This was referenced Dec 4, 2021

任意のパスをビルドキャッシュから除外する custom-cache-path を追加する HoshinoResort/hr-library-auto-update#3

Merged

任意のパスをビルドキャッシュの対象外とする HoshinoResort/hr-library-auto-update#2

Closed

robinp mentioned this issue Feb 5, 2022

Glob patterns should work for excludes actions/cache#364

Closed

knapsu added a commit to knapsu/plex-media-player-flatpak that referenced this issue Feb 11, 2022

Workaround for cache exclude pattern bug

33af8f8

actions/toolkit#713

EagleoutIce added a commit to EagleoutIce/uulm-eidi-tut-ss2022-slides that referenced this issue Aug 12, 2022

Update compile.yaml

ac4f8e8

Trying to exclude pdfs actions/toolkit#713

xanderdunn mentioned this issue Aug 25, 2022

Unable to Cache Rust Build Output Due to Unreadable .lock Files actions/cache#909

Closed

lukka mentioned this issue Dec 12, 2022

Ensure the cache content is minimum required lukka/run-vcpkg#174

Closed

YouJiacheng mentioned this issue Feb 21, 2023

Attempt to reuse build in Action YouJiacheng/IGinX#7

Open

NxPKG mentioned this issue Feb 29, 2024

[Snyk] Security upgrade archiver from 5.3.1 to 7.0.0 threatcode/toolkit#17

Open

DanyRay420 mentioned this issue Mar 1, 2024

[Snyk] Security upgrade archiver from 5.3.1 to 7.0.0 DanyRay420/toolkit#6

Open

This was referenced Mar 5, 2024

HDDS-10468. CI: Make unit depend on build apache/ozone#6333

Closed

HDDS-10470. Populate Maven dependency cache in separate workflow apache/ozone#6340

Merged

miiihi mentioned this issue Mar 22, 2024

Exclusions do not work actions/cache#1356

Open

dockerzhang mentioned this issue Apr 23, 2024

[INLONG-10048][Tools] Remove more useless paths to release more size for workflows apache/inlong#10049

Merged

austinletson mentioned this issue May 4, 2024

Redundant mathlib caching leanprover/lean-action#3

Open

ChanTsune mentioned this issue Aug 11, 2024

cache ignore path not working ChanTsune/Portable-Network-Archive#966

Closed

ChanTsune added a commit to ChanTsune/Portable-Network-Archive that referenced this issue Aug 11, 2024

👷 fix cache ignore setting

Loading
Loading status checks…

3aa05fc

actions/toolkit#713 (comment)

ChanTsune mentioned this issue Aug 11, 2024

👷 fix cache ignore setting ChanTsune/Portable-Network-Archive#967

Merged

ChanTsune added a commit to ChanTsune/Portable-Network-Archive that referenced this issue Aug 11, 2024

👷 fix cache ignore setting

Loading
Loading status checks…

4321452

actions/toolkit#713 (comment)

jonkoops linked a pull request Oct 16, 2024 that will close this issue

Allow excluding files and folders when caching #1854

Open

jonkoops mentioned this issue Oct 16, 2024

Cache Node.js artifacts from maven-frontend-plugin on CI keycloak/keycloak#33815

Merged

Coduz mentioned this issue Jan 24, 2025

♻️ [CI] Improved Kapua CI workflow artifacts and dependency caching eclipse-kapua/kapua#4179

Merged

kwvg mentioned this issue Feb 17, 2025

ci: run functional tests on GitHub Actions, upload functional test logs as artifacts dashpay/dash#6583

Merged

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cache - excluding files or folders with ! not working #713

Cache - excluding files or folders with ! not working #713

dhadka commented Feb 8, 2021

mustaphazorgati commented May 28, 2021

lhotari commented May 28, 2021

swarajpeppermint commented Jul 1, 2021 •

edited

Loading

motss commented Feb 5, 2022

robinp commented Feb 5, 2022

kamzil commented Oct 3, 2024

jonkoops commented Oct 16, 2024

Cache - excluding files or folders with ! not working #713

Cache - excluding files or folders with ! not working #713

Comments

dhadka commented Feb 8, 2021

mustaphazorgati commented May 28, 2021

lhotari commented May 28, 2021

swarajpeppermint commented Jul 1, 2021 • edited Loading

motss commented Feb 5, 2022

robinp commented Feb 5, 2022

kamzil commented Oct 3, 2024

jonkoops commented Oct 16, 2024

swarajpeppermint commented Jul 1, 2021 •

edited

Loading