Higher fidelity GitHub Actions cache server fake#232
Merged
Conversation
aomarks
force-pushed
the
github-cache-testing
branch
from
bd2c0e5 to
9164f75
Compare
aomarks
force-pushed
the
github-cache-testing
branch
from
543eea6 to
c9b1706
Compare
rictic
approved these changes
May 15, 2022
aomarks
added a commit
that referenced
this pull request
May 15, 2022
Replaces [`@actions/cache`](https://github.com/actions/toolkit/tree/main/packages/cache) with our own implementation for interacting with the GitHub Actions cache service. ### API mismatch with @actions/cache `@actions/cache` takes a key and a set of glob patterns, and the literal glob patterns themselves are automatically used to generate a cache "version". The key + version then serve as a compound key for the cache entry. This didn't work well for us because the glob implementation it uses doesn't support exclusions the way we need, so it is incompatible with our glob behavior, and may also be different in other subtle ways. We really just want to provide a list of files/directories for the tarball, and set the key independently. But if we list all the files directly, then we can never get a cache hit, because the full list of files needs to be part of the cache key, but knowing the list of files requires running the script, which is what we're trying to avoid by restoring from cache! ### The workaround we had before The workaround we had before was to reach into the `internal/` directory of `@actions/cache` and call some of the lower level functions directly, which let us control exactly what the cache key + version was, while still passing an explicit list of files to `tar`. This worked ok, but had a high risk of breakage, because it uses non-public APIs. #227 is an example of us breaking because of that. It also required us to write some of our own typings, and to turn off `skipLibCheck` in our `tsconfig.json`. It was still quite a lot of code, too. In addition, we still had to deal with an edge case relating to empty directories. By default, `tar` includes all recursive contents of a directory -- so if we wanted to cache an empty directory that wasn't actually empty on disk, we had to resort to a weird "empty directories manifest" hack. But now we can just use `tar --no-recursion` flag which avoids that whole issue. ### This PR This PR completely replaces `@actions/cache` with a custom implementation that uses `https.request` and `execFile('tar', ...)`. As a big bonus, **our install size decreased from 25MB to 2.4MB** and our **transitive dependency count decreased from 93 to 29**. ### Testing - In #232 I improved our fake in a number of ways to make it higher fidelity, such as validating headers, and handling multiple upload chunks. - I read through the implementation details in https://github.com/actions/toolkit/tree/main/packages/cache to make sure we are compatible, picking the same chunk sizes, etc. - https://github.com/google/wireit/actions/runs/2325580687 is a run that tests this PR against the live servers by installing from an `npm pack`ed version. https://github.com/google/wireit/actions/runs/2325616423 is a follow up re-run, to show that everything got restored from cache. Fixes #107
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
I'm working on a clean implementation of our GitHub Caching client, which will be much simpler because we'll have full control over cache keys and tarball generation, and will drop our install size and deps significantly.
As part of this, I wanted to improve the fidelity of the GitHub Actions cache server fake that we use in testing, plus add an extra test.
201and204status codes in some cases32MBContent-Rangeheaders when tarball chunks are uploadedContent-Type,Accept,Transfer-Encoding, andUser-AgentheadersPart of #107