Replace GitHub cache library with our own implementation#234
Merged
Conversation
rictic
approved these changes
May 15, 2022
Member
rictic
left a comment
There was a problem hiding this comment.
Excellent comments, great work validating this
| if (emptyDirs.size > 0) { | ||
| const emptyDirsManifest = JSON.stringify([...emptyDirs]); | ||
| emptyDirsManifestPath = this.#emptyDirectoriesManifestPath( | ||
| async #reserveUploadAndCommitTarball( |
Member
There was a problem hiding this comment.
Document what the boolean return means
| } | ||
|
|
||
| const tarballPath = await this.#makeTarball([...files], compressionMethod); | ||
| async #upload( |
| script: ScriptReference, | ||
| id: number, | ||
| tarballBytes: number | ||
| ): Promise<boolean> { |
bcc7f25 to
21abafa
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Replaces
@actions/cachewith our own implementation for interacting with the GitHub Actions cache service.API mismatch with @actions/cache
@actions/cachetakes a key and a set of glob patterns, and the literal glob patterns themselves are automatically used to generate a cache "version". The key + version then serve as a compound key for the cache entry.This didn't work well for us because the glob implementation it uses doesn't support exclusions the way we need, so it is incompatible with our glob behavior, and may also be different in other subtle ways. We really just want to provide a list of files/directories for the tarball, and set the key independently.
But if we list all the files directly, then we can never get a cache hit, because the full list of files needs to be part of the cache key, but knowing the list of files requires running the script, which is what we're trying to avoid by restoring from cache!
The workaround we had before
The workaround we had before was to reach into the
internal/directory of@actions/cacheand call some of the lower level functions directly, which let us control exactly what the cache key + version was, while still passing an explicit list of files totar.This worked ok, but had a high risk of breakage, because it uses non-public APIs. #227 is an example of us breaking because of that. It also required us to write some of our own typings, and to turn off
skipLibCheckin ourtsconfig.json. It was still quite a lot of code, too.In addition, we still had to deal with an edge case relating to empty directories. By default,
tarincludes all recursive contents of a directory -- so if we wanted to cache an empty directory that wasn't actually empty on disk, we had to resort to a weird "empty directories manifest" hack. But now we can just usetar --no-recursionflag which avoids that whole issue.This PR
This PR completely replaces
@actions/cachewith a custom implementation that useshttps.requestandexecFile('tar', ...).As a big bonus, our install size decreased from 25MB to 2.4MB and our transitive dependency count decreased from 93 to 29.
Testing
In Higher fidelity GitHub Actions cache server fake #232 I improved our fake in a number of ways to make it higher fidelity, such as validating headers, and handling multiple upload chunks.
I read through the implementation details in https://github.com/actions/toolkit/tree/main/packages/cache to make sure we are compatible, picking the same chunk sizes, etc.
https://github.com/google/wireit/actions/runs/2325580687 is a run that tests this PR against the live servers by installing from an
npm packed version. https://github.com/google/wireit/actions/runs/2325616423 is a follow up re-run, to show that everything got restored from cache.Fixes #107