Skip to content

Conversation

@mbg
Copy link
Member

@mbg mbg commented Nov 14, 2025

This PR modifies uploadDependencyCaches to avoid attempting to create and upload a dependency cache if a cache with the same key was previously restored, because that means we know that it already exists. Cache keys are intended to uniquely identify the contents of a cache and, based on that principle, it does not make sense to upload a cache with a key that already exists. Note that this is also not a change in behaviour here: ordinarily, the Cache API would refuse to accept a cache upload if a cache with the same key already exists. This change simply optimises uploadDependencyCaches so that we don't perform expensive tasks when we can avoid them before the API tells us that the cache already exists.

This is accomplished in the following way:

  • When restoring dependency caches in the init action, we now keep track of the cache keys that we restored.
  • These are then added to the CodeQL Action state and saved with it.
  • When creating and uploading caches after the analysis has completed, we calculate the cache key as early as possible and check if it was among the caches we already restored. If so, we skip more expensive operations like calculating the size of the files that would go into the cache.

This PR also adds general unit test coverage for uploadDependencyCaches which was missing so far.

We need to continue to check for ReserveCacheErrors when uploading caches, because there are two scenarios where we may still try to upload a cache that already exists:

  • A significant amount of time may pass between restoring caches and beginning the process of storing a cache. In the meantime, a different CodeQL workflow run may have stored a cache with the key that we are about to attempt to create ourselves.
  • We may try to create a cache with a key that we didn't try to restore, but that already existed.

Some other notes:

  • Generally speaking, for a given language, we would expect the primary cache key we compute when restoring caches in init to match the cache key we compute in uploadDependencyCaches, because our analysis should not modify any project files.
    • Nonetheless, we recompute the key in uploadDependencyCaches to ensure that it really corresponds to what we are about to put into the cache. This hasn't changed in this PR.
    • In future work, we might want to investigate whether this is really true for the extractors for all languages we support with dependency caching.
  • An alternative to keeping track of the restored cache keys would be to query the API for existing cache keys before trying to create a cache. That would provide more up-to-date information about which caches exist, but comes at the cost of an extra API call. I don't believe that to be worth it for an optimisation that should work in most cases as-is.

Risk assessment

For internal use only. Please select the risk level of this change:

  • Low risk: Changes are fully under feature flags, or have been fully tested and validated in pre-production environments and are highly observable, or are documentation or test only.

Which use cases does this change impact?

  • Advanced setup - Impacts users who have custom workflows.
  • Default setup - Impacts users who use default setup.
  • Code Scanning - Impacts Code Scanning (i.e. analysis-kinds: code-scanning).
  • Code Quality - Impacts Code Quality (i.e. analysis-kinds: code-quality).
  • GHES - Impacts GitHub Enterprise Server.

How did/will you validate this change?

  • Test repository - This change will be tested on a test repository before merging.
  • Unit tests - I am depending on unit test coverage (i.e. tests in .test.ts files).
  • End-to-end tests - I am depending on PR checks (i.e. tests in pr-checks).

If something goes wrong after this change is released, what are the mitigation and rollback strategies?

  • Rollback - Change can only be disabled by rolling back the release or releasing a new version with a fix.

How will you know if something goes wrong after this change is released?

  • Telemetry - I rely on existing telemetry or have made changes to the telemetry.
    • Dashboards - I will watch relevant dashboards for issues after the release. Consider whether this requires this change to be released at a particular time rather than as part of a regular release.
    • Alerts - New or existing monitors will trip if something goes wrong with this change.

Merge / deployment checklist

  • Confirm this change is backwards compatible with existing workflows.
  • Consider adding a changelog entry for this change.
  • Confirm the readme and docs have been updated if necessary.

Copilot AI review requested due to automatic review settings November 14, 2025 14:22
@mbg mbg requested a review from a team as a code owner November 14, 2025 14:22
@github-actions github-actions bot added the size/L May be hard to review label Nov 14, 2025
Copilot finished reviewing on behalf of mbg November 14, 2025 14:25
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR optimizes the dependency caching mechanism by skipping cache uploads when the cache key was previously restored, avoiding unnecessary expensive operations like calculating cache sizes and API calls.

Key changes:

  • Tracks restored cache keys during cache restoration and stores them in the config
  • Checks if a cache with the same key was restored before attempting to upload, skipping the upload if it was
  • Adds comprehensive unit test coverage for uploadDependencyCaches which was previously untested

Reviewed Changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
src/config-utils.ts Adds dependencyCachingRestoredKeys field to the Config interface to track restored cache keys
src/testing-utils.ts Updates createTestConfig helper to include the new dependencyCachingRestoredKeys field
src/init-action.ts Captures restored cache keys from downloadDependencyCaches and stores them in config
src/dependency-caching.ts Adds DownloadDependencyCachesResult interface; modifies downloadDependencyCaches to return restored keys; optimizes uploadDependencyCaches to skip upload for known cache keys
src/dependency-caching.test.ts Adds comprehensive unit tests for uploadDependencyCaches covering various scenarios (no config, no hash, duplicate, empty, stored, error handling)
src/config-utils.test.ts Refactors tests to use createTestConfig helper instead of manually constructing config objects
lib/init-action.js Auto-generated JavaScript from TypeScript source
lib/analyze-action.js Auto-generated JavaScript from TypeScript source

@mbg mbg force-pushed the mbg/dependency-caching/skip-uploads-for-exact-matches branch from 755b23c to b665de6 Compare November 14, 2025 14:29
@mbg mbg force-pushed the mbg/dependency-caching/skip-uploads-for-exact-matches branch from b665de6 to 1ed85b4 Compare November 14, 2025 14:31
@github github deleted a comment from CosmicJesterX Nov 14, 2025
Copy link
Contributor

@redsun82 redsun82 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@mbg mbg merged commit 249458a into main Nov 18, 2025
241 checks passed
@mbg mbg deleted the mbg/dependency-caching/skip-uploads-for-exact-matches branch November 18, 2025 15:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/L May be hard to review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants