Recover from lost CAS blobs during full repo materialization - #30226
Draft
fmeum wants to merge 11 commits into
Draft
Recover from lost CAS blobs during full repo materialization#30226fmeum wants to merge 11 commits into
fmeum wants to merge 11 commits into
Conversation
fmeum
force-pushed
the
fix-30218-materialize-missing-blob
branch
from
July 10, 2026 10:24
dc6ed3b to
2ad6d46
Compare
dzbarsky
pushed a commit
to dzbarsky/bazel
that referenced
this pull request
Jul 23, 2026
A remote repository-contents entry can outlive one of its CAS blobs. Full repository materialization then hides the retryable filesystem failure inside package lookup or module-extension evaluation. Preserve the missing-file failure, evict the incomplete in-memory repository, and bypass its stale cache entry until a successful refetch repairs the remote cache. Propagate the original cause and transient failure through repository and Bzlmod evaluation. Cover a missing .bazelrc blob, one automatic retry, and a subsequent cold cache hit. Preserve registry lockfiles in secondary test workspaces so the complete cache suite remains hermetic. Builds on the full-materialization recovery proposed in bazelbuild#30226. Fixes bazelbuild#30218. (cherry picked from commit 271a930)
dzbarsky
added a commit
to dzbarsky/bazel
that referenced
this pull request
Jul 24, 2026
A remote repository-contents entry can outlive one of its CAS blobs. Full repository materialization then hides the retryable filesystem failure inside package lookup or module-extension evaluation. Preserve the missing-file failure, evict the incomplete in-memory repository, and bypass its stale cache entry until a successful refetch repairs the remote cache. Propagate the original cause and transient failure through repository and Bzlmod evaluation. Cover a missing .bazelrc blob, one automatic retry, and a subsequent cold cache hit. Preserve registry lockfiles in secondary test workspaces so the complete cache suite remains hermetic. Builds on the full-materialization recovery proposed in bazelbuild#30226. Fixes bazelbuild#30218. (cherry picked from commit 271a930)
The remote repo contents cache injects a cached repo's metadata without verifying that every leaf blob referenced by its Tree is still available in the CAS. If a blob had been evicted by the time another repo rule triggered full materialization through rctx.path() or rctx.read(), the build failed with "Failed to materialize remote repo ...: Missing digest ..." and never reran the repo rule, not even with the default whole-command remote cache eviction retries. Full materialization now handles an all-CacheNotFoundException prefetch failure like the existing on-demand read path: the repo is marked as having lost files and a transient REMOTE_FILE_EVICTED error is raised. The failure detail is preserved through repo rule evaluation (as the EvalException cause) and PackageLookupFunction so that the command exits with REMOTE_CACHE_EVICTED and is automatically retried. Since the stale ActionResult and Tree survive in the remote cache, retried commands skip the cache lookup for such repos until their contents, including the lost blobs, have been fetched and uploaded again, which heals the cache entry. Fixes bazelbuild#30218. RELNOTES: The remote repo contents cache now recovers from cached repository contents that are no longer fully available in the remote cache by refetching the repository.
…nally Repos whose cached contents referenced lost remote files must be refetched and thus have no use for their stale in-memory contents, even if no materialization was attempted for them. This also simplifies the eviction filter for materialized repos.
fmeum
force-pushed
the
fix-30218-materialize-missing-blob
branch
from
July 24, 2026 09:52
2ad6d46 to
1634616
Compare
A module extension that triggers the full materialization of a cached repo via module_ctx.path() previously converted the transient REMOTE_FILE_EVICTED failure into a persistent EXTENSION_EVAL_ERROR, so the build failed without a retry. ExternalDepsException now derives its exit code and transience from a DetailedIOException cause so that the failure detail is preserved through module extension evaluation and the build is retried, refetching the repo.
fmeum
force-pushed
the
fix-30218-materialize-missing-blob
branch
from
July 24, 2026 09:52
1634616 to
d772794
Compare
fmeum
force-pushed
the
fix-30218-materialize-missing-blob
branch
from
July 24, 2026 13:46
c467dae to
e7d6af3
Compare
testLostRemoteFile_sourceDirectoryMaterialization covers a cached repo whose only lost CAS blob lies below a source directory input consumed by a local action: the materialization of just that subtree discovers the loss and must trigger the transparent refetch that heals the cache entry.
fmeum
marked this pull request as ready for review
July 24, 2026 14:08
fmeum
requested review from
tjgq
and removed request for
a team and
meteorcloudy
July 24, 2026 14:08
github-actions
Bot
added
team-ExternalDeps
Collaborator
Author
|
@bazel-io fork 9.3.0 |
Collaborator
Author
|
David noticed that there is a subtle problem with build rewinding and |
fmeum
marked this pull request as draft
July 24, 2026 17:35
A repo that is only reachable through a dependency edge is fetched during the analysis of the target that depends on it. If such a fetch fails because a file is no longer available in the remote cache, the resulting REMOTE_CACHE_EVICTED exit code was replaced with a generic one and the build was not retried, in both of the ways such an error can be reported: With --keep_going, the error is aggregated into an ErrorProcessingResult and BuildView#createAnalysisFailureDetail turns it into a generic loading or analysis phase failure. The detailed exit code of an analysis root cause that can be resolved by a retry is now preserved and takes precedence over these generic failure details. With --nokeep_going, the error is thrown as a ViewCreationFailedException whose failure detail does describe the lost file, but BuildTool unconditionally paired it with the PARSING_FAILURE exit code. The exit code implied by the failure detail is now kept if it allows a retry.
fmeum
force-pushed
the
fix-30218-materialize-missing-blob
branch
from
July 24, 2026 18:07
bdbb041 to
426328a
Compare
dzbarsky
pushed a commit
to dzbarsky/bazel
that referenced
this pull request
Jul 25, 2026
A remote repository-contents entry can outlive one of its CAS blobs. Full repository materialization then hides the retryable filesystem failure inside package lookup or module-extension evaluation. Preserve the missing-file failure, evict the incomplete in-memory repository, and bypass its stale cache entry until a successful refetch repairs the remote cache. Propagate the original cause and transient failure through repository and Bzlmod evaluation. Cover a missing .bazelrc blob, one automatic retry, and a subsequent cold cache hit. Preserve registry lockfiles in secondary test workspaces so the complete cache suite remains hermetic. Builds on the full-materialization recovery proposed in bazelbuild#30226. Fixes bazelbuild#30218. (cherry picked from commit 271a930)
The remote worker used by the tests serves an action result only after checking that all blobs it references are still present in the CAS. Real remote caches commonly don't do this, which is precisely what allows a repo's cached Tree to outlive the contents of its files and is the situation the remote repo contents cache has to recover from. Run the tests against a worker with --noaction_cache_integrity_check so that a repo whose blobs have been deleted is served from the cache rather than silently turning into a cache miss.
A repo that is only reachable through an implicit attribute of a top-level aspect is fetched during the analysis of that aspect. If the fetch fails because a file is no longer available in the remote cache, the resulting REMOTE_CACHE_EVICTED exit code was replaced with a generic one with --keep_going and the build was not retried. Unlike a ConfiguredValueCreationException, whose root causes carry the detailed exit code all the way into the ErrorProcessingResult, the root causes of an AspectCreationException were dropped and the aspect's IndividualErrorProcessingResult was always reported with an empty set of analysis root causes. BuildView#createAnalysisFailureDetail then fell back to NOT_ALL_TARGETS_ANALYZED. Keep the causes of an AspectCreationException, just like those of a ConfiguredValueCreationException, which both restores the retry and populates the AnalysisFailureEvent posted for a top-level aspect.
A file lost from the remote cache is never specific to a single repo, but it is only noticed when that particular file is read. Repos with lost files were thus discovered one at a time, each costing one of the bounded number of build retries: a module extension that materializes one cached repo after another aborts at the first failure and only gets to the next repo once the previous one has been refetched. With enough affected repos, --experimental_remote_cache_eviction_retries is exhausted before the build can recover. Once a lost file has been observed, discard all repos whose contents were restored from the cache, not just those in which a lost file was found, and have the next command verify a repo's cached contents against the remote cache before restoring them. A cached Tree that references a blob the cache no longer has is treated as a miss, so the repo is fetched and its contents are uploaded anew right away. All affected repos therefore recover in the same retry. Builds that don't encounter a lost file are unaffected: no additional FindMissingBlobs requests are sent and no repos are discarded.
--nobuild turns off the merged analysis and execution phase, and so do cquery and aquery, which imply it. Analysis errors are then reported through AnalysisPhaseRunner and SkyframeErrorProcessor#processErrors without the execution phase rather than through AnalysisAndExecutionPhaseRunner. Run the tests for the retry after a lost remote file discovered during analysis with --nobuild as well. The aspect variant with --keep_going fails without the preceding commit.
With --keep_going and a second, healthy target, the analysis error of a repo that could not be fetched is reported together with the results of actions that did execute. Release branches that still have --experimental_skyframe_error_handling_refactor handle that case in a separate code path that derives the exit code without looking at analysis root causes, so the lost remote file has to survive it for the build to be retried.
2 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
If a CAS blob that represents a file in a remotely cached repo has been evicted by the time another repo rule or module extension triggers full materialization
repository_ctxormodule_ctx, the build fails withFailed to materialize remote repo ...: Missing digest ...without rerunning the repo rule, not even with the default whole-command remote cache eviction retries.Full materialization now handles a
CacheNotFoundExceptionprefetch failure like the existing on-demand read path: the repo is marked as having lost files and a transientREMOTE_FILE_EVICTEDerror is raised. The failure details are now preserved through the stack of repo rule, module extension and package evaluation. Just like BwoB, retried commands skip the cache lookup for such repos until their contents, including the lost blobs, have been fetched and uploaded again, which prevents trusting the same stale AC hit again.In addition,
afterCommandnow evicts the in-memory contents of repos with lost files unconditionally instead of only when a materialization had been attempted for them: such repos have to be refetched anyway, and this also simplifies the eviction filter for materialized repos.The module extension handling is adapted from #30451 by @dzbarsky.
Motivation
Fixes #30218.
Build API Changes
No
Checklist
Release Notes
RELNOTES: The remote repo contents cache now recovers from cached repository contents, including those referenced by module extensions, that are no longer fully available in the remote cache by refetching the repository.