fix: break corrupt mirror snapshot cycle#190
Merged
Conversation
worstell
changed the title
jrobotham-square
approved these changes
Mar 13, 2026
When a corrupt or empty mirror snapshot exists in S3, pods restore it, the post-restore fetch fails, and then snapshot jobs immediately re-upload the corrupt mirror — perpetuating the cycle even after manual S3 cleanup. Three changes break this cycle: 1. FetchLenient: post-restore and startup fetches now omit the lowSpeedLimit check (same as executeClone), since large deltas after snapshot restore trigger server-side pack computation that stalls at near-zero transfer for minutes, tripping the 1KB/s threshold. 2. ResetToEmpty + fallback to clone: when the post-restore fetch fails, the corrupt mirror is removed and the repo state is reset to Empty so the code falls through to a fresh git clone --mirror instead of serving and re-uploading stale data. 3. Skip snapshot scheduling on failed fetch: snapshot and repack jobs are only scheduled after a successful fetch, both in the startup path and the post-restore path. Co-authored-by: Amp <amp@ampcode.com> Amp-Thread-ID: https://ampcode.com/threads/T-019ce5d1-9385-7026-8e69-78903ce99c47
worstell
force-pushed
the
fix-post-restore-fetch-corruption
branch
from
8255718 to
b113e07
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
When a corrupt or empty mirror snapshot exists in S3, pods enter a poison cycle:
git fetchfails —lowSpeedLimit(1KB/s for 60s) trips during server-side pack computation for the large deltaThis cycle survived the fixes in #188 and #189 because those PRs prevented the creation of corruption (concurrent restores and concurrent fetch-during-tar) but not the propagation of already-corrupt snapshots.
Fix
Three changes break the cycle:
FetchLenient: Post-restore and startup fetches omit thelowSpeedLimitcheck, matchingexecuteClone's behavior. Large deltas after snapshot restore trigger GitHub's server-side pack computation which stalls at near-zero transfer rate for minutes, tripping the 1KB/s threshold.ResetToEmpty+ fallback to clone: When the post-restore fetch fails, the corrupt mirror directory is removed and the repo state is reset to Empty. The code then falls through to a freshgit clone --mirrorinstead of serving and re-uploading stale data.Skip snapshot scheduling on failed fetch: Snapshot and repack jobs are only scheduled after a successful fetch, both in the startup path (
DiscoverExisting) and the post-restore path (startClone).Testing
All existing tests pass. The fix was validated against staging logs showing the poison cycle in action.