fix(source): self-heal a corrupt mirror and sweep orphaned staging dirs#55
Merged
Conversation
A bare mirror dir that exists but won't `gix::open` — left by a `rename` interrupted mid-publish, a disk error, or external damage — wedged every later `sync` with "does not appear to be a git repository", recoverable only by manually deleting the cache. `fetch` now treats an unopenable mirror as absent: it removes the corrupt dir and re-clones via the existing staging-then-rename path. A clone killed before its `MirrorStaging` `Drop` runs (e.g. SIGKILL) leaves a `.<key>.staging-<pid>-<nonce>` dir behind forever. Under the per-mirror lock, `fetch` now sweeps staging dirs for the key that have been idle past a one-hour grace window — long enough that an in-flight lock-free shallow clone is never reaped.
A normal `phora sync` skips the fetch whenever the lock already pins a source, but the read paths (`compute_digest` -> `open_mirror`) still open the bare mirror. Clearing the regenerable cache under `XDG_CACHE_HOME` while the lock survives left every locked source opening a vanished mirror: `open mirror <src>: ... does not appear to be a git repository`. The `fetch` self-heal never ran because the lock hit bypassed fetch. Gate the lock-hit fetch-skip on a new `SourceBackend::mirror_ready`: a missing or corrupt mirror re-fetches even under a warm lock. Reads stay offline by design; the fix lives in sync orchestration. The frozen path is untouched -- `--frozen` refusing to fetch a wiped cache is correct.
There was a problem hiding this comment.
Pull request overview
This PR improves GitBackend::fetch cache robustness by (1) self-healing when the canonical bare mirror directory is present but corrupt/unopenable, and (2) sweeping stale orphaned staging directories left behind by killed clone processes. It also updates the sync pipeline to only skip fetching on lock hits when a usable mirror is actually present.
Changes:
- Add mirror self-heal logic: open-or-reclone flow under the per-mirror lock, with re-clone via staging + rename.
- Sweep stale orphan staging dirs for a mirror key (with a grace window) before fetching.
- Introduce
SourceBackend::mirror_readyand use it in sync to avoid “lock-hit but cache-cleared” failures; update tests/docs accordingly.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| tests/scrut/lifecycle.md | Adds an integration doc case ensuring phora sync recovers after the git mirror cache is cleared. |
| src/sync/tests.rs | Updates counting/wrapper backends to forward the new mirror_ready method. |
| src/sync/resolve.rs | Changes lock-hit fetch skipping to also require backend.mirror_ready(remote) so cleared/corrupt caches re-fetch. |
| src/source.rs | Implements orphan staging sweep + mirror open/reclone behavior; adds mirror_ready implementations and targeted tests. |
| src/backend.rs | Updates router backend to expose mirror_ready based on the git-backed mirror. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
staging_is_recent swallowed WalkDir/metadata/modified errors and continued, so a partially-unreadable but possibly-live staging dir could be classified stale and reaped. Treat any traversal or mtime error as recent, honoring the sweep's never-reap-a-live-clone invariant.
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Cache-robustness fixes for
GitBackend, all under the per-mirror lock.a. Self-heal a corrupt canonical mirror. A bare mirror dir that exists but won't
gix::open— left by arenameinterrupted mid-publish, a disk error, or external interference (e.g. a racingrm -rfof the cache) — previously wedged every subsequentsyncwith libgit2'sdoes not appear to be a git repository, recoverable only by manually deleting the mirror.fetchnow treats an unopenable mirror as absent: removes the corrupt dir and re-clones through the existing staging-then-renamepath.b. Sweep orphaned staging dirs. A clone killed before
MirrorStaging'sDropruns (SIGKILL, OOM) leaves a.<key>.staging-<pid>-<nonce>dir behind indefinitely.fetchnow sweeps staging dirs for the key that have been idle past a one-hour grace window (liveness walks the whole staging tree, since a clone's pack writes leave the root mtime frozen early). Safety: the per-mirror lock excludes a concurrent full fetch of the key, and the grace window is far above any real clone, so an in-flight lock-free shallow clone is never reaped.c. Re-fetch a locked source when its mirror cache is gone. A normal
phora syncskips the fetch whenever the lock already pins a source, but the read paths (compute_digest→open_mirror) still open the bare mirror directly. Clearing the regenerable cache underXDG_CACHE_HOMEwhilephora.locksurvives left every locked source opening a vanished mirror —open mirror <src>: … does not appear to be a git repository. The self-heal from (a) never ran because the lock hit bypassedfetchentirely. A newSourceBackend::mirror_readynow gates the lock-hit fetch-skip: a missing or corrupt mirror re-fetches even under a warm lock. Reads stay offline by design, so the fix lives in sync orchestration, not the read path. The--frozenpath is untouched — refusing to fetch a wiped cache under--frozenis correct.Why
Discovered while wiring
phora syncinto a consumer repo: an interrupted clone on a slow network, a cache wipe racing a concurrent sync, or simplyrm -rf ~/.cache/phorabetween syncs left the mirror cache in a state where retries failed with a confusing error and no self-recovery.Tests
fetch_reclones_a_corrupt_canonical_mirror— corrupt the canonical mirror, assert the nextfetchrecovers to a valid bare repo and resolves.fetch_reclones_when_in_place_fetch_fails— point origin at a dead remote, assertfetchself-heals via re-clone.fetch_sweeps_a_stale_orphan_staging_dir— backdate an orphan past the grace window, assertfetchremoves it.fetch_keeps_a_staging_dir_with_recent_inner_writes— a staging dir whose root mtime is stale but whose pack write is recent is a live clone and must survive.mirror_ready_tracks_fetch_and_a_cleared_cache— not-ready → ready after fetch → not-ready after the cache is cleared.tests/scrut/lifecycle.md— end-to-end:rm -rf "\$XDG_CACHE_HOME/phora/git"thenphora syncself-heals tosync complete(reproduces the reported error verbatim on the pre-fix binary).Full suite: 1337 lib tests + 167 scrut cases pass, 0 failed; clippy
-D warningsandrustfmt --checkclean.