Skip to content

fix(source): self-heal a corrupt mirror and sweep orphaned staging dirs#55

Merged
srnnkls merged 3 commits into
mainfrom
fix/mirror-recovery
Jul 1, 2026
Merged

fix(source): self-heal a corrupt mirror and sweep orphaned staging dirs#55
srnnkls merged 3 commits into
mainfrom
fix/mirror-recovery

Conversation

@srnnkls

@srnnkls srnnkls commented Jun 30, 2026

Copy link
Copy Markdown
Owner

What

Cache-robustness fixes for GitBackend, all under the per-mirror lock.

a. Self-heal a corrupt canonical mirror. A bare mirror dir that exists but won't gix::open — left by a rename interrupted mid-publish, a disk error, or external interference (e.g. a racing rm -rf of the cache) — previously wedged every subsequent sync with libgit2's does not appear to be a git repository, recoverable only by manually deleting the mirror. fetch now treats an unopenable mirror as absent: removes the corrupt dir and re-clones through the existing staging-then-rename path.

b. Sweep orphaned staging dirs. A clone killed before MirrorStaging's Drop runs (SIGKILL, OOM) leaves a .<key>.staging-<pid>-<nonce> dir behind indefinitely. fetch now sweeps staging dirs for the key that have been idle past a one-hour grace window (liveness walks the whole staging tree, since a clone's pack writes leave the root mtime frozen early). Safety: the per-mirror lock excludes a concurrent full fetch of the key, and the grace window is far above any real clone, so an in-flight lock-free shallow clone is never reaped.

c. Re-fetch a locked source when its mirror cache is gone. A normal phora sync skips the fetch whenever the lock already pins a source, but the read paths (compute_digestopen_mirror) still open the bare mirror directly. Clearing the regenerable cache under XDG_CACHE_HOME while phora.lock survives left every locked source opening a vanished mirror — open mirror <src>: … does not appear to be a git repository. The self-heal from (a) never ran because the lock hit bypassed fetch entirely. A new SourceBackend::mirror_ready now gates the lock-hit fetch-skip: a missing or corrupt mirror re-fetches even under a warm lock. Reads stay offline by design, so the fix lives in sync orchestration, not the read path. The --frozen path is untouched — refusing to fetch a wiped cache under --frozen is correct.

Why

Discovered while wiring phora sync into a consumer repo: an interrupted clone on a slow network, a cache wipe racing a concurrent sync, or simply rm -rf ~/.cache/phora between syncs left the mirror cache in a state where retries failed with a confusing error and no self-recovery.

Tests

  • fetch_reclones_a_corrupt_canonical_mirror — corrupt the canonical mirror, assert the next fetch recovers to a valid bare repo and resolves.
  • fetch_reclones_when_in_place_fetch_fails — point origin at a dead remote, assert fetch self-heals via re-clone.
  • fetch_sweeps_a_stale_orphan_staging_dir — backdate an orphan past the grace window, assert fetch removes it.
  • fetch_keeps_a_staging_dir_with_recent_inner_writes — a staging dir whose root mtime is stale but whose pack write is recent is a live clone and must survive.
  • mirror_ready_tracks_fetch_and_a_cleared_cache — not-ready → ready after fetch → not-ready after the cache is cleared.
  • tests/scrut/lifecycle.md — end-to-end: rm -rf "\$XDG_CACHE_HOME/phora/git" then phora sync self-heals to sync complete (reproduces the reported error verbatim on the pre-fix binary).

Full suite: 1337 lib tests + 167 scrut cases pass, 0 failed; clippy -D warnings and rustfmt --check clean.

srnnkls added 2 commits June 30, 2026 23:10
A bare mirror dir that exists but won't `gix::open` — left by a `rename`
interrupted mid-publish, a disk error, or external damage — wedged every
later `sync` with "does not appear to be a git repository", recoverable
only by manually deleting the cache. `fetch` now treats an unopenable
mirror as absent: it removes the corrupt dir and re-clones via the
existing staging-then-rename path.

A clone killed before its `MirrorStaging` `Drop` runs (e.g. SIGKILL)
leaves a `.<key>.staging-<pid>-<nonce>` dir behind forever. Under the
per-mirror lock, `fetch` now sweeps staging dirs for the key that have
been idle past a one-hour grace window — long enough that an in-flight
lock-free shallow clone is never reaped.
A normal `phora sync` skips the fetch whenever the lock already pins a
source, but the read paths (`compute_digest` -> `open_mirror`) still open
the bare mirror. Clearing the regenerable cache under `XDG_CACHE_HOME`
while the lock survives left every locked source opening a vanished
mirror: `open mirror <src>: ... does not appear to be a git repository`.
The `fetch` self-heal never ran because the lock hit bypassed fetch.

Gate the lock-hit fetch-skip on a new `SourceBackend::mirror_ready`: a
missing or corrupt mirror re-fetches even under a warm lock. Reads stay
offline by design; the fix lives in sync orchestration. The frozen path
is untouched -- `--frozen` refusing to fetch a wiped cache is correct.
Copilot AI review requested due to automatic review settings July 1, 2026 11:12

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR improves GitBackend::fetch cache robustness by (1) self-healing when the canonical bare mirror directory is present but corrupt/unopenable, and (2) sweeping stale orphaned staging directories left behind by killed clone processes. It also updates the sync pipeline to only skip fetching on lock hits when a usable mirror is actually present.

Changes:

  • Add mirror self-heal logic: open-or-reclone flow under the per-mirror lock, with re-clone via staging + rename.
  • Sweep stale orphan staging dirs for a mirror key (with a grace window) before fetching.
  • Introduce SourceBackend::mirror_ready and use it in sync to avoid “lock-hit but cache-cleared” failures; update tests/docs accordingly.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
tests/scrut/lifecycle.md Adds an integration doc case ensuring phora sync recovers after the git mirror cache is cleared.
src/sync/tests.rs Updates counting/wrapper backends to forward the new mirror_ready method.
src/sync/resolve.rs Changes lock-hit fetch skipping to also require backend.mirror_ready(remote) so cleared/corrupt caches re-fetch.
src/source.rs Implements orphan staging sweep + mirror open/reclone behavior; adds mirror_ready implementations and targeted tests.
src/backend.rs Updates router backend to expose mirror_ready based on the git-backed mirror.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/source.rs
staging_is_recent swallowed WalkDir/metadata/modified errors and
continued, so a partially-unreadable but possibly-live staging dir could
be classified stale and reaped. Treat any traversal or mtime error as
recent, honoring the sweep's never-reap-a-live-clone invariant.
@srnnkls
srnnkls merged commit 74dd0a0 into main Jul 1, 2026
10 checks passed
@github-actions github-actions Bot mentioned this pull request Jul 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants