feat: automatically recover repositories which fail to update

[notheotherben]

Summary

When a backup run is interrupted part-way through a fetch (power failure, OOM-kill, etc.), the local repository can be left in a state which blocks every subsequent update — most commonly stale git lock files such as packed-refs.lock:

Unable to fetch from remote git repository 'https://github.com/SierraSoftworks/grey.git' (User error)
This was caused by:
 - Failed to update references to their new position to match their remote locations
 - The lock for the packed-ref file could not be obtained
 - The lockfile at '/backups/repos/SierraSoftworks/grey/.git/packed-refs.lock' might need manual deletion.

Until now the only way out was manual intervention. The git engine now attempts automatic recovery whenever an existing local repository fails to update, staged from least to most invasive and controlled by a new per-policy recovery property (alongside the existing refspecs property):

Mode	Behaviour
none	Never attempt recovery; report the error as before.
non-destructive (default)	Remove stale git lock files and retry the fetch. Only locks older than 15 minutes are removed, so locks held by a live git operation are never touched.
destructive (opt-in)	Additionally clone a fresh copy of the repository into a hidden staging directory alongside the backup and, only if the clone succeeds, swap it into place.

- kind: github/repo
  from: repos/SierraSoftworks/grey
  to: /backups/github
  properties:
    recovery: destructive

The successful clone acts as a canary gating the destructive path: if the remote is unreachable, credentials are wrong, or the disk is full, the clone fails and the existing backup is left completely untouched — a transient remote failure can never destroy local data. The staging and discard directories live alongside the target so the swap uses same-filesystem (atomic) renames, and if moving the fresh clone into place fails, the original repository is restored.

Backups completed through recovery are annotated in the summary (e.g. unchanged at abc123; recovered by removing stale lock files), and recovery steps are logged as warnings.

Changes

• src/engines/git.rs — recovery pipeline: recover() (staging/ordering logic), remove_stale_locks() (recursive *.lock sweep of .git with a staleness threshold), reclone_and_replace() (clone → canary check → atomic swap → rollback on failure)
• src/entities/recovery.rs — new RecoveryMode enum with parsing/validation
• src/sources/github_repo.rs, src/sources/github_gist.rs — plumb the recovery policy property into GitRepo entities, with validation of the value at policy-validation time
• docs/advanced/recovery.md — new documentation page (registered in the VuePress nav/sidebar), plus an example in examples/config.yaml

Testing

• Unit tests for RecoveryMode parsing, policy validation, stale-lock sweeping (threshold respected; non-lock files and fresh locks untouched), and state annotation
• End-to-end tests (network-gated like the existing test_backup): stale-lock recovery retries the fetch successfully, and destructive recovery replaces a repository with a corrupted .git/config then fetches normally afterwards
• cargo test --features pure_tests (134 passed), cargo test engines::git including network tests (9 passed), cargo clippy and cargo fmt --check clean

🤖 Generated with Claude Code

https://claude.ai/code/session_017s9x49j7QCzeqBLWku995F

---

Generated by Claude Code

[codecov]

Codecov Report

❌ Patch coverage is 90.35370% with 30 lines in your changes missing coverage. Please review.
✅ Project coverage is 75.74%. Comparing base (3edb835) to head (0070951).

Files with missing lines	Patch %	Lines
src/engines/git.rs	89.05%	30 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #617      +/-   ##
==========================================
+ Coverage   73.25%   75.74%   +2.48%     
==========================================
  Files          30       31       +1     
  Lines        2109     2420     +311     
==========================================
+ Hits         1545     1833     +288     
- Misses        564      587      +23     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[notheotherben]

@copilot Can we avoid using a .unwrap() here to ensure we don't panic if parsing fails for some reason? Create a new PR against main with the fix