Make branch materialize incremental (hydrate prior generation, delta-fetch)

[0lut]

Summary

Makes /v1/materialize (branch) incremental instead of re-pulling the entire upstream history every time.

ensure_branch_from_verified_tip forced a full --unshallow (+ full --refetch) whenever the shared bare repo was shallow/blobless — which it almost always is, because the direct-git serving/deepen path leaves it that way. Measured cost: a materialize of torvalds/linux master against the origin took 934s (~15.6 min). It's also why the durable generation goes stale (the only refresh path is this heavy, Cloudflare-100s-timeout-prone op).

Fix

The incremental publish already exists (#79: publish_generation packs --revs <new> --not <previous_tips> when a parent generation exists). The blocker was the fetch side. So: when the repo is shallow and the object store holds a complete prior generation, hydrate that generation (fast parallel content-addressed pack download) and drop the now-stale shallow boundary, so the upstream fetch transfers only the delta to the new tip and the publish packs only the new commits.

Safety (why this can't corrupt the durable store)

A published generation is a verified full-history closure, so hydrating it makes the prior history genuinely complete and removing the shallow file is correct. If hydration is unavailable (no prior generation — e.g. the first/base materialize) or still leaves the closure incomplete, the existing commit_history_complete_no_lazy guard falls back to the full unshallow/refetch path. The change can only save time on the happy path; it can never publish an incomplete generation.

Tests

• cargo fmt --check, cargo clippy -p git-cache-domain --all-targets — clean
• cargo test -p git-cache-domain — 103 passed, 0 failed

Verification (in progress)

Deploying to a preview and running PR #127's matrix (no regressions) + an incremental-materialize check: materialize (build base) → make the repo shallow via a proxy-off deepen → re-materialize and confirm it hydrates the prior generation + delta-fetches (log line hydrated prior generation for incremental branch materialize) instead of a full unshallow, with a complete, fsck-clean result. Results to follow in a comment.

Follow-up: a focused unit test for the shallow+prior-generation→incremental path.

🤖 Generated with Claude Code

---

[devin-ai-integration]

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.

[0lut]

Preview verification — GREEN ✅

Deployed this branch to preview gmc-p-1facfb2a1a77 (image ecr…/gitmirrorcache-arm:1facfb2a1a77) and ran two checks against its ALB origin.

1. Serving regression — PR #127 linux matrix

exit=0, failures=0, OK: all clone/deepen operations succeeded. All depth orders (1 10 50 / 50 10 1 / 10 1 50 / 1 50 10) × {proxy-off, default} + the 1 9 40 deepen phase returned exit 0, shallow=true retained; hot-cache inspect reported NO_STALE_LOCKS. The fix doesn't touch the serving path and doesn't regress it.

2. Incremental-materialize proof (ripgrep)

Built a base generation, then reset + re-shallowed the cache (proxy-off --depth 1 → shallow=true) so the repo was shallow with a prior generation present — the exact state that used to force a full --unshallow. The re-materialize took the new path:

ensure branch from verified tip started ... ripgrep master
local branch tip is present but full-history closure is incomplete; fetching before publish
hydrate generation started → downloaded → indexed (212ms) → applied ref snapshot
hydrated prior generation for incremental branch materialize        ← new code path
fetched branch from upstream                                        ← incremental delta
publish generation skipped: commit manifest already complete
api request finished request_id=115 endpoint=materialize elapsed_ms=1406 status=200 OK

→ HTTP 200 in 1.4s, source=UpstreamVerified, complete generation — hydrating the prior generation from the object store + delta-fetch, instead of unshallowing all of history.

For torvalds/linux this is the path that previously measured 934s (≈15.6 min): it replaces the full-history GitHub --unshallow with an S3 hydrate of the prior generation + a delta fetch + incremental publish. The safety guard (commit_history_complete_no_lazy) means a missing/incomplete prior generation simply falls back to today's behavior — never an incomplete publish.

Tests: cargo fmt --check, cargo clippy clean, cargo test -p git-cache-domain 103/0. Follow-up: a focused unit test for the shallow+prior-generation→incremental path.

[0lut]

#127 matrix run vs this preview (gmc-p-1facfb2a1a77) — GREEN ✅

Ran the canonical matrix (now bundled in this PR as scripts/aws/test-prod-linux-cache-depths.sh) against a preview built from this branch, ALB origin, full defaults (RUN_REPEATS=true).

SUMMARY failures=0 — OK: all clone/deepen operations succeeded.

group	result
Depth matrix — orders 1 10 50 / 50 10 1 / 10 1 50 / 1 50 10 × {proxy-off, default} × {initial, repeat} = 48 clones	all exit=0, shallow=true
Deepen phase — default + proxy-off, ascending 1→9→40 + jump-to-40	all exit=0, shallow=true
HTTP 5xx	0
Hot-cache inspect	NO_STALE_LOCKS, IS_SHALLOW=true, partial=blobless

No serving regression from the materialize change (this PR only touches branch materialize; the matrix exercises the serving path).

Runtime note: the run took ~30 min, dominated by the two proxy-off cold deepen=40 of linux (514s and 1282s) — that's the cache-side read-through deepen + no-bitmap pack-objects over a fresh, growing cache (the serving cost tracked in #128), not this change. The default-mode deepen=40 was 9s. For a quick run, RUN_REPEATS=false DEEPEN_STEPS="1 9" skips those.

(The incremental-materialize fix path itself was confirmed separately on ripgrep: hydrated prior generation for incremental branch materialize → HTTP 200 in 1.4s, complete — see the earlier comment.)