fix: harden the stall defect class found auditing the 1.2.2 fix

[gloryfromca]

Audit follow-up to #384 / #386. The version-cleanup stall those fixed had a shape worth generalising: an unbounded await in a path whose scheduler treats "not finished" as "skip" — so one hang stops that work forever, and because nothing failed, every health signal built from failure counters reads as healthy.

Seven instances were found. Five are fixed here; the reasoning for what is deferred is at the bottom.

What each fix addresses

Defect	Consequence if hit	Evidence grade
5 read ops had no deadline	whole md → LanceDB projection stops; rows stuck in processing; /health green	never observed in ~120h of soak
heartbeat / rebuild loops had no outer try	that loop dies permanently, zero log output	not observed; reachable only via create_task on a closing loop
fallback rebuild reset the alert counter	optimize_failure_streak >= 5 unreachable (~1% observable)	logic certain; the triggering condition never observed
husk sweep relied on a lock it cannot hold	table left with no FTS index → that kind's searches 500	three-way race, very low
memory-root lock wait unbounded + silent	startup looks like a hang with no explanation	not observed

None is high-probability. What they share with the fixed bug is the shape, and one of them (reads) has a strictly larger blast radius than the original — it stops all projection, not one table's cleanup.

Notable reasoning

Reads were deliberately skipped last round on the grounds that a read takes no lock and so blocks no writer. That is true and was the wrong conclusion: the drain loop reads on every batch and advances one batch at a time, so a hung read parks the pipeline even though it parks no lock.

The husk sweep's docstring contract was unsatisfiable. It required the write lock, but the sweep runs via asyncio.to_thread and a deadline cancels the future, not the thread — so an orphan sweep outlives any critical section, and Path.iterdir is a lazy os.scandir that can yield a dir created after the scan began. Holding the lock harder does not fix it; an age filter (skip dirs younger than lance's own 7-day unverified threshold) holds regardless of lock ownership, so that is the guarantee now.

The lock wait was made pollable rather than merely bounded. Wrapping a blocking flock in a timeout is worse than no timeout: cancelling the coroutine leaves the worker thread to acquire the lock later with nobody left to release it. Short LOCK_NB attempts on an interval give the same semantics while being bounded and cancellable.

Loop supervision escalates to process exit (SIGTERM, so the ASGI server shuts down gracefully) after 5s/15s/45s restarts. This assumes a restarting supervisor; without one the process stops, which still beats serving searches from a silently frozen index.

Verification

• 1925 unit + 182 integration passing; ruff clean; import-linter 3/3 contracts kept.
• Mutation-verified — each fix reverted individually, confirming the new test turns red rather than merely passing alongside it:

Reverted to	Test outcome
table handle outside the read deadline	hangs (the defect's actual signature)
husk sweep back inside the write lock	fail
age floor set to 0	fail
restart budget removed	fail
process-exit request removed	fail
alert counter reset restored	fail
waiting log removed	fail

Deferred, with reasons

• Rebuild starvation — the optimize runner's loop condition is "while there is unindexed data", which under sustained writes is never, so the 12h rebuild sweep waits it out. The 180s bound here makes that visible (cascade_lancedb_rebuild_skipped_optimize_unfinished) but not absent. The functional fix needs the runner to yield when a rebuild is pending, which changes the optimize↔rebuild mutual-exclusion contract — both commit on the same manifest version, so getting it wrong is worse than the bug. It also cannot be validated today: the cadence is 12h and no soak run has exceeded 2h, so the periodic sweep has only ever run at boot (where there is no write load). Making the cadence configurable is a prerequisite.
• OME config hot-reload (awatch outside its try) — only the optional OME subsystem's hot reload dies; running config is unaffected and a restart clears it.
• Progress clocks in /health — five of the seven are invisible because the verdict is assembled from failure counters, which a hang leaves untouched. last_prune_at is the one exception and is the only reason the original bug was ever found. Giving each long-lived loop a last-success clock is the structural fix; the drain half is nearly free, since max_lsn / last_processed_lsn are already computed and then dropped when building CascadeHealth.

Review follow-up (adversarial pass)

Four findings from the adversarial review, fixed on this branch (each with a mutation-verified test where applicable):

• Supervisor restart budget is now per incident, not per process lifetime — a body that ran 60s+ before crashing gets the full ladder back, so independent transients days apart cannot pool into a SIGTERM.
• A husk-sweep timeout no longer fails prune() — the cleanup commit has already succeeded at that point; letting the best-effort sweep's deadline escape counted a prune "failure" and stalled the prune-staleness clock for a stall that did not happen. Reachable at ceiling-load steady state, where sweep time sits right at the 60s budget.
• The husk reclaim horizon is ~14 days, not 7 — POSIX bumps a dir's mtime when the cleanup empties it, so the age gate restarts then; docstring numbers corrected to ~1.8M dirs / ~7GB at ceiling (the "never more aggressive than lance" property is unaffected).
• Memory-root lock default raised to 30min — 300s sat at the edge of the legitimate hold; the bound's job is diagnosis (the wait is already logged), so it now sits an order of magnitude above, and the timeout message points at the still-alive holder.

🤖 Generated with Claude Code