fix(dig-node): guard long-lived background loops against silent panic-death (#2067)

[MichaelTaylor3d]

Closes dig_ecosystem#2067. Extends the #2044 panic-guard from the tier-0 precache loop to the other long-lived tokio::spawn background loops in dig-node-core, so a transient per-iteration panic no longer silently kills a subsystem until process restart.

Approach

A single shared combinator crate::shared::catch_iteration(loop_name, fut) (new shared/panic_guard.rs) is now the ONE place the AssertUnwindSafe(fut).catch_unwind() machinery lives — Ok passes the value through transparently; Err (a caught panic) logs a fixed-shape tracing::warn!(loop_name, …) and returns None so the loop continues. tier0_live::run_round_catching (#2044) was refactored to delegate to it (DRY — no copy-pasted unwind machinery; its two #2044 tests stay green, behaviour unchanged).

Triage (every tokio::spawn in crates/dig-node-core/src/ classified)

• A — long-lived subsystem loop → GUARDED: chainwatch.rs run_loop (subscription-read + gap-fill tick), pex.rs run_tick_loop (delta flow), dht.rs run_maintenance (republish/refresh/gc). Plus tier0_live.rs (already #2044, now via the shared combinator).
• B — one-shot / per-connection / per-stream handler → left as-is: serve_one_stream, run_send_direction, per-capsule single-flight pull, spawn_capsule_warm, ctrl-c withdraw, etc. A panic kills one unit of work, not a subsystem — task-death is the correct fail-safe.
• C — test-only spawns → skipped.
• A-survive but DEFERRED (scope focus): a set of inline loops inside large peer.rs bring-up fns / helper fns (selector-registry feed, DHT-routing feed, pool-stats, relay-acceptor, spawn_pool_feeder, spawn_gc, run_first_peer_announcer, run_holdings_ingest) — each needs its per-iteration body extracted into a testable helper first. Tracked as a follow-up (see below).

AssertUnwindSafe soundness (per guarded site)

Verified per site: no MutexGuard crosses the catch boundary in any guarded loop — each iteration carries only Arc<…> seam handles, and every awaited call locks-and-releases internally (returns a plain value), so a caught mid-iteration panic can never observe a poisoned/half-held lock. No loop was left unguarded for lock-safety reasons; no site was genuinely ambiguous.

Real (non-panic) errors are never masked — the combinator intercepts ONLY unwinds; each loop's normal Result/outcome handling is untouched (a dedicated test asserts a plain Err flows through unchanged). Log is fixed-shape; the panic payload is never logged (#1603).

How verified (TDD)

• shared::panic_guard::a_panicking_iteration_is_contained_as_none — panic → None; non-vacuous (removing the catch_unwind aborts the test).
• shared::panic_guard::a_completed_iteration_passes_its_value_through_unchanged — transparent happy path.
• shared::panic_guard::a_normal_error_value_is_not_intercepted — a plain Err is NOT swallowed.
• chainwatch::a_panicking_tick_is_caught_and_does_not_propagate — a PanickingResolver makes a REAL run_tick unwind; the guard contains it (non-vacuous — unwrapping run_tick directly aborts). The pex/dht loop bodies share the same combinator mechanism, fully covered by the panic_guard panic-injection test.
• cargo test -p dig-node-core --lib → 662 passed, 0 failed; fmt + clippy -D warnings clean; build OK.

Follow-up (to be filed)

Guard the remaining inline (A)-survive loops (the deferred peer.rs/pex.rs/download.rs set above) by extracting each per-iteration body and wrapping it in crate::shared::catch_iteration. peer.rs run_relay_connection's loop lives in the dig_nat crate — panic containment there is a dig_nat concern.

Version

root [workspace.package].version 0.93.5 → 0.93.6 (patch, fix:); Cargo.lock regenerated.

---

Generated by Claude Code