fix(dig-node): keep the tier-0 precache loop alive across a round panic (#2044) - #173
Merged
Merged
Conversation
Co-Authored-By: Claude <noreply@anthropic.com>
…ic (#2044) A panic inside a run_round tick would unwind out of the spawned tokio task and silently stop tier-0 precache for the rest of the process (the node itself stays up), until restart. Wrap the round future in catch_unwind via a small, unit-testable run_round_catching helper: a caught panic becomes a bounded WARNING + skip-to-next-tick, while a completed round passes through unchanged. AssertUnwindSafe is sound — the round's only &mut state (rng, rate limiter) are plain value types with no lock/guard held across an await. Closes #2044 Co-Authored-By: Claude <noreply@anthropic.com>
MichaelTaylor3d
added a commit
that referenced
this pull request
Aug 3, 2026
…ot Ok(None)
The production `MeltChain` derived a melt from `AnchoredRootResolver::anchored_root()
== Ok(None)`, which was wrong in both directions and made a remote-triggered,
irreversible delete unsafe:
- `Ok(None)` is the node's fail-closed sentinel for "no confirmed generation".
`CoinsetResolver` produces it for "launcher coin is unspent (store not minted
yet)" — a store whose lineage has not STARTED. Deleting on it wrongfully erases
live data and broadcasts a false melt network-wide, correlated across holders.
- A genuine melt never produces it. A melted tip is spent without a datastore
child, so the lineage walk returns Err("singleton spend did not yield a store"),
which fail-closed to Unknown — the gate could not fire on a real melt.
Replace it with `confirm_melt_via_chain(&dyn ChainReads, store_id)`, composed from
two POSITIVE chain facts, neither of which re-walks the lineage (#747):
1. The launcher coin whose `coin_id == store_id` exists and is SPENT. `coin_id ==
store_id` is a 256-bit hash preimage that cannot be ground, so identity is
pinned to the one unforgeable on-chain anchor, never to a look-alike singleton
that merely curries `launcher_id == store_id` (#1473). An UNSPENT launcher is
Live — "not minted yet" is the opposite of melted.
2. The launcher's hint index is NON-EMPTY and every generation under it is SPENT.
One `coin_records_by_hint(store_id, include_spent = true)` read gives both. The
non-empty half is load-bearing: an empty index is indistinguishable from an
un-indexed store, so it resolves to Unknown rather than authorizing a delete.
Every error, absence, or ambiguity resolves away from deletion, and fact 2 is
deliberately conservative — candidates are not launcher-anchored before they count
as live, so an adversary who can plant hints can only suppress a deletion, never
cause one.
Tests: 8 new cases drive the real `ChainReads` trait with crafted coin records —
the link the spy-driven policy tests could not reach. Unused trait methods on the
mock are `unimplemented!()` so the gate cannot silently grow a chain dependency,
and the mock asserts `include_spent = true`. All six inverting mutations of the
gate were confirmed to fail their test.
Also: both new background loops wrap their per-iteration body in
`shared::catch_iteration` (#173/#174/#175) with `recv()`/`tick()` outside the
guard, so a persistently-panicking iteration paces instead of hot-spinning;
`TombstoneSet` recovers from lock poisoning rather than panicking forever after
one contained panic; SPEC.md 14.5 corrected — it documented the `Ok(None)` rule.
root [workspace.package].version -> 0.94.0 (minor, new capability).
Co-Authored-By: Claude <noreply@anthropic.com>
MichaelTaylor3d
added a commit
that referenced
this pull request
Aug 3, 2026
…ot Ok(None)
The production `MeltChain` derived a melt from `AnchoredRootResolver::anchored_root()
== Ok(None)`, which was wrong in both directions and made a remote-triggered,
irreversible delete unsafe:
- `Ok(None)` is the node's fail-closed sentinel for "no confirmed generation".
`CoinsetResolver` produces it for "launcher coin is unspent (store not minted
yet)" — a store whose lineage has not STARTED. Deleting on it wrongfully erases
live data and broadcasts a false melt network-wide, correlated across holders.
- A genuine melt never produces it. A melted tip is spent without a datastore
child, so the lineage walk returns Err("singleton spend did not yield a store"),
which fail-closed to Unknown — the gate could not fire on a real melt.
Replace it with `confirm_melt_via_chain(&dyn ChainReads, store_id)`, composed from
two POSITIVE chain facts, neither of which re-walks the lineage (#747):
1. The launcher coin whose `coin_id == store_id` exists and is SPENT. `coin_id ==
store_id` is a 256-bit hash preimage that cannot be ground, so identity is
pinned to the one unforgeable on-chain anchor, never to a look-alike singleton
that merely curries `launcher_id == store_id` (#1473). An UNSPENT launcher is
Live — "not minted yet" is the opposite of melted.
2. The launcher's hint index is NON-EMPTY and every generation under it is SPENT.
One `coin_records_by_hint(store_id, include_spent = true)` read gives both. The
non-empty half is load-bearing: an empty index is indistinguishable from an
un-indexed store, so it resolves to Unknown rather than authorizing a delete.
Every error, absence, or ambiguity resolves away from deletion, and fact 2 is
deliberately conservative — candidates are not launcher-anchored before they count
as live, so an adversary who can plant hints can only suppress a deletion, never
cause one.
Tests: 8 new cases drive the real `ChainReads` trait with crafted coin records —
the link the spy-driven policy tests could not reach. Unused trait methods on the
mock are `unimplemented!()` so the gate cannot silently grow a chain dependency,
and the mock asserts `include_spent = true`. All six inverting mutations of the
gate were confirmed to fail their test.
Also: both new background loops wrap their per-iteration body in
`shared::catch_iteration` (#173/#174/#175) with `recv()`/`tick()` outside the
guard, so a persistently-panicking iteration paces instead of hot-spinning;
`TombstoneSet` recovers from lock poisoning rather than panicking forever after
one contained panic; SPEC.md 14.5 corrected — it documented the `Ok(None)` rule.
root [workspace.package].version -> 0.94.0 (minor, new capability).
Co-Authored-By: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes dig_ecosystem#2044. Resilience close-out for the tier-0 precache flywheel (the subsystem hardened this session via #2033/#1934/#2041/#2053).
Why
spawn_tier0_precache(crates/dig-node-core/src/tier0_live.rs) runsrun_round(...).awaiteach tick inside a baretokio::spawnloop. The workspace has nopanic=abortoverride (default unwind), so a panic inside a round unwinds and kills the spawned task — the node itself stays up (fail-safe for node availability), but precache then silently stops until the next process restart, degrading the whole tier-0 flywheel. Flagged as a non-blocking residual by the #1934 PR-3 adversarial gate.What changed
run_round_catching<F: Future<Output = RoundOutcome>>(round, tick) -> Option<RoundOutcome>— wraps the round instd::panic::AssertUnwindSafe(round).catch_unwind().await(viafutures::FutureExt, already a dependency):Ok→Some(outcome)(fully transparent on the happy path);Err(_)→ a fixed-shapetracing::warn!(tick, …)+None.let Some(outcome) = run_round_catching(round, tick).await else { continue };— a panicking round no longer unwinds the task; the next tick runs a fresh round with the carried-overrng/rate.AssertUnwindSafeis sound here: the round's only&mutstate (SplitMix64rng,RoundRateLimiter) are plain value types holding no lock/guard across an await, so a caught mid-round panic leaves them in a valid state and the loop continues safely.tokio::spawnsupervisor was deliberately NOT used — the round borrows non-'static&mutstate, so a nested spawn would force restructuring ownership.run_round's normal (non-panic) outcome/error handling is untouched — the guard only intercepts unwinds, never masks real outcomes. Log message is fixed-shape; the panic payload is never logged (#1603 hygiene).How verified (TDD)
a_panicking_round_is_caught_and_does_not_propagate— builds a REALrun_round(...)with aPanickingProbe(panics inobserve_near) and assertsrun_round_catchingreturnsNone. Non-vacuous: removing thecatch_unwindmakes the test itself unwind/abort instead of returning — so the guard, not the harness, contains the panic. (A panic backtrace may print to stderr while the test still reportsok.)a_non_panicking_round_returns_its_outcome_unchanged— the catch is transparent on the happy path (Some(outcome)).cargo test -p dig-node-core --lib→ 658 passed, 0 failed; fmt +clippy -D warningsclean; build OK.Follow-up (filed)
Other
tokio::spawnbackground loops in dig-node-core (chain-watch, subscriptions, gap-fill) may share the same silent-panic-death pattern — a defense-in-depth audit sweep is tracked as dig_ecosystem#2067.Version
root
[workspace.package].version0.93.4 → 0.93.5 (patch,fix:);Cargo.lockregenerated.Generated by Claude Code