fix(peer): require agreement between independent sources before a reflexive address is believed - #571
Conversation
WIP stub so a session cap cannot lose this lane (CLAUDE.md 1.8).
…flexive address is believed Adopts `dig_stun::establish` (0.1.1, published) rather than a hand-rolled agreement rule (dig-node#566). Per address family: every reading must agree UNANIMOUSLY on one IP, reported by >=2 independent source classes (>=3 when every agreeing class is a peer class), and the agreed IP must be global-unicast. This retires `mirror::advertise::PublicAddress:: corroborated_addresses`, a rival implementation of the same primitive that could not fail closed on a dissenting THIRD reading -- it only checked whether SOME pair of differently-sourced readings agreed, so two agreeing sources beside a disagreeing third would still have been believed. Gather (crates/dig-node-core/src/seams/dig_peer/net.rs): - `gather_reflexive_readings` queries operator + relay tiers in FULL (this node's own infrastructure -- no third-party disclosure), then consults the public tier per address family, ONLY while that family still has fewer than `dig_stun::establish::MIN_INDEPENDENT_CLASSES` distinct classes without it. Once sufficient, public is never queried for that family. - Class labels are rendered via `dig_stun::establish::SourceClass` so this crate never hand-formats the grammar `establish` parses back out; a class is attached at the HOST (operator entry, relay host, public host) before DNS resolution collapses it into a bare `SocketAddr`, so two public hosts (Google/Cloudflare) count as different classes and a dual-stack relay's v4 + v6 answers count as the SAME class. - `StunPlan::discover_reflexive` (first-answer-wins) is UNCHANGED and keeps serving the hole-punch tier / DHT transport, which needs exactly one endpoint -- the two uses are deliberately not collapsed into one. Wiring (peer.rs, lib.rs): `PeerStatus` now stores every reading gathered (`Vec<(SocketAddr, String)>`), not a single `(addr, &'static str)` -- before this, `dig.getNetworkInfo`'s `reflexive_addr` array could structurally never carry more than one entry, so the agreement gate could never receive enough to agree over. `network_info()` publishes the full set. Gate (dig-node-service/src/mirror/advertise.rs): `PublicAddress::established` delegates to `dig_stun::establish`; `established_addresses()` replaces `corroborated_addresses()` as `effective_urls`'s AGREEMENT input. The `Effective`/`AdvertiseState` contract is UNCHANGED (still 6 variants, still the same wire shape via `dig-node-control-interface` 0.33) -- this PR only changes what counts as agreement, never the state machine around it (confirmed HIGH/CRITICAL blast radius via gitnexus impact: `effective_urls` fans into both the mirror-coin creation pass and the `control.mirror.*` RPC surface, 21 impacted symbols). Diagnostics: `dign network-info` now renders every reflexive-address reading plus the establish() verdict per family, so an operator debugging "why is my node uncorroborated" sees which sources answered, what each said, and which of unanimous/insufficient/disagreeing/not-global applies. Breaking (`!`): a reflexive address that a prior pairwise-agreement read would have believed (two sources agreeing beside an unrelated dissenting third) is now refused. Same class of changed-default as #569's cross-family rejection. Version 0.254.85 (assigned; main was 0.254.84, unmoved). Closes #566
…as no-public-address
Follow-up to the previous commit in this same branch: `established_addresses`
folds the routability check INTO agreement (`dig_stun::establish` refuses
anything short of `Scope::GlobalUnicast`), which `effective_urls`'s old
empty/non-empty split could not see -- a family that agreed perfectly on a
private or loopback reading fell through to `Uncorroborated` instead of
`NoPublicAddress`, even though that state's own doc already covers exactly
this case ("what was reported is not a public address"). Added
`PublicAddress::any_family_not_global` and used it in the branch.
Caught by three now-fixed pre-existing tests
(`a_derived_address_outside_global_unicast_is_refused`,
`an_operator_may_publish_a_lan_address_the_derived_path_refuses`,
`a_this_machine_reflexive_address_is_never_derived_into_a_coin`) plus one of
my own (`two_agreeing_classes_render_as_established`, network_info.rs) --
all four used to pass under the retired pairwise check, which never folded
routability into agreement at all.
`a_this_machine_reflexive_address_is_never_derived_into_a_coin`'s fixture
also needed reshaping, not just the state fix: its old shape put THREE bad
IPv4 addresses beside the good one in the SAME family, which the new
unanimous rule correctly reads as disagreement (refusing all four, not
"filter the bad ones") -- and its replacement accidentally used `::1`,
which folds to `0.0.0.1` under `dig_stun`'s fold-before-bucket discipline
and lands in PUBLIC_V4's own family, reproducing the same disagreement.
Settled on `fe80::1` (link-local, does not fold, stays genuinely IPv6).
Verified: `cargo test -p dig-node-service --lib` -- 838 passed, 0 failed.
dig-node-core unaffected (not touched by this commit; last full run on
these sources was 1129 passed, 0 failed).
|
IN PROGRESS -- not the verdict. Security-audit findings so far, head 1. The required
Root cause (read, not re-run -- traced through a) b) 2. The PR body is still the stub: "DRAFT -- WIP, do not merge, gate round not yet started ... Will update this body with the full change description before requesting review." It was never updated. There is no stated evidence of what was and was not reachable on a real host (dev machine is NAT'd, no outbound IPv6, per the dispatch) -- the SS2.6 end-to-end evidence bar is entirely unaddressed. 3. Positive confirmations (read against
Full verdict to follow. |
Security audit verdict: CHANGES-REQUIREDHead audited: Why not PASSThis is not a rejection of the design -- the AGREEMENT mechanism itself (
What I verified by READING (not by running)Traced Ranked findingsF1 -- BLOCKING (readiness, not a live custody defect). RED required check + draft/stub PR body + no e2e evidence. Fix: land a green F2 -- Correctness, should be fixed before merge, NOT a fund-safety issue. F3 -- Named, not gating: an available availability/griefing trade-off that should be a recorded decision, not an accidental byproduct of a stale test. Per-family unanimity (correctly implemented) means one dissenting/degenerate reading in a family can permanently deny establishment for that whole family even when a separate, properly 2-class-agreed address also exists in the same reading set -- confirmed by the third failing test, which still asserts the OLD (weaker, per-address) behaviour. This is the intended fail-closed direction per SPEC.md's own wording and I read it as SAFE for custody (never publishes a wrong address) -- but it is presently unresolved: is a single misbehaving/misconfigured source (bad relay, bad operator STUN entry, or a future dishonest peer under #3199) intended to be able to indefinitely block a legitimate node's establishment? Recommend deciding this explicitly and updating the stale test/SPEC.md to match, rather than leaving a red test as the only signal that the behaviour changed. F4 -- Cosmetic, not gating. Confirmed clean (see interim comment for detail)No rival implementation of the agreement primitive; class grammar correctly renders distinct classes per host ( Bottom line for the orchestratorDo not merge as-is. F1 is the actual blocker (red CI + draft body + no evidence) and is mechanical to clear. F2-F4 should ride in the same PR before it's marked ready. Once green, the PR body is filled in with real evidence, and F2 is fixed (F3 is a judgment call to record, not necessarily change), this is a re-gate against the new head, not a full re-audit -- the core |
…ix stale doc Responds to the security-audit gate on this PR (comment thread, head ddb5af5). Two of the four findings (F1's red tests, F2's NotGlobal mislabel) were already fixed in the previous commit before the gate's audited head; this commit addresses F3 and F4, which were genuinely new. F3 -- the design question. Measured, not guessed: per-family unanimity running BEFORE the global-unicast check (so a degenerate/non-global reading counts as ordinary dissent rather than being excluded first) is dig-stun's OWN documented behaviour, confirmed by reading its SPEC.md directly -- section 7.3 orders unanimity as step 3 and global-unicast scope as step 5, with the crate's own stated rationale: "a node behind a multi-egress NAT, a misconfigured relay, and a lying peer all look the same from here, and in every one of those cases advertising is wrong." It is not an artifact of how dig-node feeds readings into `establish` -- dig-node passes raw IPs through with no pre-filtering, and the discard is entirely dig-stun's own step ordering. Recorded in both places the gate asked for: `PublicAddress::established`'s doc comment and SPEC.md's §25.10 (the on-chain-facing normative contract), each stating the availability/griefing cost this accepts and why weakening it to a majority vote would be worse (an attacker who can cheaply add sources would outvote the honest ones). F4 -- `Reflexive::source`'s doc comment described the RETIRED inequality-only rule. Rewritten to describe the CLASS grammar `dig_stun::establish::SourceClass` actually keys agreement on, and to name that a bare tier label (`"relay"`) still corroborates correctly (string inequality is all `establish` needs for class-counting) but can never render as a `peer:*` class for the escalated three-class floor. No production code changed; `cargo test -p dig-node-service --lib`: 838 passed, 0 failed (same count as the previous commit, as expected for a doc-only change).
|
Responding to the CHANGES-REQUIRED verdict (audited head F1 (red F2 (NotGlobal→Uncorroborated mislabel) — fixed, same commit as F1 (same root cause, same fix). F3 (design question) — answered, measured against F4 (stale PR body replaced with the full description (including this response) — no longer the push-early Still draft pending your re-gate. |
|
Full-workspace evidence added to the PR body: |
|
Real CI confirmed (not just local runs): |
|
IN PROGRESS -- not the verdict. Security re-gate of PR #571, head Item 1 verified: the three test fixes are for the RIGHT reason, and my own prior hypothesis (bare-label grammar) was WRONG about the root causeDiffed
None of the three fixes weakens an assertion to force green; each is accompanied by a doc comment citing the specific mechanism ( Suite ran to completion, no filtering. Read the actual CI log for job 101378377967 ( Still verifying: item 2 (effective_urls mislabel exhaustiveness), item 3 (the degenerate-reading design answer), PR body honesty, and whitespace-collapse check. Posting this now per the post-as-you-go rule. |
Security re-gate verdict: PASSHead audited: 96179ee (resolved myself via gh pr view 571 --json headRefOid, matches dispatch). Scoped re-gate against the prior gates ddb5af5. 1. The three test fixes are for the right reason -- CONFIRMED, and my own prior "bare label" hypothesis was wrongAlready posted as an interim finding (see comment above). Diffed ddb5af5..c4b1bc4 directly: all three fixes are IP-range fixes (swapping RFC 5737 documentation-range addresses that dig_stun's own routability gate correctly refuses, for genuinely global-unicast test addresses), not label-grammar fixes. I additionally verified this against dig_stun::establishs actual source (verdict_for_family, fetched from the v0.1.1 tag): class-counting is done on the raw source string, deduped -- it does not require SourceClass::parse to succeed. A bare label like "relay" and a different bare label "stun.example" are two distinct strings and count as 2 independent classes just fine; an unparseable string only ever fails to reach the escalated 3-class peer:* floor (peer_only computation), which none of the three fixed tests exercise. So the bare labels the two mirror/advertise.rs tests kept were never wrong, and the fix commits own root-cause account (a real production bug in effective_urls's empty/non-empty split, not a fixture-label problem) is the accurate one. Suite genuinely ran to completion: CI log for job 101378377967 (Test + coverage) shows Summary [753.466s] 3152 tests run: 3152 passed (12 slow), 4 skipped -- the full count, not a truncated or zero-matched filter. 2. effective_urls() state mapping -- CONFIRMED correct and exhaustiveRead dig_stun::establishs actual FamilyVerdict enum (5 variants: NoReadings, Disagreement, Insufficient, NotGlobal, Established) and verdict_for_familys body. The new any_family_not_global() correctly isolates exactly the one variant (NotGlobal) that means "a family agreed but the address itself is not public" -- mapped to NoPublicAddress. Every other empty-agreed cause (NoReadings, Disagreement, Insufficient) correctly falls through to Uncorroborated, which is the semantically right bucket for all three (nothing reported, sources disagree, or too few independent classes -- all "need a second/better source," never "the reading itself is broken"). dig-apps two different remedies (already shipped at 15.1.0) get the right signal in every case I enumerated. 3. The degenerate-reading design question -- the lane LEFT THE BEHAVIOUR AS-IS and documented it; I agree with that call, verified independently against dig-stuns actual source, not just its SPEC proseFetched dig-stun v0.1.1's establish.rs and SPEC.md section 7.3 directly (not trusting the PR's paraphrase). Confirmed:
Also confirmed
Non-gating note (process, not security)dig-stun 0.2.0 was published at 2026-09-05T20:00:20Z, roughly 84 minutes before this PR's last commit (96179ee at 21:24:17Z), and Cargo.toml/Cargo.lock still pin 0.1.1. Per CLAUDE.md section 2.4b (bump touched crates' dig-* deps to latest in the same PR), this should have picked up 0.2.0. Confirmed via gh api compare v0.1.1...v0.2.0 that this is NOT a security concern -- the 0.2.0 diff is purely additive (a new unrelated credential module), establish.rs is byte-identical between the two versions, so nothing in this audits analysis is affected. Flagging for a follow-up bump, not gating the merge on it. What I verified by RUNNING vs. reading
No security defect found in this scoped delta. PASS. |
Implements #566: require agreement between independent sources before this node's reflexive
address is believed durably. Adopts
dig_stun::establish(0.1.1, published) rather than thiscrate's own agreement rule — this is a security primitive, and a second hand-rolled
implementation of one is exactly the rival CLAUDE.md's "centralize rival implementations" rule
exists to catch.
Response to the security-audit gate (head
ddb5af5a, this PR is now at a later head)The gate ran against my FIRST commit and returned CHANGES-REQUIRED with 4 findings. Status of each:
Test + coverage, 3 failures inmirror/advertise.rs) — FIXED, second commit(
c4b1bc4c, pushed before the gate's interim comment landed). Root cause was real, not a fixtureartifact:
established_addresses()folds routability into agreement, andeffective_urls's oldempty/non-empty split couldn't distinguish "genuinely nothing agreed" from "something agreed
perfectly but wasn't a public address." Added
PublicAddress::any_family_not_global(). One of thethree tests (
a_this_machine_reflexive_address_is_never_derived_into_a_coin) also needed itsfixture reshaped, not just the state fixed — detailed under "The gate" below.
guessed. Read
dig-stun's ownSPEC.md§7.3 directly: unanimity is step 3, global-unicast scopeis step 5 — the crate discards the whole family on ANY dissent, degenerate or not, BEFORE it ever
asks whether an individual reading is routable. This is
dig-stun's own documented behaviour, notan artifact of how this PR feeds it: dig-node passes raw IPs through with no pre-filtering, and the
step ORDER is entirely dig-stun's. Its own stated reason: "a node behind a multi-egress NAT, a
misconfigured relay, and a lying peer all look the same from here, and in every one of those cases
advertising is wrong." Recorded (third commit,
96179ee8) in both places asked for:PublicAddress::established's doc comment and SPEC.md §25.10. I did not change the behaviour —the gate itself said not to, and weakening unanimity to a majority vote would let an attacker who
can cheaply add sources outvote the honest ones.
Reflexive::sourcedoc comment) — FIXED, same third commit.What was actually broken
dig.getNetworkInfo'sreflexive_addralready published the[{"source","addr"}]wire shape(#567), and
mirror::advertise::PublicAddress::corroborated_addressesalready existed as anagreement check — but the two never actually met:
StunPlan::discover_reflexivestops at theFIRST tier that answers, and
PeerStatusstored at most ONE(SocketAddr, &'static str)reading.So the array published to
reflexive_addrcould structurally never carry more than one entry inproduction, and
corroborated_addresses(which needs 2+) was dead code. That is the "declines tocreate coins because a single STUN reading yields
AdvertiseState::Uncorroborated" state theticket names — this PR is what lets it stop declining.
corroborated_addresseswas also, independently, the WRONG rule: it accepted any PAIR ofdifferently-sourced readings that agreed, with no unanimity check. A dissenting THIRD reading
never blocked it — see the revert-proof below.
The gather (
crates/dig-node-core/src/seams/dig_peer/net.rs)gather_reflexive_readingsqueries the operator + relay tiers in FULL (this node's owninfrastructure — no third party learns anything) then consults the public tier per address
family, ONLY while that family still has fewer than
dig_stun::establish::MIN_INDEPENDENT_CLASSES(2) distinct classes without it. Once a family is sufficient, public is never queried for it.
dig_stun::establish::SourceClass(never hand-formatted), attachedat the HOST before DNS resolution collapses it into a bare
SocketAddr— two public hosts(
stun.l.google.com/stun.cloudflare.com) count as different classes; a dual-stack relay's v4relay:<host>, one class per relay host).StunPlan::discover_reflexive(first-answer-wins) is UNCHANGED and keeps serving thehole-punch tier / DHT transport (
bring_up_dht), which needs exactly one endpoint. The two usesare deliberately not collapsed into one, per the brief.
Wiring (
peer.rs,lib.rs)PeerStatusnow stores every reading gathered (Vec<(SocketAddr, String)>), not a single(addr, &'static str).network_info()publishes the full set in the SAME wire shape SPEC.mdalready documented (no wire-format change, just actually populated with N>1 entries).
The gate (
crates/dig-node-service/src/mirror/advertise.rs)PublicAddress::established()delegates todig_stun::establish;established_addresses()replaces
corroborated_addresses()aseffective_urls's agreement input.effective_urls'ssignature and the
Effective/AdvertiseStatecontract are unchanged — still 6 states, still thesame
dig-node-control-interface0.33 wire shape. Confirmed via gitnexusimpactbefore touchingeither symbol:
corroborated_addresseseffective_urlsdiscover_reflexivePeerStatus::set_reflexiveeffective_urlsfans into BOTH the mirror-coin creation pass (spawn_mirror_passes) and thecontrol.mirror.*RPC surface (dispatch_owned) — this is why its external contract is untouchedand only the internal agreement computation changed.
A follow-up fix needed on top of the swap (caught by 3 pre-existing tests + 1 of my own, all
now green):
established_addresses()folds the routability check INTO agreement (dig_stun::establishrefuses anything short of
Scope::GlobalUnicast), whicheffective_urls's old empty/non-emptysplit couldn't see — a family that agreed perfectly on a private/loopback reading fell through to
Uncorroboratedinstead ofNoPublicAddress, even though that state's own doc already covers this("what was reported is not a public address"). Added
PublicAddress::any_family_not_global()andused it in the branch. Second commit in this PR; both are pushed and green.
Diagnostics —
dign network-infoPer the brief ("an operator debugging 'why is my node uncorroborated' needs to see which sources
answered and what each said"): the CLI now renders every reflexive-address reading plus the
establish() verdict per family (unanimous/insufficient/disagreeing/not-global/no-reading), reusing
PublicAddress::from_network_info/establishedrather than re-deriving anything.Revert-proofs (committed first, per the harness's own rule)
established_addresses(), rana_dissenting_third_reading_blocks_establishment_even_though_two_others_agree— FAILED, and forthe intended reason:
Effective { urls: ["dig://93.184.216.34:9444"], state: Derived }— the oldcode silently advertised the majority address despite an independent dissenter. Restored.
if short { }guard ingather_from_classes, ranpublic_tier_is_skipped_once_operator_alone_reaches_the_class_floor— FAILED: the dissentingpublic:would-dissentreading appeared (3 readings instead of 2). Restored.A trap worth recording for whoever reads this next
dig-nat's own STUN client (query_reflexive_address) rejects a non-globally-usable mappedaddress, INCLUDING documentation ranges (RFC 5737
203.0.113.0/24), asStunError::NoMappedAddress— this file already had ONE comment warning about it
(
a_cross_family_stun_answer_is_discarded_the_other_direction_too) and I still walked into itbuilding my 3 new net.rs gather tests (all fixtures used
203.0.113.x; all 3 failed with emptyreadings until I switched to genuinely global-unicast literals —
93.184.216.34,1.1.1.1,matching the file's OWN established convention of
100.64.x.x/pure-function-only for that reason).Separately,
::1folds to0.0.0.1underdig_stun's fold-before-bucket discipline (SPEC.md §5.3)and is NOT safe as an "isolated in its own family" IPv6 test address —
fe80::1is.Found but explicitly NOT touched (§2.4b dep sweep, out of scope for this ticket)
crates/dig-node-service/src/mirror/advertise.rs's ownis_globally_routable/is_globally_routable_v4/is_globally_routable_v6are a RIVAL ofdig_stun::scope:: is_globally_routable— dig-stun's own SPEC.md already names this exact duplication("dig-node's on-chain gate") as one of the two tables it was built to unify. The brief said to
report rather than fix; reconciling it is a separate, larger PR (it's the routability check
derived_urlsruns, now largely redundant-but-harmless defense-in-depth sinceestablish()already excludes non-global addresses before
derived_urlsever sees them).dig-dht = "0.15"stays — bumping to the published0.16.1would split the dig-dht line: bothdig-download 0.22.1anddig-peer-selector 0.11.1(this crate's other direct/transitive users)still declare
dig-dht ^0.15on the registry. Upstream hasn't moved yet; matches the exactprecedent this crate's own Cargo.toml comment already documents for the 0.13→0.15 jump.
dig-constants/dig-chainsource-interfaceneeded-p name@versiondisambiguation duringcargo updatefor the same reason (two lines already resolved); neither is newly split by this PR.chia-protocol/chia-traits/chia-bls/chia-sha2/chia-puzzle-typesall show0.48.0oncrates.io while this crate pins
0.36.1, butchia-sdk-driver/chia-sdk-types/chia-sdk-utils(also direct deps here) are STILL at
0.36.0on the registry — bumping the first five alone wouldsplit the chia line exactly the way CLAUDE.md §2.4b warns against, and the prerequisite (the
chia-wallet-sdk line publishing a 0.48-compatible release) hasn't happened. Confirmed via the
registry, not memory.
dig-ipc-protocolWAS bumped (=0.3.0→=0.3.2, exact-pinned by design): checked itsCHANGELOG first — 0.3.1 is a chia-bls dep bump onto the SAME 0.36.1 line this workspace already
uses, 0.3.2 is docs-only. Neither touches the IPC wire shape the exact pin protects.
dig-*deps bumped viacargo update -p <name>(patch-level, already inside theirdeclared caret range, no
Cargo.tomltext change needed): dig-identity, dig-rpc-protocol,dig-message, dig-keystore, dig-nat, dig-constants, dig-social-profile, dig-pex, dig-download,
dig-peer, dig-peer-selector, dig-tls, dig-sex, dig-store-cache, dig-mirror-collateral,
dig-mirror-coin, dig-chainsource-interface, dig-cert, dig-logging, dig-urn-resolver.
Evidence bar (§2.6) — what I could and could not reach on this machine
This machine is NAT'd with no outbound IPv6, so a genuine end-to-end run (a real host reaching ≥2
independent classes and reporting a non-null established
reflexive_addr) was not something Icould produce here — stated plainly rather than implied. What I DID verify, all on real code paths:
public_tier_is_queried_and_kept_when_the_family_is_shortandoperator_and_relay_are_always_queried_in_fullprove the ordering + skip logic against REAL fakeUDP STUN servers on loopback (not mocks) — real Binding transactions, real responses.
dig_stun::establishcrate (not reimplemented), fed throughPublicAddress::from_network_infoexactly as production does.
always, plus 1 public query IF the relay answers (to reach the floor of 2), 2 public queries if
the relay is silent — never both public hosts once the floor is met. Never queries public at all
once operator+relay alone reach 2 classes.
Version
0.254.85(assigned; main was0.254.84, confirmed unmoved before bumping).cargo update -w --offlinesyncedCargo.lock— onlydig-node-service's own lock entry moved.Verified (all commands from the brief, actual counts)
cargo fmt --all -- --check— clean, exit 0.cargo clippy --workspace --all-targets --all-features --locked -- -D warnings— clean, exit 0(1m25s).
cargo test --workspace --all-targets --all-features --locked --jobs 4— exit 0, 3142 testspassed across 55 binaries, 0 failed (
grep -c "test result: FAILED"= 0). Includesdig-node-core: 1129 passed, 0 failed, 1 ignored (the live-network probe, correctly skipped
offline); dig-node-service: 838 passed, 0 failed; the rest across dig-wallet/dig-runtime/
dig-chat-protocol integration suites. Baseline for comparison: PR feat(peer): harden reflexive discovery (cross-family reject, stun host, mirror-advertise control) #569 (the immediately prior PR
on this file) reported ~3132 — the small increase is this PR's own new tests.
Blast radius checked (gitnexus
impact, per-worktree index, 14,490 nodes / 38,922 edges): tableabove.
detect_changes({scope: "compare", base_ref: "main"})run: 53 changed symbols across 10files, 3 affected cross-community processes (
dispatch_owned → Reflexive,run_peer_network → Derived_network_label,run_peer_network → Genesis_challenge_from), riskmedium— all match the intended blast radius (the gather machinery innet.rs,PeerStatusinpeer.rs,PublicAddress/AdvertiseStateinadvertise.rs, the CLI renderer innetwork_info.rs); no symbol outside that set was touched.Closes #566