fix(node): make peer bring-up honest and add a peer-ping ladder diagnostic

[MichaelTaylor3d]

Two connected peer-network defects: a bring-up that goes silent for 13 minutes, and the diagnostic
that would have explained it.

Closes DIG-Network/dig_ecosystem#1974
Closes DIG-Network/dig_ecosystem#1985

---

1. Bring-up (#1974) — the ticketed root cause was wrong

The ticket blamed the S3-backed capsule scan. It is not that: cache_list_cached only stats each
capsule, it never reads capsule bytes. The cost is announce_inventory — a sequential
for id in &ids { dht.announce_provider(id).await }, where each announce is a full iterative
Kademlia lookup plus a PUT at the k closest peers, and every RPC inside it is bounded by the 5s
DHT timeout. Against the near-empty routing table bootstrap leaves behind, those RPCs time out
rather than resolve, so an announce costs roughly its whole timeout budget:

44 capsules across 34 stores = 68 content ids × ~11.2s = 761s, against the 760s observed.

The #1927 fleet nodes look fast on the same version for a reason that has nothing to do with EBS:
their caches are empty, so they announce nothing. An EBS-backed node holding 44 capsules would
have stalled identically. Worse, the announces ran before spawn_dht_routing_feed, so all 68
queried the emptiest routing table they would ever see.

Fix. Record the inventory content ids on the DhtHandle during bring-up (cheap — a readdir plus
one stat per capsule), and move the network announce to a background task started after the
pool→routing feed is live, with bounded concurrency and start/progress/completion logging. The
recorded ids are authoritative from step 4 either way, so a reconcile_inventory during the
announce still diffs against the truth. cache_list_cached also moves onto spawn_blocking — on
the S3-backed cache each stat is a round trip that was parking a tokio worker.

The concurrency test is mutation-checked: reverting announce_inventory_ids to the sequential
loop fails it with announcing 68 ids with 8 in flight took 340s, which is not meaningfully faster than the 340s a one-at-a-time loop would cost. The bound is self-calibrating (it times one real
announce_provider against the same stub transport rather than hardcoding a duration) and runs on
tokio's paused clock, so it is deterministic rather than a wall-clock race.

2. The connection-ladder diagnostic (#1985)

control.peers.ping and dig-node peers ping <peer> [--peer-id <64hex>]: dial a peer one ladder
rung at a time and report every rung plus a graded verdict. An open port is not a peer connection
— it says nothing about whether the mTLS handshake succeeds or whether the certificate binds the
identity asked for — and "connected" hides whether the direct path was available, which is exactly
what hid #1929.

What makes it honest rather than a port probe:

• The ladder, not the winner. Every rung is probed even after one succeeds; a rung the deadline
  pre-empted is reported skipped with a reason, never dropped.
• Identity outranks reachability. A rung that reaches the right address with the wrong
  certificate grades identity-mismatch / error, however well it connected. An explicit peer_id
  param always wins over what the node believes is at that address — otherwise the mismatch case
  could never be exercised, because it would be silently repaired into a pass.
• A relay-only success is warn, not error, and says so in words. Most peers on the network
  are behind NAT and relay-reachable only; grading that as failure would report a healthy network as
  broken to every user who ran it.
• No anonymous dial. dig-nat pins the expected peer_id in its TLS verifier, so an address with
  no known identity is refused with an explanation rather than downgraded to a TCP probe.
  Degrading there would answer exactly the "an open port means connected" question this replaces.
• One implementation. net::single_tier_nat_config shares its builder with full_nat_config,
  differing only in narrowing enabled_methods — so the ping is the real dialer restricted to one
  rung, not a parallel prober that could drift. No dig-nat change was needed.
• "Not configured here" is not "failed there". A tier this node cannot compose (UPnP with no
  port mapping, NAT-PMP/PCP with no IPv4 gateway, hole-punch with no reflexive address) reports
  unavailable with the missing LOCAL precondition and no elapsed_ms, never failed — otherwise a
  perfectly reachable peer comes back with four red rows blaming the peer for this node.
• Read-only + bounded. Each connection is dropped the instant it is graded (no pooled session,
  no retained relay circuit); per-tier timeouts plus a 45s overall deadline.

Anti-amplification. The method makes the node dial a caller-supplied address, which is a
request-forgery shape. The gate lives on PeerPingContext, not in the control shell, so a second
caller cannot reach the dialer without it: single-flight (the hard load bound — however many callers
ask, never more than one ladder's worth of dials outstanding) plus 6 starts per 60s (single-flight
alone does not bound a target that refuses every tier instantly). A concurrency refusal costs no
rate budget, and resolution runs before the gate, so an unparseable argument cannot lock out a
caller who then types a real one. A refusal returns the new catalogued PEER_PING_REFUSED (-32060)
— nothing was dialed, so it is not dressed up as a ladder result.

Shell-owned, deliberately. The delegated control.* methods dispatch on the Method enum in the
external dig-rpc-protocol crate, so a delegated ping would need that crate released first. Ping
changes no node↔node wire contract, so it is owned where it costs no cross-crate release.

Also

The new background announce is wrapped in crate::shared::catch_iteration (#2067/#2068, which
landed on main while this branch sat). A panic in a detached tokio::spawn is swallowed with the
dropped JoinHandle and logs nothing — this node would silently never publish its inventory for the
rest of the process, the exact invisible failure moving the announce off the bring-up path was meant
to end.

Blast radius checked

peer.rs::run_peer_network / bring_up_dht (the announce call site + the new context install),
dht.rs::announce_inventory (one other caller: the inventory-refresh hook, unchanged semantics),
capsule_store.rs::cache_list_cached (walk body relocated to a free function, byte-identical),
net.rs::full_nat_config (every node dial site — refactored to share a builder, same output), and
the three lockstep gates that fail on an unregistered control method
(control_methods_partition_into_owned_and_delegated, cli_covers_every_node_control_method, the
OpenRPC drift guard).

How verified

• cargo test --workspace --locked — 35 binaries, 1535 passed, 0 failed.
• cargo clippy --workspace --all-targets --locked -- -D warnings — clean.
• cargo fmt --all -- --check — clean.
• Version-increment, Commitlint, Native-install-packages — green on this head.

Every new test was falsified, not just run — the guarded code was broken, the test was confirmed
RED, and the original bytes restored. The harness asserts the mutated bytes actually differ on disk
before trusting a kill, so a replacement that silently no-ops cannot be reported as a pass:

Broken on purpose	Test that went red
removed the single-flight compare_exchange	only_one_ladder_may_run_at_a_time
charged the rate window before the single-flight claim	a_concurrent_refusal_costs_no_rate_budget
removed the starts >= MAX_PINGS_PER_WINDOW check	the_start_rate_is_bounded_within_the_window
made the window never reopen	the_window_reopens_once_it_has_elapsed
pinned the severity marker to a constant	the_summary_marker_tracks_the_graded_severity
rendered only the connected rung	every_rung_is_rendered_not_just_the_winner
dropped control.peers.ping from the CLI verb list	cli_covers_every_node_control_method
dropped it from OWNED_CONTROL_METHODS	control_methods_partition_into_owned_and_delegated
deleted the dispatch_owned arm	control_peers_ping_is_reachable_token_gated_and_degrades_honestly
graded an uncomposable tier as failed	a_tier_this_node_cannot_compose_reads_as_unavailable_not_failed
emitted elapsed_ms on an unavailable rung	an_unavailable_rung_serialises_distinctly_and_claims_no_elapsed_time

The pre-existing concurrency test was already mutation-checked the same way: reverting
announce_inventory_ids to the sequential loop fails it.

What is NOT verified — read this before treating the numbers as measurements

#1974's "after" is a model, not an observation, and the ticket's original root cause was wrong for
exactly that reason. The 12m40s "before" is the reported observation; the arithmetic
(68 content ids x ~11.2s = 761s against the 760s observed) is a fit to that observation, and the fix
is supported by unit tests plus that fit. No restart has been measured end to end.

I could not close that gap here, and the reason is specific rather than a matter of effort:
DhtService::announce_provider returns Ok(0) immediately when seed_contacts is empty
(dig-dht-0.9.0/src/service.rs:230). A lone local node therefore announces instantly and cannot
reproduce the pathology at all — it needs a routing table that is populated but slow to answer,
which is what the unit test's seeded SlowTransport models and what a real bootstrapped node on a
sparse network has. Reproducing it for real means a multi-node fleet (the loop-e2e-p2p EC2
harness), not a local run.

The honest acceptance test is one restart of the S3-backed rpc.dig.net node plus a log read: the
mTLS peer-RPC listening on [::]:9444 line should now appear within seconds of the relay
reservation instead of ~13 minutes after it, and the new DHT announcing N content id(s) /
announced N of M / announced N ... in Xs lines should carry the announce afterwards. Until that
run exists, this PR has changed the ordering and proved the ordering in tests — it has not
demonstrated the wall-clock improvement on real hardware.

#1985's acceptance criteria — two proven against a real mTLS handshake, one still unproven live.

The wrong-peer_id criterion was the interesting one: it was not merely unproven, it was
unreachable in production. dig-tls pins the expected id in its certificate verifier, so a
mismatched certificate aborts the handshake and no connection is produced — verdict() only
inspected connections, so a real impersonation reported unreachable. It is now surfaced from the
handshake error, and crates/dig-node-core/tests/peer_ping_identity.rs proves it against a REAL
serve_peer_rpc_listener over loopback: wrong pin → identity-mismatch naming who actually
answered; right pin → direct on the direct rung. Both directions are mutation-checked.

#1985 criterion	Evidence
wrong peer_id reports an identity mismatch	proven over a real mTLS handshake (loopback)
a reachable peer reports direct success	proven over a real mTLS handshake (loopback, IPv4)
fleet node direct over IPv6	unproven — needs a real fleet; loopback here is 127.0.0.1
NAT'd peer reports relayed-not-broken	unproven — needs a genuinely NAT'd peer and a relay

Reachability of the method itself is separately proven end to end: an integration test drives
control.peers.ping over the real HTTP control surface on a live node and asserts it is token-gated,
registered, routed to its own handler, and honest when no peer network is running.

Version

[workspace.package] 0.94.0, against main at 0.93.9 — minor: control.peers.ping is a new capability and
dig-node-core gains public API (PeerNetwork::set_peer_ping_context/peer_ping_context,
ping::*, net::single_tier_nat_config). Nothing removed, no wire contract changed.

Follow-ups (separate repos, not silently dropped)

• dig-app — #1985 asks for a ping-node option in the app. The node-side method exists now.
• docs.dig.net — control.peers.* is documented in docs/protocol/peer-network.md and
  docs/run-a-node/run-a-relay.md across 14 locales; the new verb belongs there (§4.3).

[MichaelTaylor3d]

Lane paused on a §1.4 single-writer conflict: another lane now owns dig-node for DIG-Network/dig_ecosystem#1997 (live rpc.dig.net outage) and DIG-Network/dig_ecosystem#1995 (.dig suffix restore).

Everything is pushed at 9c4c4fb and green (cargo test -p dig-node-core --lib → 546 passed, 0 failed). Staying draft until the owning lane merges.

Overlap to be aware of: this branch relocated the cache_list_cached directory walk out of the CapsuleStore impl into a free list_cached_capsules(&Path) in the same file, and #1995 targets that same walk. Textual conflict, trivial semantic one — the walk body is byte-identical, only moved.

Full resume-ready detail: DIG-Network/dig_ecosystem#1974 and DIG-Network/dig_ecosystem#1985.

[MichaelTaylor3d]

PR #149 is out of draft and gate-ready

• Repo/branch: DIG-Network/dig-node fix/1974-bringup-visibility
• HEAD: 5ede045, rebased onto origin/main 1e9dd52, 0 commits behind
• PR: fix(node): make peer bring-up honest and add a peer-ping ladder diagnostic #149 — no longer a draft
• Version: [workspace.package] 0.94.0 (main is 0.93.8)

Local gates

Gate	Result
cargo test --workspace --locked	35 binaries, 1535 passed, 0 failed
cargo clippy --workspace --all-targets --locked -- -D warnings	clean
cargo fmt --all -- --check	clean
Check Version Increment	green
Commitlint	green
Native install packages	green

Every new test was falsified — 11 mutations, each confirmed to turn its test RED, then reverted.
The harness asserts the mutated bytes actually differ on disk before trusting a kill, so a
replacement that silently no-ops cannot be reported as a pass. Full table in the PR body.

Two defects found and fixed while finishing the wiring

1. An uncomposable tier read as a failure. dig-nat composes UPnP only with a mapped local port,
   NAT-PMP/PCP only with an IPv4 gateway, hole-punch only with a reflexive address + coordinator —
   so on an ordinary node several rungs compose to nothing and returned NoMethodsEnabled. Reporting
   those as failures blames the PEER for THIS node's configuration and shows a perfectly reachable
   peer as four red rows: the "reads as the network is broken" outcome #1985 exists to prevent.
   TierOutcome::Unavailable is now a distinct outcome with its own wire token and no elapsed_ms.
2. dht::announce_inventory became dead once the announce left the bring-up path, and the one
   test still calling it was named startup_announce_publishes_every_held_capsule while exercising a
   wrapper startup no longer uses. Removed; the test now drives the pair bring-up actually runs.

⚠️ Version collision with PR #148

PR #148 is also at 0.94.0. Whichever merges second fails Check Version Increment and must
re-bump. Mine should go to 0.95.0 (minor — new capability + new public API), not 0.94.1, if #148
lands first. Flagging rather than pre-empting, since merge order is the gate's call.

What is NOT proven — stated plainly

#1974's "after" is a model, not a measurement, and the ticket's ORIGINAL root cause was wrong for
exactly that reason. The arithmetic (68 ids x ~11.2s = 761s vs the 760s observed) is a fit to the
reported observation. No restart has been measured end to end.

I could not close that here, for a specific reason rather than for lack of effort:
DhtService::announce_provider returns Ok(0) immediately when seed_contacts is empty
(dig-dht-0.9.0/src/service.rs:230), so a lone local node announces instantly and cannot reproduce
the pathology at all. It needs a routing table that is populated but slow to answer — what the
unit test's seeded SlowTransport models and what a real bootstrapped node on a sparse network has.
Reproducing it for real needs a multi-node fleet (the loop-e2e-p2p EC2 harness), not a local run.

The acceptance test is one restart of the S3-backed rpc.dig.net node plus a log read:
mTLS peer-RPC listening on [::]:9444 should appear within seconds of the relay reservation instead
of ~13 minutes after it, with the new DHT announcing N content id(s) / announced N of M /
announced N ... in Xs lines carrying the announce afterwards.

#1985's three acceptance criteria are unproven live (fleet node direct over IPv6, NAT'd peer
relayed-not-broken, wrong peer_id mismatch). What IS proven end to end is reachability — an
integration test drives control.peers.ping over the real HTTP control surface on a live node and
asserts it is token-gated, registered, routed to its own handler, and honest with no peer network.

Sibling work, flagged not dropped

• dig-app — #1985 asks for the ping-node option in the app; the node-side method exists now.
• docs.dig.net — control.peers.* is documented in docs/protocol/peer-network.md and
  docs/run-a-node/run-a-relay.md across 14 locales; the new verb belongs there (§4.3).

[MichaelTaylor3d]

Gate delta — all five findings addressed

HEAD 626d207 on fix/1974-bringup-visibility.

1. IdentityMismatch was unreachable in production — now surfaced from the real handshake error

This was the substantive one, and the gate's diagnosis was exactly right: dig-tls pins the expected
id in pin_and_bind, so a mismatched certificate aborts the handshake and no PeerConnection is
produced. verdict() only inspected connections, and a connection's observed_peer_id equals the pin
by construction — so peers ping <addr> --peer-id <wrong> reported unreachable.

The truth arrives only as a dial FAILURE, so classify_dial_error now recognises it:

TierFailure::IdentityMismatch → TierOutcome::IdentityMismatch → PingVerdict::IdentityMismatch

with the answering identity recovered from the error where disclosed. The classification never
depends on that recovery — a dig-tls wording change loses the detail (observed: None →
UNDISCLOSED_IDENTITY), never the verdict. There is no typed path (the mismatch never becomes
NatError::PeerIdentityMismatch on the dial route, and dig-tls's captured id is not exposed on the
error), so the marker string IS the contract; identity_mismatch_pin_matches_the_real_dig_tls_message
pins the exact wording so a dig-tls/dig-nat bump fails loudly instead of silently regressing. A
companion test proves an ordinary connection refused is NOT swallowed as a mismatch.

The hand-built test is deleted. It built a Connected report with a mismatched id — a shape the
real dialer cannot emit. The Connected-mismatch branch in verdict() stays, but now explicitly
labelled defence-in-depth over the dialer class (a future in-crate dialer that does not pin), never
as #1985's acceptance criterion.

The criterion is now proven over a REAL mTLS handshake, not a mock:
crates/dig-node-core/tests/peer_ping_identity.rs stands up a real serve_peer_rpc_listener on
loopback and drives ping::ping_peer against it — wrong pin → identity-mismatch naming the peer
that actually answered; right pin → direct on the direct rung.

2. The gate-free path to the dialer is closed

NatTierDialer, its new, and run_ladder are now pub(crate). The doc claim is true of the code.

3. The rate limit is no longer mutation-survivable

the_rate_limit_is_enforced_by_the_production_entry_point drives ping_peer MAX_PINGS_PER_WINDOW
times against a real peer and asserts the next call is RateLimited. Deleting
ctx.gate.try_enter(..) now fails it. A second test covers the previously unproven ordering property
(resolution before the gate — 12 unresolvable calls do not lock the caller out) and all four
unresolved_json branches.

4. "Writes nothing" narrowed

The module doc and SPEC §7.4a now say the UPnP rung is a port-MAPPING method that leaves a real ~2h
mapping on the operator's own router, once per ping — this node's NAT device, not network state, and
the ordinary dial does the same, but not "writes nothing".

5. Paired-token reachability stated

SPEC §7.4a records that control.peers.ping is NOT a pairing-admin method, so a paired controller
token drives it as well as the master token — deliberate, since dig-app is the intended consumer and
the method is read-shaped. It also records what the gate confirmed: never peer-reachable (absent from
dig_rpc_protocol::Method), never on the public-read allowlist.

Filed, not fixed here

dig_ecosystem#2078 — target restriction for control.peers.connect AND control.peers.ping,
covering both because connect is the worse of the two (arbitrary caller-supplied SocketAddr over
the full ladder with no rate limit at all). Priority Low; the gate's finding that this is a
connectivity oracle rather than SSRF-to-internal-API is recorded in the ticket so it is not
re-derived.

Commitlint

The interim commit was headed wip:, which would have redded the PR. Amended into a Conventional
Commit; Commitlint is green on 626d207.

Version

Still 0.94.0 against main's 0.93.8. #177 has not landed yet and #148 is also at 0.94.0 — re-check at
merge time; if either lands first this needs 0.95.0 (minor: new capability + new public API), not a
patch bump.

Falsification of the delta — 5/5, and the one the gate named

Broken on purpose	Test that went red
deleted ctx.gate.try_enter(..) from ping_peer	the_rate_limit_is_enforced_by_the_production_entry_point
removed the IdentityMismatch arm from verdict()	a_rung_that_reached_the_wrong_identity_grades_as_a_mismatch_not_unreachable
removed the mismatch branch from classify_dial_error	identity_mismatch_pin_matches_the_real_dig_tls_message
same, aimed at the REAL-WIRE test (proves it is non-vacuous)	a_wrong_peer_id_at_a_reachable_address_reports_an_identity_mismatch
dropped the WRONG-PEER row from the CLI renderer	a_wrong_peer_rung_is_called_out_by_name

The first row is the exact mutation the gate reported as surviving. It no longer does.
Running total for the PR: 16/16 mutations falsified, each verified to have changed the bytes on
disk before the kill was trusted.

Local results on this head

• cargo test -p dig-node-core --lib -> 697 passed, 0 failed (+8 from the delta)
• cargo test -p dig-node-core --test peer_ping_identity -> 4 passed, 0 failed (the real-wire file)
• cargo fmt --all -- --check clean; CI Rustfmt/Clippy/Release-script tests green on 626d207

An interim commit was headed wip:, which would have redded Commitlint. Amended; Commitlint green.