Skip to content

feat(dig-node): serve getProof/getMetadata/getPublicManifest/getCapsule locally - #179

Open
MichaelTaylor3d wants to merge 11 commits into
mainfrom
feat/2071-node-read-method-sweep
Open

feat(dig-node): serve getProof/getMetadata/getPublicManifest/getCapsule locally#179
MichaelTaylor3d wants to merge 11 commits into
mainfrom
feat/2071-node-read-method-sweep

Conversation

@MichaelTaylor3d

@MichaelTaylor3d MichaelTaylor3d commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

Task

Serve the read methods dig-node classified as "passthrough aliases" — relayed to an upstream, and
therefore -32601 on any node that has none. rpc.dig.net is meant to be an ordinary node (#1997),
so it has none, and every client calling these got method-not-found from a node that held the bytes.

Second half of DIG-Network/dig_ecosystem#2071. #177 is merged (95141b7) and this is now rebased
directly onto main
— no longer stacked.

Scope note, since this was briefly mis-sequenced: #177 is what restores *.on.dig.net. This PR
does not, and never could — the deployed resolver's only RPC call is dig.getContent
(grep -o 'rpcCall("[^"]*"' sw.js dig-embed.js returns exactly that one). This PR matters because
rpc.dig.net is an ordinary node and OTHER clients — the hub, the extension, the CLI — do call these.

Methods now served locally

method result
dig.getProof { inclusion_proof, root, chunk_lens, program_hash, execution_proof: null, execution_proof_status: "unavailable" }
dig.getMetadata { manifest | null, program_hash, root } — publisher metadata, data-section id 6
dig.getPublicManifest { manifest | null, root } — the enveloped form of dig.getManifest
dig.getCapsule / dig.getModule one window of the whole .dig module, in the getContent envelope

dig.getProof is served genuinely, not stubbed

The constraint on #2071 is explicit: a proof that does not verify is worse than -32601,
because the client would then decrypt unverified bytes with no error raised anywhere.

So dig.getProof does not derive a proof of its own. It runs the ordinary dig.getContent read
— same mandatory anchored-root pin (§14.4), same local-first → peer → upstream ladder — and discards
the ciphertext. That makes the proof it returns provably the proof a content read of the same
resource would have verified against; a second derivation could pin a different generation and no
client could tell. The proof itself is computed by the module's own guest wasm inside serve_blind.

When no proof can be produced, the underlying read's error is returned verbatim — never a
proof-shaped result with an empty inclusion_proof.

The test decodes the returned proof and checks it, rather than asserting it is non-empty:

assert!(proof.verify(), "the merkle path resolves to its declared root");
assert_eq!(proof.root.to_hex(), root.to_hex(),
    "the proof is rooted at the CHAIN-anchored root, not some other generation");

The old guard was rewritten, not deleted

get_proof_is_not_served_as_a_verified_proof_by_the_node asserted dig.getProof must be -32601.
"Never fabricate a proof" was always the invariant; "never implement the method" was only the
cheapest way to hold it, and it cost every client the ability to re-verify bytes it already held.
The guard now asserts the invariant directly: an unobtainable proof is an error, never a blank
that looks like a proof. No execution attestation is fabricated — execution_proof: null with
execution_proof_status: "unavailable" (#126/#134, SECURITY.md residual #3).

Security property found while reviewing this, now recorded in the code

dig.getPublicManifest is dispatched by NAME (it is not yet a dig_rpc_protocol::Method variant).
That absence from the catalogue is what keeps it off the permissionless peer surface
is_peer_reachable_method ends in Method::from_name(m).is_some_and(|m| m.is_peer_reachable()), so
an unknown name is filtered before the peer surface reaches this dispatch. It therefore serves the
loopback / in-process / gateway surface only.

Promoting the method into dig-rpc-protocol later MUST make a deliberate is_peer_reachable()
decision rather than inherit one, or a gateway-only public read silently widens to the whole peer
network. That reasoning is in dispatch.rs, not just here.

Anti-rollback applies to capsule-scoped reads too

root may be omitted or "latest", which every client uses to mean "whatever the chain says" — the
node resolves the tip itself rather than making the caller walk the singleton. An explicitly
requested root MUST equal the anchored tip or the read fails closed with -32005, with the same
#747/#841 bounded-verify tolerance the content read has. Tested across all three capsule-scoped
methods.

Still NOT served, deliberately

  • dig.listCapsules — needs a chain generation walk this node does not perform.
  • dig.getProofStatus — polls an execution-proof JOB this node does not run; inventing a status
    would be the fabrication above.

Both stay -32601 and the passthrough guard now pins exactly those two, with the reasoning recorded
in the test.

Bump

0.93.9 → 0.94.0 (minor — new served capability on the public read tier); dig-node-core
0.41.1 → 0.42.0.

Refs DIG-Network/dig_ecosystem#2071

@MichaelTaylor3d
MichaelTaylor3d force-pushed the feat/2071-node-read-method-sweep branch from 16c4e23 to df07cd9 Compare August 3, 2026 23:39
@MichaelTaylor3d
MichaelTaylor3d marked this pull request as ready for review August 3, 2026 23:39
Base automatically changed from fix/2071-node-read-methods to main August 3, 2026 23:54
@MichaelTaylor3d
MichaelTaylor3d force-pushed the feat/2071-node-read-method-sweep branch 2 times, most recently from 6052a87 to 1212c92 Compare August 4, 2026 00:24

@MichaelTaylor3d MichaelTaylor3d left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: CHANGES-REQUIRED (recorded as a comment review — this token shares the PR author identity, so REQUEST_CHANGES is a self-review 422; the blocking findings are the open inline threads below.)

CHANGES-REQUIRED — reviewed at 1212c92e5cecda9d0dbe6ad2405a7e9908db4b09

Independent correctness gate. Own worktree (C:\tmp\worktrees\dn179g, private CARGO_TARGET_DIR); the shared primary checkout was not touched. Full suite green at head: 676 passed / 0 failed (cargo test -p dig-node-core --lib, 253s). All 15 required checks green.

The implementation is good and the PR body is honest — it explicitly disclaims the *.on.dig.net justification, which I independently agree with. Three things block.

Blocking

  1. lib.rs:2299 — the anti-fabrication guard has no test. Mutating if proof.is_empty() to if false leaves the entire 676-test suite green. The named guard get_proof_errors_rather_than_fabricating_a_proof_it_cannot_produce never reaches it (guard-not-exercised).
  2. SPEC.md:887-888 — untouched, and now normatively wrong. Still classifies dig.getCapsule and dig.getProof as passthrough; dig.getMetadata / dig.getPublicManifest are absent from the catalogue entirely. SPEC.md is the contract an independent reimplementation is built from (§4.2), so it is a contract split, not a doc nit.
  3. lib.rs:2467 / meta.rs:88 — doc asserts a shape the code does not produce. Both say a capsule window carries no inclusion_proof; it always carries "", and the PR's own test asserts exactly that.

Non-blocking (posted inline, resolved by me)

  1. peer.rs:1156 — the load-bearing peer-reachability reasoning has no assertion.
  2. lib.rs:9357 — the anti-rollback test does not discriminate the arm it names.

Confirmed sound — do not re-litigate

  • dig.getProof is client-verifiable. Probe against a 3-file fixture: proof.leaf == SHA-256(reassembled ciphertext) (eadd3580a2cb…), path.len() == 2 (non-trivial path), proof.verify() true, proof.root == the chain-anchored root. A DigstoreProofVerifier client can verify what this returns against a chain-anchored root.
  • Both three-window fixtures survived the rebases (lib.rs:8603, lib.rs:8701, both 2 * WINDOW + 500, middle window asserted). Mutating content_window_envelope's clamp to if start == 0 { (start + WINDOW).min(total) } else { total } fails a_client_can_reassemble_a_multi_window_resource_from_the_envelope_alone.
  • The two-window fixture at lib.rs:8569 is fine — keep it. Ruling on the judgement call you surfaced: your reasoning holds. The middle-window placement property is discriminated by the reassembly test above (windows_seen == 3), proven by that same mutation, so uniformity buys nothing here.
  • dispatch.rs:88-89 reasoning verified in the tree, not taken on trust: peer.rs:1156 does terminate in Method::from_name(m).is_some_and(|m| m.is_peer_reachable()), and dig-rpc-protocol 0.6.0 method.rs:218-233 excludes GetProof/GetMetadata/GetCapsule/GetModule. No peer-surface widening by any of the five new methods.
  • Redirects are handled correctly. A redirect is an error.data.redirect envelope, so it takes get_proof's passthrough branch as the rustdoc claims — it is not silently flattened to -32004 with the provider list discarded.
  • Two further mutations killed by their own named assertions: pinning offset = 0 in get_capsule fails at lib.rs:9309 ("a window must be served at the offset requested"); reverting start == 0 && !chunk_lens.is_null() to start == 0 fails at lib.rs:9302 (the chunk_lens-omitted assertion). Every mutation was md5-verified as a real on-disk byte change before its run.
  • Advertised-vs-implemented now matches for all five new names; dig.getProofStatus is correctly absent from the catalogue and dig.listCapsules correctly passthrough.

dig-constants check (mandatory, both directions)

Nothing in this diff belongs in dig-constants, and nothing here should be consuming it. The values introduced are JSON-RPC method-name strings and error codes already sourced from dig_rpc_protocol::Method and the crate's existing ROOT_NOT_ANCHORED / download::RESOURCE_UNAVAILABLE constants — no new cross-repo literal, address, port, asset id, or version pin is defined locally. WINDOW is pre-existing and out of scope.

Comment thread crates/dig-node-core/src/lib.rs Outdated
Comment thread crates/dig-node-core/src/lib.rs Outdated
Comment thread crates/dig-node-service/src/meta.rs Outdated
Comment thread crates/dig-node-core/src/seams/dig_rpc/dispatch.rs
Comment thread crates/dig-node-core/src/lib.rs
Comment thread crates/dig-node-core/src/seams/dig_rpc/dispatch.rs
@MichaelTaylor3d
MichaelTaylor3d force-pushed the feat/2071-node-read-method-sweep branch from 13ef6de to 995b5d7 Compare August 4, 2026 02:55
MichaelTaylor3d added a commit that referenced this pull request Aug 4, 2026
…-writable hint

The melt gate is the authority for an irreversible, peer-triggered, network-correlated
delete. Two cheaper signals were tried for it and both were unsound; this replaces the
second with the singleton lineage itself, and the choice is settled by measurement
against mainnet rather than by argument.

What was wrong
--------------
The previous cut concluded "melted" from a NON-EMPTY, all-spent `store_id` hint index.
A hint is an unauthenticated CREATE_COIN memo over an arbitrary 32-byte value (#1473),
so ANY party can place a record under ANY store's hint for the price of a dust coin.
Enumerating all 53 DataLayer launcher coins on mainnet shows why that is fatal: 30 of
the 53 LIVE stores have a completely EMPTY store_id hint index — their generations are
not hinted to store_id at all. For every one of them a single planted spent coin makes
the index non-empty and entirely spent, which the gate could not distinguish from a
terminated lineage. Cost to erase a live store network-wide: dust plus fee, no
permission, no P2P access, no key material. `run_melt_tick` would have fired it on a
timer with no announcement at all. `get_coin_records_by_hint` is also truncatable, and
truncation surfaces spent records first — the exact order that manufactures a false
melt.

What replaces it
----------------
A forward walk of the singleton lineage along real COIN PARENTAGE:

1. Identity + minted — the launcher coin whose `coin_id == store_id` exists and is
   SPENT. An unspent launcher is Live (not minted yet is the opposite of melted). This
   fact discriminates nothing by itself; it anchors where the walk starts.
2. Walk forward — follow the single ODD-amount child at each hop. An UNSPENT successor
   is Live. A spent coin with NO successor is Melted.

A coin's `parent_coin_info` is fixed by which coin was actually spent to create it, so
placing a coin anywhere in this walk requires spending a generation of the store, which
requires the owner's authority. The walk is unwritable by anyone but the owner, and it
never consults a hint — the mock panics if either hint query is touched.

Fail-closed everywhere else: any transport error INCLUDING mid-walk (an outage must not
read as "the lineage ended here"), more than one odd child, an absent launcher, and
exceeding the hop ceiling. Zero children at hop 0 is Unknown, not a melt: a minted
launcher always created the eve singleton, so an empty first hop means the answer is
untrustworthy — which also closes the trap that `coin_records_by_parent_ids` has an
empty DEFAULT impl on the trait.

Measured against mainnet
------------------------
All 53 DataLayer stores: 51 Live, 1 Melted (the one genuinely terminated store, ending
at hop 1), 1 previously mis-capped. Deepest live lineage 599 generations; 29 stores have
their tip one hop from the launcher; no ambiguous fork anywhere. MAX_LINEAGE_HOPS is
sized from that measurement. The four stores the gate named as live-with-empty-hint-index
all classify Live here.

Because the walk costs one read per generation and the receive path runs per inbound
announcement, verdicts are memoised for a short TTL so a flood of announcements for one
held store cannot multiply into repeated walks. A stale verdict can only DELAY a real
melt, never cause a delete.

Tests: 12 cases drive the real ChainReads trait with a crafted lineage, including the
composition the gate flagged as untested and lethal — an empty hint index plus one
planted spent coin — asserting Live. All ten inverting mutations of the gate were
confirmed to fail their test; the hop-cap test asserts the EXACT read count, because a
`<=` bound is also satisfied by a walk that stops far too early.

root [workspace.package].version 0.94.0 -> 0.96.0 (minor, new capability). Skips
0.95.0, which PR #179 holds.

Co-Authored-By: Claude <noreply@anthropic.com>
@MichaelTaylor3d
MichaelTaylor3d force-pushed the feat/2071-node-read-method-sweep branch from 07eb9d2 to afec8b5 Compare August 4, 2026 03:24
MichaelTaylor3d added a commit that referenced this pull request Aug 4, 2026
…-writable hint

The melt gate is the authority for an irreversible, peer-triggered, network-correlated
delete. Two cheaper signals were tried for it and both were unsound; this replaces the
second with the singleton lineage itself, and the choice is settled by measurement
against mainnet rather than by argument.

What was wrong
--------------
The previous cut concluded "melted" from a NON-EMPTY, all-spent `store_id` hint index.
A hint is an unauthenticated CREATE_COIN memo over an arbitrary 32-byte value (#1473),
so ANY party can place a record under ANY store's hint for the price of a dust coin.
Enumerating all 53 DataLayer launcher coins on mainnet shows why that is fatal: 30 of
the 53 LIVE stores have a completely EMPTY store_id hint index — their generations are
not hinted to store_id at all. For every one of them a single planted spent coin makes
the index non-empty and entirely spent, which the gate could not distinguish from a
terminated lineage. Cost to erase a live store network-wide: dust plus fee, no
permission, no P2P access, no key material. `run_melt_tick` would have fired it on a
timer with no announcement at all. `get_coin_records_by_hint` is also truncatable, and
truncation surfaces spent records first — the exact order that manufactures a false
melt.

What replaces it
----------------
A forward walk of the singleton lineage along real COIN PARENTAGE:

1. Identity + minted — the launcher coin whose `coin_id == store_id` exists and is
   SPENT. An unspent launcher is Live (not minted yet is the opposite of melted). This
   fact discriminates nothing by itself; it anchors where the walk starts.
2. Walk forward — follow the single ODD-amount child at each hop. An UNSPENT successor
   is Live. A spent coin with NO successor is Melted.

A coin's `parent_coin_info` is fixed by which coin was actually spent to create it, so
placing a coin anywhere in this walk requires spending a generation of the store, which
requires the owner's authority. The walk is unwritable by anyone but the owner, and it
never consults a hint — the mock panics if either hint query is touched.

Fail-closed everywhere else: any transport error INCLUDING mid-walk (an outage must not
read as "the lineage ended here"), more than one odd child, an absent launcher, and
exceeding the hop ceiling. Zero children at hop 0 is Unknown, not a melt: a minted
launcher always created the eve singleton, so an empty first hop means the answer is
untrustworthy — which also closes the trap that `coin_records_by_parent_ids` has an
empty DEFAULT impl on the trait.

Measured against mainnet
------------------------
All 53 DataLayer stores: 51 Live, 1 Melted (the one genuinely terminated store, ending
at hop 1), 1 previously mis-capped. Deepest live lineage 599 generations; 29 stores have
their tip one hop from the launcher; no ambiguous fork anywhere. MAX_LINEAGE_HOPS is
sized from that measurement. The four stores the gate named as live-with-empty-hint-index
all classify Live here.

Because the walk costs one read per generation and the receive path runs per inbound
announcement, verdicts are memoised for a short TTL so a flood of announcements for one
held store cannot multiply into repeated walks. A stale verdict can only DELAY a real
melt, never cause a delete.

Tests: 12 cases drive the real ChainReads trait with a crafted lineage, including the
composition the gate flagged as untested and lethal — an empty hint index plus one
planted spent coin — asserting Live. All ten inverting mutations of the gate were
confirmed to fail their test; the hop-cap test asserts the EXACT read count, because a
`<=` bound is also satisfied by a walk that stops far too early.

root [workspace.package].version 0.94.0 -> 0.96.0 (minor, new capability). Skips
0.95.0, which PR #179 holds.

Co-Authored-By: Claude <noreply@anthropic.com>
MichaelTaylor3d and others added 11 commits August 3, 2026 21:52
…le locally

WIP for the #2071 method sweep — parked while PR #177 (the P0 envelope fix) is re-gated.

Co-Authored-By: Claude <noreply@anthropic.com>
getCapsule/getModule joined getProof/getMetadata/getPublicManifest as served=local, so the guard now pins what genuinely REMAINS unserved — listCapsules (needs a chain generation walk) and getProofStatus (polls an execution-proof job this node does not run) — and records why each is honestly unserved rather than merely unwritten.

Co-Authored-By: Claude <noreply@anthropic.com>
Rebased onto the #177 re-gate fixes. inclusion_proof is now always emitted (as "" when absent), so the capsule test's absence assertion becomes an empty-string assertion: a capsule window has no per-resource proof and says so explicitly rather than by omission.

chunk_lens keeps its omission, and the guard returns with the caller that motivated it. The two are deliberately different: empty-vs-absent distinguishes two states of an APPLICABLE field (inclusion_proof), while chunk_lens is INAPPLICABLE to a whole module and absent says exactly that.

Co-Authored-By: Claude <noreply@anthropic.com>
Minor, not patch: getProof/getMetadata/getPublicManifest/getCapsule/getModule are new served capability on the public read tier.

Co-Authored-By: Claude <noreply@anthropic.com>
…er-reachable

Absence from the Method catalogue also filters it off the permissionless peer surface (is_peer_reachable_method ends in Method::from_name(..).is_some_and(..)), so this arm serves loopback/in-process/gateway only. Promoting the method into dig-rpc-protocol later must decide is_peer_reachable() deliberately rather than inherit it.

Co-Authored-By: Claude <noreply@anthropic.com>
Security gate B1/N1/N3/N4. WIP checkpoint — see the follow-up commit for tests.

Co-Authored-By: Claude <noreply@anthropic.com>
… dropped program_hash

SPEC.md still listed getCapsule/getProof as passthrough and omitted getMetadata/getPublicManifest; the capsule-window docs in lib.rs and meta.rs still claimed no inclusion_proof rides a capsule window, which content_window_envelope contradicts (it is present, empty). Bumped to 0.95.0 — 0.94.0 was taken by #149.

Co-Authored-By: Claude <noreply@anthropic.com>
…sifiable

The empty-proof guard was a false green: the only test aiming at it made the INNER read fail, which returns at the passthrough branch one level above, so 'if proof.is_empty()' -> 'if false' left all 676 tests passing. The reduction is now a pure function (proof_from_content_answer) driven directly with a SUCCESSFUL read that carries no proof - the one shape where a blank could be dressed as a result.

getCapsule gains a bytes-READ guard. Both implementations return byte-identical responses, so no correctness assertion can distinguish a windowed read from a whole-module slurp; the amplification IS the defect. A far-past-EOF request must read ~0, not 128 MiB to discover it has nothing to send.

Also pins dig.getPublicManifest as non-peer-reachable, so promoting it into dig-rpc-protocol has to be a deliberate is_peer_reachable() decision.

Co-Authored-By: Claude <noreply@anthropic.com>
…laky

A single global counter passed in isolation and FAILED in the full suite: tests run in parallel and the read happens on a spawn_blocking thread, so neither a global nor a thread-local can attribute a read to the test that caused it. Per-root attribution can. A flaky guard is worse than none - it gets deleted.

Co-Authored-By: Claude <noreply@anthropic.com>
…oc list lint

The memoization commit inserted DATA_SECTION_MEMO_CAP between read_public_manifest_blocking's doc comment and its function, so the doc was orphaned onto the const and the function had none. Clippy caught it as doc_lazy_continuation - the const's prose was being parsed as an unindented continuation of the function's Returns list.

Fixed structurally rather than by indenting: the doc goes back on its function (updated to note the read is now memoized), and the const gets its own.

Co-Authored-By: Claude <noreply@anthropic.com>
…ame from

The blob memo was strictly worse than the whole-file read it replaced. FIXED_BLOB_LEN pads every DIGS blob to 128 MiB and ChunkPool is a section INSIDE it, so an entry was ~99% content, not metadata - my doc claimed the opposite. Capped in ENTRIES at 256, that bounds at ~32 GiB, and it is RETAINED for the life of the process where the old cost was transient and needed concurrency. cache.listCached is requires_auth:false, so an attacker enumerates the exact capsule list and pins ~16 of them to exhaust a 2 GiB host.

Now caches the two DECODED manifests, which genuinely are kilobytes, so an entry cap is a sound bound. Cold extractions are serialized by a single-flight lock: it coalesces duplicate work (N concurrent requests for one capsule cost one read) and bounds transient memory to one extraction's ~385 MiB rather than N of them.

Residency is invisible to every functional assertion - a blob cache returns byte-identical manifests - so a size guard was added. Falsified by re-adding the blob to the entry: it reports 134218536 bytes retained and fails.

Co-Authored-By: Claude <noreply@anthropic.com>
@MichaelTaylor3d
MichaelTaylor3d force-pushed the feat/2071-node-read-method-sweep branch from 9030c04 to 39dea76 Compare August 4, 2026 04:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant