rpc/jsonrpc, node, execution/commitment: serve debug_executionWitness on minimal nodes via head-capture - #22663
Merged
Merged
Conversation
Pure refactor splitting the witness-building pipeline (buildAccessedState through the append-and-sort tail) into a shared DebugAPIImpl.buildWitnessResult method, so the on-demand handler and the upcoming eager cache builder produce byte-identical results. Adds a determinism-and-sort guard test.
Background worker that eagerly builds legacy-mode debug_executionWitness results into the shared witnessCache as canonical headers arrive, reusing the buildWitnessResult seam so cached bytes are byte-identical to on-demand. - shouldBuild: pure tip-gate (single-block advance to the freshest unbuilt tip) - decideCommittedHead / waitCommittedHead: Fork-1 safe-commit gate that polls the committed head via a fresh temporal RO tx per attempt, matching the hash before building and treating a mismatch as a reorg-away - RunWitnessCacheBuilder: coalesce-to-latest loop that reconciles the cache on every batch and builds only the newest tip-gated header
…uilder Add witness.cache.blocks (default 0, capped 96) and witness.cache.maxmb (default 1024) flags, thread them through HttpCfg, and wire the eager witness-cache builder into the embedded node. The cache is embedded-RPC only and gated on the DB-persisted commitment-history flag; standalone rpcdaemon and mcp pass nil. APIList gains the shared *witnessCache param; the builder-owned DebugAPIImpl shares that same pointer.
rpc/jsonrpc: add witness_cache_* counters (hit/miss, build_ok, build_fail_verify/other, evict, coalesce_drop), bytes/entries_resident gauges, and a build_duration histogram, wired at the serve, build, and eviction sites. Classify verify failures via an errWitnessVerifyFailed sentinel wrapped around the shared build seam. Enrich the witness cache flag help text with the raw-size cap context.
rpc/jsonrpc: builder resolves the block by the canonical hash it validated, not by number, so the built witness is provably keyed under (num, hash). Tests: fix canonical-bypass subtest that could never fail (discarded error + nil-safe assertion); assert hit/miss counter deltas on serve; cover newWitnessCache clamp/boundary and decodeHeaderRefs/processHeaderBatch newest-selection; join the builder goroutine before DB teardown in builder tests to avoid a use-after-close race under -race. docs/plans: correct RunWitnessCacheBuilder signature (drop notifications arg).
Store the marshaled JSON once in the builder and serve a cache hit through MarshalFastJSON, skipping the per-hit struct marshal (~26ms for a 15MB witness -> ~11ns, zero allocs). Byte-identical to the on-demand response.
Replace the hand-rolled number-keyed witness cache with the same hashicorp lru.Cache used for the block cache, keyed by block hash. Hash keying makes reorgs self-evicting — a reorged hash is never requested again and ages out — so the reconcile path is gone. Memory is bounded by the block count; drops the byte cap and --witness.cache.maxmb.
Gate eager building on whether the tip's hash is already cached rather than a high-water block number, so a reorged head (a new hash at an already-built height) is rebuilt instead of falling through to on-demand forever. Removes the frozen high-water var.
…ain-history reader NewHeadCaptureStateReader composes a LatestStateReader bound to a pinned parent tx (commitment domain only) with a HistoryStateReader bound to a separate committed tx at a caller-supplied plainStateAsOf txNum (account/storage/code) — the dual-tx variant of CommitmentReplayStateReader a minimal node needs to source parent commitment from a pinned RO snapshot while reading plain state from the history it retains. withHistory=false so the build's own SharedDomains routes PutBranch to its throwaway in-memory batch (discarded on Close, never flushed).
…wap commitment reader)
…e for head-capture witnesses Adds the head-capture builder path: a rolling RO pin lagging the tip by one committed block supplies parent(B) commitment-latest, combined with the committed >=B tx (from waitCommittedHead) for plain history. decidePin gates the pin as a canonical parent (Finish==B-1, hash still canonical); a stale pin (tip jump, reorg, or none yet) skips B to out-of-window and re-pins. buildAndCache routes through the head-capture build in head-capture mode. Reads the pinned parent commitment plane directly via tx.GetLatest instead of through a SharedDomains getter. The pinned SharedDomains shared aggregator-scoped cache state with the build's own commitment fold, so parent-commitment reads were contaminated by post-state branches on storage-heavy / account-creating blocks (root mismatch / malformed witness nodes). Direct snapshot reads isolate the parent plane; this also drops the extra SharedDomains. The trie-phase reader now reports WithHistory()==true so witness-capture PutBranch no-ops, matching durable.
…itment history Decouple the eager witness cache from the DB commitment-history flag: a minimal node with --witness.cache.head-capture set now enables the cache in cache-only, pinned-parent mode. WitnessCacheShouldEnable gains a headCapture arg and a new WitnessCacheMode resolves (enable, headCapture); commitment history present keeps the durable recompute path even when the flag is on.
…ash canonical guard Head-capture minimal nodes now serve debug_executionWitness / eth_getWitness strictly from the cache: a miss returns a typed out-of-window error and never falls through to a history recompute. serveFromWitnessCache re-checks the canonical hash on by-hash resolves so a still-resident orphan is not served as canonical, and reports the reorged-away bucket distinctly. The witness cache reference moves to BaseAPI so the eth and debug serve paths read one mode source.
…ead-capture tests
Add a "Head-capture serving (minimal nodes)" section to the debug_executionWitness spec covering the two new flags (--witness.cache.head-capture, --witness.cache.maxmb) and the cache-only/tip-only/by-hash-canonical/cold-after-restart semantics. Correct the now-conditional "requires commitment history" statements and refresh stale code-map symbols (toWitnessTrie/GenerateWitness/ witnessCreateAccountNode removed by the fold refactor). Move the completed head-capture plan to docs/plans/completed/.
- debug_executionWitness: canonical-mode request on a cache-only node now returns a distinct errWitnessCanonicalUnavailable instead of the misleading out-of-window error (the eager cache only builds legacy witnesses); covered by a new cache-only serve subtest. - witness_cache_builder: drop the dead pin-drift Finish re-read in tryHeadCaptureBuild — the pin holds a fixed RO snapshot whose Finish cannot drift (proven by TestRollingPinStableUnderTipAdvance), so the guard was unreachable and its comment misleading. - commitmentdb: remove the unused SharedDomainsGetter type alias (the head-capture reader reads via tx.GetLatest, no SharedDomains handle). - flags: --witness.cache.blocks usage no longer claims commitment history is required; head-capture provides the minimal-node cache-only path. - ChangeLog: add the v3.6.0 witness cache / head-capture entry.
Lint (after the db/services -> db/dbservices rename in the main merge): gofmt witness_cache_builder import ordering, modernize the clamp test loop to range-over-int. Metric fidelity (code-review): - record the build-duration histogram only on successful builds, not failures - don't count a cache miss when the request's block can't be resolved
ExecutionWitnessResult has no MarshalJSON, so encoding/json never dispatches to MarshalFastJSON — there is no recursion for executionWitnessResultView to guard against. Marshal the pointer directly, avoiding both the view type and the per-call struct copy.
Resolve APIList conflict from #22680 (rpc: extract api config pkg more): adopt main's config-object constructors NewPrivateDebugAPI(...DebugApiConfig) and NewTraceAPI(...TraceApiConfig), keeping debugImpl.witnessCache wiring. Update the witness builder + its tests to the new NewPrivateDebugAPI signature.
- clamp --witness.cache.maxmb to bytes to avoid int overflow (32-bit / large values) - make cache-only witness errors endpoint-agnostic (returned from eth_getWitness too) - drop absolute worktree path from the completed plan doc
…apture # Conflicts: # node/cli/flags.go # node/eth/backend.go # rpc/jsonrpc/daemon.go # rpc/jsonrpc/debug_execution_witness.go # rpc/jsonrpc/eth_api.go # rpc/jsonrpc/witness_cache.go # rpc/jsonrpc/witness_cache_builder.go # rpc/jsonrpc/witness_cache_builder_test.go # rpc/jsonrpc/witness_cache_test.go
tryHeadCaptureBuild called ObserveDuration both before the error check and again after it, so every successful build recorded two samples. Observed on a minimal mainnet node: 178 samples for 89 builds. Keep the success-only observation, matching the durable path.
WitnessNodes returns the output of WitnessNodesForKeysFromNodes, whose only append site stores a freshly cloned slice, so buildWitnessTrie re-cloning every node copied heap into heap. Measured on a minimal mainnet node: 10.2GB and 29M objects over ~15min, ~4.7MB per witness build.
awskii
marked this pull request as ready for review
July 29, 2026 09:56
awskii
requested review from
AskAlexSharov,
bloxster,
lupin012,
mh0lt,
taratorio and
yperbasis
as code owners
July 29, 2026 09:56
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 30 out of 30 changed files in this pull request and generated no new comments.
Comments suppressed due to low confidence (1)
rpc/jsonrpc/witness_cache.go:122
residentBytesaccounting usesintand is updated viac.residentBytes += size. On 32-bit builds (which the maxMB→bytes clamp explicitly calls out), a cache holding many large witnesses can overflowresidentBytes(e.g., 96×~25MB > 2GB), makingResidentBytes()/byte-cap eviction logic incorrect. Use a wider type (int64) or saturating arithmetic when adding sizes to prevent overflow.
if existed {
c.residentBytes -= prev
}
c.entryBytes[hash] = size
c.residentBytes += size
…l-headcapture # Conflicts: # cmd/rpcdaemon/cli/config.go # cmd/rpcdaemon/cli/httpcfg/http_cfg.go # cmd/utils/flags.go # node/cli/default_flags.go # node/cli/flags.go # node/eth/backend.go # rpc/jsonrpc/debug_api_test.go # rpc/jsonrpc/debug_execution_witness.go # rpc/jsonrpc/witness_cache.go # rpc/jsonrpc/witness_cache_builder.go # rpc/jsonrpc/witness_cache_builder_test.go # rpc/jsonrpc/witness_cache_metrics.go # rpc/jsonrpc/witness_cache_test.go # rpc/jsonrpc/witness_cache_wiring_test.go
AskAlexSharov
approved these changes
Jul 31, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A
--prune.mode=minimalnode keeps account/storage/code history but no commitment history, so the eager witness cache anddebug_executionWitnessare both hard-gated off — a minimal node can't serve execution witnesses at all.This adds a head-capture mode that builds each head block's witness with commitment parent state read from an RO snapshot pinned at the parent (where commitment-latest already equals the parent trie) and plain state from the history a minimal node keeps. No commitment history is required or produced.
Changes
--witness.cache.head-capture: build the last-N head witnesses against a rolling one-block-lag pinned parent snapshot, using a dual-tx reader (commitment-latest from the pinned parent tx and plain history from the committed tx). No code on the consensus commitment path.witnessRoot==parent,computedRoot==block, and stateless verify gate every cache insert.--witness.cache.maxmb).Tip-only and cache-only by design: after restart the cache is empty and re-warms forward, so the last ~N blocks are out-of-window until N new blocks pass.
Stacked on #22384.