Skip to content

rpc/jsonrpc, node, execution/commitment: serve debug_executionWitness on minimal nodes via head-capture - #22663

Merged
AskAlexSharov merged 43 commits into
mainfrom
awskii/witness-minimal-headcapture
Jul 31, 2026
Merged

rpc/jsonrpc, node, execution/commitment: serve debug_executionWitness on minimal nodes via head-capture#22663
AskAlexSharov merged 43 commits into
mainfrom
awskii/witness-minimal-headcapture

Conversation

@awskii

@awskii awskii commented Jul 22, 2026

Copy link
Copy Markdown
Member

A --prune.mode=minimal node keeps account/storage/code history but no commitment history, so the eager witness cache and debug_executionWitness are both hard-gated off — a minimal node can't serve execution witnesses at all.

This adds a head-capture mode that builds each head block's witness with commitment parent state read from an RO snapshot pinned at the parent (where commitment-latest already equals the parent trie) and plain state from the history a minimal node keeps. No commitment history is required or produced.

Changes

  • --witness.cache.head-capture: build the last-N head witnesses against a rolling one-block-lag pinned parent snapshot, using a dual-tx reader (commitment-latest from the pinned parent tx and plain history from the committed tx). No code on the consensus commitment path.
  • Serve cache-only: typed out-of-window on miss (never a history recompute); by-hash requests are canonical-checked so a reorged-out orphan is never served as canonical.
  • Fail-closed: witnessRoot==parent, computedRoot==block, and stateless verify gate every cache insert.
  • Cache is now a struct with a resident-bytes cap (--witness.cache.maxmb).

Tip-only and cache-only by design: after restart the cache is empty and re-warms forward, so the last ~N blocks are out-of-window until N new blocks pass.

Stacked on #22384.

awskii added 30 commits July 10, 2026 18:45
Pure refactor splitting the witness-building pipeline (buildAccessedState
through the append-and-sort tail) into a shared DebugAPIImpl.buildWitnessResult
method, so the on-demand handler and the upcoming eager cache builder produce
byte-identical results. Adds a determinism-and-sort guard test.
Background worker that eagerly builds legacy-mode debug_executionWitness
results into the shared witnessCache as canonical headers arrive, reusing
the buildWitnessResult seam so cached bytes are byte-identical to on-demand.

- shouldBuild: pure tip-gate (single-block advance to the freshest unbuilt tip)
- decideCommittedHead / waitCommittedHead: Fork-1 safe-commit gate that polls
  the committed head via a fresh temporal RO tx per attempt, matching the hash
  before building and treating a mismatch as a reorg-away
- RunWitnessCacheBuilder: coalesce-to-latest loop that reconciles the cache on
  every batch and builds only the newest tip-gated header
…uilder

Add witness.cache.blocks (default 0, capped 96) and witness.cache.maxmb
(default 1024) flags, thread them through HttpCfg, and wire the eager
witness-cache builder into the embedded node. The cache is embedded-RPC only
and gated on the DB-persisted commitment-history flag; standalone rpcdaemon and
mcp pass nil. APIList gains the shared *witnessCache param; the builder-owned
DebugAPIImpl shares that same pointer.
rpc/jsonrpc: add witness_cache_* counters (hit/miss, build_ok,
build_fail_verify/other, evict, coalesce_drop), bytes/entries_resident
gauges, and a build_duration histogram, wired at the serve, build, and
eviction sites. Classify verify failures via an errWitnessVerifyFailed
sentinel wrapped around the shared build seam. Enrich the witness cache
flag help text with the raw-size cap context.
rpc/jsonrpc: builder resolves the block by the canonical hash it validated,
not by number, so the built witness is provably keyed under (num, hash).

Tests: fix canonical-bypass subtest that could never fail (discarded error +
nil-safe assertion); assert hit/miss counter deltas on serve; cover
newWitnessCache clamp/boundary and decodeHeaderRefs/processHeaderBatch
newest-selection; join the builder goroutine before DB teardown in builder
tests to avoid a use-after-close race under -race.

docs/plans: correct RunWitnessCacheBuilder signature (drop notifications arg).
Store the marshaled JSON once in the builder and serve a cache hit through
MarshalFastJSON, skipping the per-hit struct marshal (~26ms for a 15MB witness
-> ~11ns, zero allocs). Byte-identical to the on-demand response.
Replace the hand-rolled number-keyed witness cache with the same
hashicorp lru.Cache used for the block cache, keyed by block hash.
Hash keying makes reorgs self-evicting — a reorged hash is never
requested again and ages out — so the reconcile path is gone. Memory
is bounded by the block count; drops the byte cap and --witness.cache.maxmb.
Gate eager building on whether the tip's hash is already cached rather than
a high-water block number, so a reorged head (a new hash at an already-built
height) is rebuilt instead of falling through to on-demand forever. Removes
the frozen high-water var.
…ain-history reader

NewHeadCaptureStateReader composes a LatestStateReader bound to a pinned parent
tx (commitment domain only) with a HistoryStateReader bound to a separate
committed tx at a caller-supplied plainStateAsOf txNum (account/storage/code) —
the dual-tx variant of CommitmentReplayStateReader a minimal node needs to source
parent commitment from a pinned RO snapshot while reading plain state from the
history it retains. withHistory=false so the build's own SharedDomains routes
PutBranch to its throwaway in-memory batch (discarded on Close, never flushed).
…e for head-capture witnesses

Adds the head-capture builder path: a rolling RO pin lagging the tip by one
committed block supplies parent(B) commitment-latest, combined with the
committed >=B tx (from waitCommittedHead) for plain history. decidePin gates
the pin as a canonical parent (Finish==B-1, hash still canonical); a stale pin
(tip jump, reorg, or none yet) skips B to out-of-window and re-pins. buildAndCache
routes through the head-capture build in head-capture mode.

Reads the pinned parent commitment plane directly via tx.GetLatest instead of
through a SharedDomains getter. The pinned SharedDomains shared aggregator-scoped
cache state with the build's own commitment fold, so parent-commitment reads were
contaminated by post-state branches on storage-heavy / account-creating blocks
(root mismatch / malformed witness nodes). Direct snapshot reads isolate the
parent plane; this also drops the extra SharedDomains. The trie-phase reader now
reports WithHistory()==true so witness-capture PutBranch no-ops, matching durable.
…itment history

Decouple the eager witness cache from the DB commitment-history flag: a minimal
node with --witness.cache.head-capture set now enables the cache in cache-only,
pinned-parent mode. WitnessCacheShouldEnable gains a headCapture arg and a new
WitnessCacheMode resolves (enable, headCapture); commitment history present keeps
the durable recompute path even when the flag is on.
…ash canonical guard

Head-capture minimal nodes now serve debug_executionWitness / eth_getWitness
strictly from the cache: a miss returns a typed out-of-window error and never
falls through to a history recompute. serveFromWitnessCache re-checks the
canonical hash on by-hash resolves so a still-resident orphan is not served as
canonical, and reports the reorged-away bucket distinctly. The witness cache
reference moves to BaseAPI so the eth and debug serve paths read one mode source.
Add a "Head-capture serving (minimal nodes)" section to the
debug_executionWitness spec covering the two new flags
(--witness.cache.head-capture, --witness.cache.maxmb) and the
cache-only/tip-only/by-hash-canonical/cold-after-restart semantics.
Correct the now-conditional "requires commitment history" statements
and refresh stale code-map symbols (toWitnessTrie/GenerateWitness/
witnessCreateAccountNode removed by the fold refactor). Move the
completed head-capture plan to docs/plans/completed/.
- debug_executionWitness: canonical-mode request on a cache-only node now
  returns a distinct errWitnessCanonicalUnavailable instead of the misleading
  out-of-window error (the eager cache only builds legacy witnesses); covered
  by a new cache-only serve subtest.
- witness_cache_builder: drop the dead pin-drift Finish re-read in
  tryHeadCaptureBuild — the pin holds a fixed RO snapshot whose Finish cannot
  drift (proven by TestRollingPinStableUnderTipAdvance), so the guard was
  unreachable and its comment misleading.
- commitmentdb: remove the unused SharedDomainsGetter type alias (the
  head-capture reader reads via tx.GetLatest, no SharedDomains handle).
- flags: --witness.cache.blocks usage no longer claims commitment history is
  required; head-capture provides the minimal-node cache-only path.
- ChangeLog: add the v3.6.0 witness cache / head-capture entry.
AskAlexSharov and others added 8 commits July 24, 2026 14:36
Lint (after the db/services -> db/dbservices rename in the main merge):
gofmt witness_cache_builder import ordering, modernize the clamp test loop
to range-over-int.

Metric fidelity (code-review):
- record the build-duration histogram only on successful builds, not failures
- don't count a cache miss when the request's block can't be resolved
ExecutionWitnessResult has no MarshalJSON, so encoding/json never dispatches
to MarshalFastJSON — there is no recursion for executionWitnessResultView to
guard against. Marshal the pointer directly, avoiding both the view type and
the per-call struct copy.
Resolve APIList conflict from #22680 (rpc: extract api config pkg more):
adopt main's config-object constructors NewPrivateDebugAPI(...DebugApiConfig)
and NewTraceAPI(...TraceApiConfig), keeping debugImpl.witnessCache wiring.
Update the witness builder + its tests to the new NewPrivateDebugAPI signature.
- clamp --witness.cache.maxmb to bytes to avoid int overflow (32-bit / large values)
- make cache-only witness errors endpoint-agnostic (returned from eth_getWitness too)
- drop absolute worktree path from the completed plan doc
…apture

# Conflicts:
#	node/cli/flags.go
#	node/eth/backend.go
#	rpc/jsonrpc/daemon.go
#	rpc/jsonrpc/debug_execution_witness.go
#	rpc/jsonrpc/eth_api.go
#	rpc/jsonrpc/witness_cache.go
#	rpc/jsonrpc/witness_cache_builder.go
#	rpc/jsonrpc/witness_cache_builder_test.go
#	rpc/jsonrpc/witness_cache_test.go
tryHeadCaptureBuild called ObserveDuration both before the error check and
again after it, so every successful build recorded two samples. Observed on a
minimal mainnet node: 178 samples for 89 builds. Keep the success-only
observation, matching the durable path.
Base automatically changed from awskii/witness-cache to main July 28, 2026 09:18
WitnessNodes returns the output of WitnessNodesForKeysFromNodes, whose only
append site stores a freshly cloned slice, so buildWitnessTrie re-cloning every
node copied heap into heap. Measured on a minimal mainnet node: 10.2GB and 29M
objects over ~15min, ~4.7MB per witness build.
@awskii
awskii marked this pull request as ready for review July 29, 2026 09:56
@yperbasis yperbasis added this to the 3.7.0 milestone Jul 29, 2026
@yperbasis yperbasis added the RPC label Jul 29, 2026
@yperbasis
yperbasis requested a review from Copilot July 29, 2026 10:50

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 30 out of 30 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (1)

rpc/jsonrpc/witness_cache.go:122

  • residentBytes accounting uses int and is updated via c.residentBytes += size. On 32-bit builds (which the maxMB→bytes clamp explicitly calls out), a cache holding many large witnesses can overflow residentBytes (e.g., 96×~25MB > 2GB), making ResidentBytes()/byte-cap eviction logic incorrect. Use a wider type (int64) or saturating arithmetic when adding sizes to prevent overflow.
	if existed {
		c.residentBytes -= prev
	}
	c.entryBytes[hash] = size
	c.residentBytes += size

…l-headcapture

# Conflicts:
#	cmd/rpcdaemon/cli/config.go
#	cmd/rpcdaemon/cli/httpcfg/http_cfg.go
#	cmd/utils/flags.go
#	node/cli/default_flags.go
#	node/cli/flags.go
#	node/eth/backend.go
#	rpc/jsonrpc/debug_api_test.go
#	rpc/jsonrpc/debug_execution_witness.go
#	rpc/jsonrpc/witness_cache.go
#	rpc/jsonrpc/witness_cache_builder.go
#	rpc/jsonrpc/witness_cache_builder_test.go
#	rpc/jsonrpc/witness_cache_metrics.go
#	rpc/jsonrpc/witness_cache_test.go
#	rpc/jsonrpc/witness_cache_wiring_test.go
@AskAlexSharov
AskAlexSharov added this pull request to the merge queue Jul 31, 2026
Merged via the queue into main with commit ced5b59 Jul 31, 2026
107 checks passed
@AskAlexSharov
AskAlexSharov deleted the awskii/witness-minimal-headcapture branch July 31, 2026 08:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants