[r3.5] cmd/integration: collate state files incrementally in parallel stage_exec - #22262
Merged
Merged
Conversation
…exec The parallel executor never triggers step collation during stage_exec, so every step accumulates in the DB (chaindata grows unbounded) and a later offline finalize has to build each step by scanning the whole DB — O(n^2). The serial executor collates implicitly, keeping the DB small. Kick BuildFilesInBackground each commit in the stageExec loop (as the node's executor.go CommitCycle does); the loop's existing PruneSmallBatches then reclaims the collated steps, so the DB stays small on the parallel path too.
AskAlexSharov
approved these changes
Jul 6, 2026
AskAlexSharov
enabled auto-merge (squash)
July 6, 2026 10:34
yperbasis
added a commit
that referenced
this pull request
Jul 10, 2026
Adds the **v3.5.1** release notes to `ChangeLog.md`, following the v3.4.1–v3.4.4 point-release format (codename **Tidal Tails**, kept from 3.5.0; date left as `TBD` pending the release). Scope: user-facing **Bugfixes** and **Improvements** merged to `release/3.5` since the `v3.5.0` tag. Each entry cites the `[r3.5]` PR that landed on the branch and credits the original fix author, matching the existing point-release style. The six issues closed in [milestone 3.5.1](https://github.com/erigontech/erigon/milestone/78) are all covered: #21992 & #22101 (commitment consistency), #22013 (`stage_custom_trace` prune), #22275 (snapshot reset), #22337 (Bellatrix reconstruction loop), #22351 (forkchoice `GetHead` stall). Deliberately omitted as minor/internal/tooling/docs-only: - `cmd/integration` / `seg` build tweaks (#22137, #22262), QA sync-wait bump (#22280), docs & changelog commits (#22062, #22068, #22090), downloader concurrency hardening (#22319). - The **net-zero** rcache pair — #22047 (bump rcache domain to v3.1) and its revert #22207 — cancels out, so it produces no 3.5.0→3.5.1 change. Opened as a **draft**: fill in the release date on tag, and confirm nothing else is expected to land in 3.5.1 before finalizing. _No `make lint` run — the diff is markdown-only (`ChangeLog.md`), which golangci-lint does not scan._
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
integration stage_execon the parallel execution path (EXEC3_PARALLEL=true) never triggers step collation, so every step stays in the DB.stepsInDBgrows monotonically to the full run andchaindataballoons. A later offline finalize (seg retire→agg.BuildFiles) then has to build each step by scanning the whole DB — effectively O(n²).The serial path collates implicitly during execution, keeping the DB small.
Observed on a mainnet run to step 260 (block 4,665,800):
[agg] buildingduring execstepsInDBat endseg retireFix
Kick
agg.BuildFilesInBackground(agg.EndTxNumMinimax() + agg.StepSize())each commit in thestageExecloop — the same call the node executor makes in itsCommitCycle(execution/execmodule/executor.go). The loop's existingPruneExecutionStage→PruneSmallBatchesthen reclaims the collated steps, so the DB stays small on the parallel path too (build alone collates; build + the existing prune shrinks).The
buildFilesInBackgroundgate does not depend onLockWorkersEditing, so the call is effective during offline execution.Testing
Behavior fix in the integration tool (not easily unit-testable without a full historical exec). Verified by the measurements above: before, parallel
stage_execleaves 260 steps in the DB; the change makes it collate/prune incrementally like the serial path and the node.