Skip to content

stage_exec: allow more workers for build/collate/merge#19207

Merged
AskAlexSharov merged 3 commits intomainfrom
stage_exec_more_collate
Feb 19, 2026
Merged

stage_exec: allow more workers for build/collate/merge#19207
AskAlexSharov merged 3 commits intomainfrom
stage_exec_more_collate

Conversation

@sudeepdino008
Copy link
Member

@sudeepdino008 sudeepdino008 commented Feb 16, 2026

i'm basicaly interested in making stage_exec/stage_custom_Trace/rebuild commitment faster by delaying/reducing the merge operation. Because now we do periodic merges there, and because we produce lot of files and end up "repeating" a lot of the merge.

in offline commands:
one thing was using optimal merge ranges - #19146
instead of periodic build+merge, we can do periodic BuildFilesInBackground(doMerge=false) and then 1 MergeLoop at the end after all executing is done-- I'll do it in some future PR. This ensures one-shot merge of (lots of) step1 files into a big file, rather than progressive merging into bigger and bigger files - which is repeated work.
integration stage_exec currently uses 1 worker for build/merge; it can be more (e.g. spawn_custom_trace use more workers) -- #19207 is meant to increase the number of workers for stage_exec (other offline commands look okay).

I don't intend to change behavior when erigon binary is running - chaintip or no chaintip. Just changing behaviour for offline commands with these changes.


  • not entirely happy with the solution (lock/unlock workers editing). Needed because ExecV3 modifies the workers.
  • future: spend less time in merging in offline commands (just perform one at the end) -- will make stage_exec/spawn_custom_trace spend less time in redundant merges

@sudeepdino008 sudeepdino008 force-pushed the stage_exec_more_collate branch from ccc37bd to fcc73ae Compare February 16, 2026 07:01
@AskAlexSharov
Copy link
Collaborator

sorry, don't understand this PR

@AskAlexSharov AskAlexSharov merged commit e5fb3a9 into main Feb 19, 2026
25 checks passed
@AskAlexSharov AskAlexSharov deleted the stage_exec_more_collate branch February 19, 2026 03:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants