Can't enable both parallel state root calculating and multiproof prewarming

[fynnss]

Describe the bug

I noticed that when --engine.legacy-state-root is enabled, the cache prewarming for multiproof is entirely disabled.

This is due to the logic in insert_block_inner, where spawn_cache_exclusive() is used if use_state_root_task is true and trie_input.prefix_sets is empty.

let mut handle = if use_state_root_task {
            // ....

            // Use state root task only if prefix sets are empty, otherwise proof generation is too
            // expensive because it requires walking over the paths in the prefix set in every
            // proof.
            if trie_input.prefix_sets.is_empty() {
                self.payload_processor.spawn(
                    header,
                    txs,
                    provider_builder,
                    consistent_view,
                    trie_input,
                    &self.config,
                )
            } else {
                debug!(target: "engine::tree", block=?block_num_hash, "Disabling state root task due to non-empty prefix sets");
                use_state_root_task = false;
                self.payload_processor.spawn_cache_exclusive(header, txs, provider_builder)
            }
        } else {
            self.payload_processor.spawn_cache_exclusive(header, txs, provider_builder)
        };

In that case, to_multi_proof is always set to None, so no prewarming happens for multiproof targets.

// crates/engine/tree/src/tree/payload_processor/mod.rs:250
let prewarm_handle = self.spawn_caching_with(header, transactions, provider_builder, None);

This creates a performance issue: even though we're using the legacy state root calculation (which is already parallel in foreground, not backgroud task), the multiproof task could still benefit from cache prewarming — but it doesn't work under this logic.

BTW, if --engine.legacy-state-root is not set, and the parallel root calculation fails (e.g. due to mismatch), it falls back to serial calculation.

2025-07-28T09:25:58.325384Z  WARN engine::tree: State root task returned incorrect state root state_root=0xc3ce4734893ed87f062462c90db3d745994865be65ec2575a500a70d1b1cbd1f block_state_root=0xe0c8c6fe648270d3b2ab29f33d386d70070d6ff097dbc522868515012a2f89ad
2025-07-28T09:25:58.325389Z  WARN engine::tree: Failed to compute state root in parallel block=NumHash { number: 1000050, hash: 0xf30cdb4efe57d0980026a79775a8dda83cb05763d3a329a994a9882bd3d799fb } persisting_kind=NotPersisting

Is this behavior expected? This cause that i can't have both parallel state root and prewarm proofs at the same. So the questions are: (:TLDR)

1. Why is prewarming proofs only enabled when use_state_root_task is true and prefix_sets is empty? What’s the reason behind this condition?
2. Why does the parallel root calculation always fail and fall back to the serial path when using state root task?

Thanks! Any feedback or clarification would be greatly appreciated.

Steps to reproduce

1. pipeline sync to 100w

RUST_LOG=info ./target/release/reth-bsc node \
    --chain=bsc \
    --datadir=./data_dir \
    --debug.tip=0x48917a17f56f1f1e67771be69f08f0de69e76bed3149433d74ef7bcdba296546

2. live sync

RUST_LOG=info ./target/release/reth-bsc node \
    --chain=bsc \
    --datadir=./data_dir \
    --authrpc.addr "0.0.0.0" \
    --authrpc.port 8551 \
    --authrpc.jwtsecret ./data_dir/jwt.hex \
    --http \
    --engine.legacy-state-root \
    --http.addr "0.0.0.0" \
    --http.port 8545 \
    --metrics 0.0.0.0:6060 \
    --http.api "admin, debug, eth, net, trace, txpool, web3, rpc, reth, ots" ' > reth-bsc.log 

Node logs

Platform(s)

Linux (x86)

Container Type

Not running in a container

What version/commit are you on?

v1.5.1

What database version are you on?

Current database version: 2
Local database version: 2

Which chain / network are you on?

bsc

What type of node are you running?

Full via --full flag

What prune config do you use, if any?

No response

If you've built Reth from source, provide the full command you used

cargo build --release

Code of Conduct

• I agree to follow the Code of Conduct

[fynnss]

A draft fix. 2b09e74

[Rjected]

Why is prewarming proofs only enabled when use_state_root_task is true

Transaction prewarming is enabled, but we only do proof fetching for the purposes of sparse trie root calculation, which relies on proofs to load state. Sparse trie root calculation is meant to minimize the time between execution being finished, and the state root being calculated. The other root algorithms are not optimal for that use case. Because we wouldn't use any of the proofs in the other root calculation algorithms, we don't spawn the multiproof task when we aren't using the sparse trie.

and prefix_sets is empty

This is because of the comment (from the snippet):

            // Use state root task only if prefix sets are empty, otherwise proof generation is too
            // expensive because it requires walking over the paths in the prefix set in every
            // proof.

Why does the parallel root calculation always fail and fall back to the serial path when using state root task?

The parallel root calculation should not always fail with a root mismatch, that would be either a bug in the state (execution) or in the parallel root code.

[fynnss]

Thanks for the clarification — that makes sense.

The other root algorithms are not optimal for that use case.

The trie nodes (proof-related data) will be prefetched during execution, and then reused from the DB cache during validation — that should help as well for other root calculation algorithms, right?

[Rjected]

The trie nodes (proof-related data) will be prefetched during execution, and then reused from the DB cache during validation — that should help as well for other root calculation algorithms, right?

Maybe, we've never measured the impact of this though + have been focused more on improving perf when sparse trie is being used

[fynnss]

The parallel root calculation should not always fail with a root mismatch, that would be either a bug in the state (execution) or in the parallel root code.

So it sounds like this isn’t expected — thanks for confirming. I’ll dig deeper to check whether the mismatch is caused by some bugs in the root calculation logic.

[fynnss]

I find it's not always fails, but the success rate is very low. As the logs show below, only 1000086 is success.

Are there any parameters or configurations that could help improve the success rate? @Rjected

2025-07-30T02:38:53.734688Z  INFO Block added to canonical chain number=1000083 hash=0xa48b4f51f84358e5520debbc0c6f5d40caf46255cfe0973d351ed52960c340e5 peers=6 txs=6 gas=599.22 Kgas gas_throughput=56.29 Mgas/second gas_limit=30.00 Mgas full=2.0% base_fee=0.00gwei blobs=0 excess_blobs=0 elapsed=10.645791ms
2025-07-30T02:38:53.734771Z  INFO Canonical chain committed number=1000083 hash=0xa48b4f51f84358e5520debbc0c6f5d40caf46255cfe0973d351ed52960c340e5 elapsed=35.792µs
2025-07-30T02:38:55.633217Z  INFO State root task finished state_root=0x736e1184491d496e71443990ebee42aae2b334159e1bbb05af7ded6af0cf076d elapsed=129.917µs
2025-07-30T02:38:55.633234Z  WARN State root task returned incorrect state root state_root=0x736e1184491d496e71443990ebee42aae2b334159e1bbb05af7ded6af0cf076d block_state_root=0xc35da60d8aded181f77ca3af5b1ee1d19996091f6e5c68c702c21878ba76b64b
2025-07-30T02:38:55.633240Z  WARN Failed to compute state root in parallel block=NumHash { number: 1000084, hash: 0x77a9f6db6fc521c7f295b1b911d0a67c2460a36ad11e3d81d00268a7eb4bd936 } persisting_kind=NotPersisting
2025-07-30T02:38:55.635014Z  INFO Block added to canonical chain number=1000084 hash=0x77a9f6db6fc521c7f295b1b911d0a67c2460a36ad11e3d81d00268a7eb4bd936 peers=5 txs=2 gas=128.48 Kgas gas_throughput=42.42 Mgas/second gas_limit=30.00 Mgas full=0.4% base_fee=0.00gwei blobs=0 excess_blobs=0 elapsed=3.028916ms
2025-07-30T02:38:55.635080Z  INFO Canonical chain committed number=1000084 hash=0x77a9f6db6fc521c7f295b1b911d0a67c2460a36ad11e3d81d00268a7eb4bd936 elapsed=18.542µs
2025-07-30T02:38:57.623109Z  INFO State root task finished state_root=0xc35da60d8aded181f77ca3af5b1ee1d19996091f6e5c68c702c21878ba76b64b elapsed=921.375µs
2025-07-30T02:38:57.623121Z  WARN State root task returned incorrect state root state_root=0xc35da60d8aded181f77ca3af5b1ee1d19996091f6e5c68c702c21878ba76b64b block_state_root=0x2f25f48c3f85637b5ddfb4b6dc169aa35159a3852d3ee2399aeb6d36643088ce
2025-07-30T02:38:57.623126Z  WARN Failed to compute state root in parallel block=NumHash { number: 1000085, hash: 0x2853f80d0b1802b4be934ec339a53c4c243d1e9bd103e76b84dca264c4f9cdb9 } persisting_kind=NotPersisting
2025-07-30T02:38:57.623737Z  INFO Block added to canonical chain number=1000085 hash=0x2853f80d0b1802b4be934ec339a53c4c243d1e9bd103e76b84dca264c4f9cdb9 peers=5 txs=2 gas=55.08 Kgas gas_throughput=25.87 Mgas/second gas_limit=30.00 Mgas full=0.2% base_fee=0.00gwei blobs=0 excess_blobs=0 elapsed=2.129ms
2025-07-30T02:38:57.623750Z  INFO Canonical chain committed number=1000085 hash=0x2853f80d0b1802b4be934ec339a53c4c243d1e9bd103e76b84dca264c4f9cdb9 elapsed=8.417µs
2025-07-30T02:38:59.594261Z  INFO State root task finished state_root=0x2f25f48c3f85637b5ddfb4b6dc169aa35159a3852d3ee2399aeb6d36643088ce elapsed=391.625µs
2025-07-30T02:38:59.594440Z  INFO Block added to canonical chain number=1000086 hash=0xee1e9d32e2261c3a99e8305ce3094a76fbf1c5e3733de0d062118fcc0ee98ce0 peers=5 txs=0 gas=0.00 Kgas gas_throughput=0.00 Kgas/second gas_limit=30.00 Mgas full=0.0% base_fee=0.00gwei blobs=0 excess_blobs=0 elapsed=1.586709ms
2025-07-30T02:38:59.594484Z  INFO Canonical chain committed number=1000086 hash=0xee1e9d32e2261c3a99e8305ce3094a76fbf1c5e3733de0d062118fcc0ee98ce0 elapsed=19.875µs
2025-07-30T02:39:01.626016Z  INFO State root task finished state_root=0x2f25f48c3f85637b5ddfb4b6dc169aa35159a3852d3ee2399aeb6d36643088ce elapsed=291.5µs
2025-07-30T02:39:01.626043Z  WARN State root task returned incorrect state root state_root=0x2f25f48c3f85637b5ddfb4b6dc169aa35159a3852d3ee2399aeb6d36643088ce block_state_root=0x3f7b5098927fb7aaceb05c0595e1930002ea5ebe0f7019e4c389c75b91af2835
2025-07-30T02:39:01.626089Z  WARN Failed to compute state root in parallel block=NumHash { number: 1000087, hash: 0xe2a58ccf83c59ad719c7112572d5e9163265486affbb0767508e07a3a77ea68e } persisting_kind=NotPersisting
2025-07-30T02:39:01.630902Z  INFO Block added to canonical chain number=1000087 hash=0xe2a58ccf83c59ad719c7112572d5e9163265486affbb0767508e07a3a77ea68e peers=5 txs=3 gas=301.91 Kgas gas_throughput=32.49 Mgas/second gas_limit=30.00 Mgas full=1.0% base_fee=0.00gwei blobs=0 excess_blobs=0 elapsed=9.291375ms
2025-07-30T02:39:01.631009Z  INFO Canonical chain committed number=1000087 hash=0xe2a58ccf83c59ad719c7112572d5e9163265486affbb0767508e07a3a77ea68e elapsed=35.958µs
2025-07-30T02:39:12.677232Z  INFO Status connected_peers=6 latest_block=1000087

the run command.

RUST_LOG=info ./target/release/reth-bsc node \
    --chain=bsc \
    --datadir=./data_dir \
    --authrpc.addr "0.0.0.0" \
    --authrpc.port 8551 \
    --authrpc.jwtsecret ./data_dir/jwt.hex \
    --engine.parallel-sparse-trie \
    --http \
    --http.addr "0.0.0.0" \
    --http.port 8545 \
    --metrics 0.0.0.0:6060 \
    --http.api "admin, debug, eth, net, trace, txpool, web3, rpc, reth, ots"

debug logs

2025-07-30T03:07:44.470060Z  INFO reth_node_events::node: Block added to canonical chain number=1000150 hash=0xa9a4f81b2c33725fc6e5ffb99acc26a4e228c56f4e06eb4ac85841488402d45c peers=3 txs=2 gas=188.02 Kgas gas_throughput=43.57 Mgas/second gas_limit=30.00 Mgas full=0.6% base_fee=0.00gwei blobs=0 excess_blobs=0 elapsed=4.315041ms
2025-07-30T03:07:44.470099Z  INFO reth_node_events::node: Canonical chain committed number=1000150 hash=0xa9a4f81b2c33725fc6e5ffb99acc26a4e228c56f4e06eb4ac85841488402d45c elapsed=32.75µs
2025-07-30T03:07:46.446981Z DEBUG engine::root: Checking end condition proofs_processed=0 state_update_proofs_requested=0 prefetch_proofs_requested=0 no_pending=true updates_finished=true
2025-07-30T03:07:46.447011Z DEBUG engine::root: State updates finished and all proofs processed, ending calculation
2025-07-30T03:07:46.447017Z DEBUG engine::root: All proofs processed, ending calculation total_updates=0 total_proofs=0 total_time=None time_since_updates_finished=Some(39.583µs)
2025-07-30T03:07:46.447064Z DEBUG engine::root: All proofs processed, ending calculation num_iterations=0
2025-07-30T03:07:46.447469Z  INFO engine::tree: State root task finished state_root=0xb43e5bf3e86263a694f7f4649ffdef3ce90f05567a2737affedde891c28a52ac elapsed=431.875µs
2025-07-30T03:07:46.447602Z  INFO reth_node_events::node: Block added to canonical chain number=1000151 hash=0x6839a74b0a7afcf761a9daac6e8c964cfb31bf8126217b836cdaaff816d272ea peers=3 txs=0 gas=0.00 Kgas gas_throughput=0.00 Kgas/second gas_limit=30.00 Mgas full=0.0% base_fee=0.00gwei blobs=0 excess_blobs=0 elapsed=1.513416ms
2025-07-30T03:07:46.447634Z  INFO reth_node_events::node: Canonical chain committed number=1000151 hash=0x6839a74b0a7afcf761a9daac6e8c964cfb31bf8126217b836cdaaff816d272ea elapsed=11.583µs
2025-07-30T03:07:48.585674Z DEBUG engine::root: Started state root calculation
2025-07-30T03:07:48.585722Z DEBUG engine::root: Prefetching proofs account_targets=3 storage_targets=1 prefetch_proofs_requested=1
2025-07-30T03:07:48.585897Z DEBUG engine::root: Checking end condition proofs_processed=0 state_update_proofs_requested=0 prefetch_proofs_requested=1 no_pending=true updates_finished=true
2025-07-30T03:07:48.586286Z DEBUG engine::root: Prefetching proofs account_targets=6 storage_targets=7 prefetch_proofs_requested=2
2025-07-30T03:07:48.586311Z DEBUG engine::root: Processing calculated proof sequence=0 total_proofs=1
2025-07-30T03:07:48.586326Z DEBUG engine::root: Checking end condition proofs_processed=1 state_update_proofs_requested=0 prefetch_proofs_requested=2 no_pending=true updates_finished=true
2025-07-30T03:07:48.586339Z DEBUG engine::root: Updating sparse trie num_updates=1 account_proofs=15 storage_proofs=3
2025-07-30T03:07:48.587171Z DEBUG engine::root: Processing calculated proof sequence=1 total_proofs=2
2025-07-30T03:07:48.587188Z DEBUG engine::root: Checking end condition proofs_processed=2 state_update_proofs_requested=0 prefetch_proofs_requested=2 no_pending=true updates_finished=true
2025-07-30T03:07:48.587194Z DEBUG engine::root: State updates finished and all proofs processed, ending calculation
2025-07-30T03:07:48.587197Z DEBUG engine::root: All proofs processed, ending calculation total_updates=0 total_proofs=2 total_time=Some(1.524458ms) time_since_updates_finished=Some(1.300416ms)
2025-07-30T03:07:48.587211Z DEBUG engine::root: Updating sparse trie num_updates=1 account_proofs=25 storage_proofs=5
2025-07-30T03:07:48.587503Z DEBUG engine::root: All proofs processed, ending calculation num_iterations=2
2025-07-30T03:07:48.587560Z  INFO engine::tree: State root task finished state_root=0xb43e5bf3e86263a694f7f4649ffdef3ce90f05567a2737affedde891c28a52ac elapsed=1.195167ms
2025-07-30T03:07:48.587568Z  WARN engine::tree: State root task returned incorrect state root state_root=0xb43e5bf3e86263a694f7f4649ffdef3ce90f05567a2737affedde891c28a52ac block_state_root=0xe099186cf3249d9e90a43f7a64636970792a39d80d4c94bf6bcda4cfe174b81b
2025-07-30T03:07:48.587575Z  WARN engine::tree: Failed to compute state root in parallel block=NumHash { number: 1000152, hash: 0xefc445e775560fb4c1bad18d48bc1758b18a9cbb29abf33c34ac87ce5ff949e1 } persisting_kind=NotPersisting
2025-07-30T03:07:48.588608Z  INFO reth_node_events::node: Block added to canonical chain number=1000152 hash=0xefc445e775560fb4c1bad18d48bc1758b18a9cbb29abf33c34ac87ce5ff949e1 peers=2 txs=3 gas=172.91 Kgas gas_throughput=50.03 Mgas/second gas_limit=30.00 Mgas full=0.6% base_fee=0.00gwei blobs=0 excess_blobs=0 elapsed=3.456125ms
2025-07-30T03:07:48.588670Z  INFO reth_node_events::node: Canonical chain committed number=1000152 hash=0xefc445e775560fb4c1bad18d48bc1758b18a9cbb29abf33c34ac87ce5ff949e1 elapsed=13.791µs

[fynnss]

Hi，@Rjected. the issues caused by the state hook doesn't impl in the reth-bsc and I submitted a PR to fix it.

loocapro/reth-bsc#51

The root hash has changed in state root task, but it still mismatch. Could there be a missing hook call somewhere?

=====

the missing hook call here: https://github.com/loocapro/reth-bsc/pull/51/files#diff-137d435d4b6eb89233776ce66d846dd3499172e5df477ec8b2c76bc1198430f4R480-R486

After remove the state of SYSTEM_ADDRESS, the state root task has been passed.

[fynnss]

the missing hook call here: https://github.com/loocapro/reth-bsc/pull/51/files#diff-137d435d4b6eb89233776ce66d846dd3499172e5df477ec8b2c76bc1198430f4R480-R486