op-reth dev mode: Failed to insert downloaded block: receipt root mismatch

[Troublor]

Describe the bug

When running op-reth in dev mode with block time interval, every block produced by AutoSeal is considered invalid by BeaconConsensusEngine, saying receipt root mismatch.

Steps to reproduce

1. checkout v1.0.0-rc.2
2. cargo run --bin op-reth --features optimism -- node --dev --dev.block-time 1s

Node logs

2024-06-18T19:26:44.531092Z  INFO reth::cli: Initialized tracing, debug log directory: /home/zhan4987/.cache/reth/logs/dev
2024-06-18T19:26:44.540595Z  INFO reth::cli: Starting reth version="1.0.0-rc.2-dev (d786b459d)"
2024-06-18T19:26:44.541365Z  INFO reth::cli: Opening database path="/home/zhan4987/.local/share/reth/dev/db"
2024-06-18T19:26:44.775028Z DEBUG storage::db::mdbx: Commit total_duration=1.684631ms commit_latency=Some(CommitLatency(MDBX_commit_latency { preparation: 0, gc_wallclock: 2, audit: 0, write: 0, sync: 105, ending: 0, whole: 109, gc_cputime: 0, gc_prof: MDBX_commit_latency__bindgen_ty_1 { wloops: 1, coalescences: 0, wipes: 0, flushes: 0, kicks: 0, work_counter: 1, work_rtime_monotonic: 1, work_xtime_cpu: 0, work_rsteps: 1, work_xpages: 0, work_majflt: 0, self_counter: 5, self_rtime_monotonic: 1, self_xtime_cpu: 0, self_rsteps: 0, self_xpages: 0, self_majflt: 0 } })) is_read_only=false
2024-06-18T19:26:44.778101Z DEBUG reth_node_builder::launch::common: Raised file descriptor limit from=1048576 to=1048576
2024-06-18T19:26:44.789333Z  INFO reth::cli: Configuration loaded path="/home/zhan4987/.local/share/reth/dev/reth.toml"
2024-06-18T19:26:44.845610Z  INFO reth::cli: Verifying storage consistency.
2024-06-18T19:26:44.860683Z  INFO reth::cli: Database opened
2024-06-18T19:26:44.860738Z DEBUG reth::cli: Initializing genesis chain=dev genesis=0x2f980576711e3617a5e4d83dd539548ec0f7792007d505a3d2e9674833af2d7c
2024-06-18T19:26:44.860927Z DEBUG reth_db_common::init: Genesis already written, skipping.
2024-06-18T19:26:44.860984Z  INFO reth::cli: 
Pre-merge hard forks (block based):
- Frontier                         @0
- Homestead                        @0
- Dao                              @0
- Tangerine                        @0
- SpuriousDragon                   @0
- Byzantium                        @0
- Constantinople                   @0
- Petersburg                       @0
- Istanbul                         @0
- MuirGlacier                      @0
- Berlin                           @0
- London                           @0
- Bedrock                          @0
Merge hard forks:
- Paris                            @0 (network is known to be merged)
Post-merge hard forks (timestamp based):
- Regolith                         @0
- Shanghai                         @0
- Cancun                           @0
- Ecotone                          @0
2024-06-18T19:26:44.861005Z DEBUG reth::cli: Spawning stages metrics listener task
2024-06-18T19:26:44.861499Z DEBUG reth::cli: creating components
2024-06-18T19:26:44.861669Z DEBUG txpool::blob: Removed blob store directory blob_dir="/home/zhan4987/.local/share/reth/dev/blobstore"
2024-06-18T19:26:44.861683Z DEBUG txpool::blob: Creating blob store blob_dir="/home/zhan4987/.local/share/reth/dev/blobstore"
2024-06-18T19:26:44.921052Z  INFO reth::cli: Transaction pool initialized
2024-06-18T19:26:44.921165Z DEBUG reth::cli: Spawned txpool maintenance task
2024-06-18T19:26:44.921611Z  INFO net::peers: Loading saved peers file=/home/zhan4987/.local/share/reth/dev/known-peers.json
2024-06-18T19:26:44.924133Z DEBUG reth::cli: configured blockchain tree
2024-06-18T19:26:44.924161Z DEBUG reth::cli: calling on_component_initialized hook
2024-06-18T19:26:44.924630Z  INFO reth::cli: StaticFileProducer initialized
2024-06-18T19:26:44.924681Z  INFO reth::cli: Starting Reth in dev mode
2024-06-18T19:26:44.924768Z  INFO reth::cli: Allocated Genesis Account: 00. 0x14dC79964da2C08b23698B3D3cc7Ca32193d9955 (1000000.000000000000000000 ETH)
2024-06-18T19:26:44.924807Z  INFO reth::cli: Allocated Genesis Account: 01. 0x15d34AAf54267DB7D7c367839AAf71A00a2C6A65 (1000000.000000000000000000 ETH)
2024-06-18T19:26:44.924844Z  INFO reth::cli: Allocated Genesis Account: 02. 0x1CBd3b2770909D4e10f157cABC84C7264073C9Ec (1000000.000000000000000000 ETH)
2024-06-18T19:26:44.924881Z  INFO reth::cli: Allocated Genesis Account: 03. 0x23618e81E3f5cdF7f54C3d65f7FBc0aBf5B21E8f (1000000.000000000000000000 ETH)
2024-06-18T19:26:44.924917Z  INFO reth::cli: Allocated Genesis Account: 04. 0x2546BcD3c84621e976D8185a91A922aE77ECEc30 (1000000.000000000000000000 ETH)
2024-06-18T19:26:44.924953Z  INFO reth::cli: Allocated Genesis Account: 05. 0x3C44CdDdB6a900fa2b585dd299e03d12FA4293BC (1000000.000000000000000000 ETH)
2024-06-18T19:26:44.924990Z  INFO reth::cli: Allocated Genesis Account: 06. 0x70997970C51812dc3A010C7d01b50e0d17dc79C8 (1000000.000000000000000000 ETH)
2024-06-18T19:26:44.925027Z  INFO reth::cli: Allocated Genesis Account: 07. 0x71bE63f3384f5fb98995898A86B02Fb2426c5788 (1000000.000000000000000000 ETH)
2024-06-18T19:26:44.925063Z  INFO reth::cli: Allocated Genesis Account: 08. 0x8626f6940E2eb28930eFb4CeF49B2d1F2C9C1199 (1000000.000000000000000000 ETH)
2024-06-18T19:26:44.925100Z  INFO reth::cli: Allocated Genesis Account: 09. 0x90F79bf6EB2c4f870365E785982E1f101E93b906 (1000000.000000000000000000 ETH)
2024-06-18T19:26:44.925137Z  INFO reth::cli: Allocated Genesis Account: 10. 0x976EA74026E726554dB657fA54763abd0C3a0aa9 (1000000.000000000000000000 ETH)
2024-06-18T19:26:44.925173Z  INFO reth::cli: Allocated Genesis Account: 11. 0x9965507D1a55bcC2695C58ba16FB37d819B0A4dc (1000000.000000000000000000 ETH)
2024-06-18T19:26:44.925209Z  INFO reth::cli: Allocated Genesis Account: 12. 0xa0Ee7A142d267C1f36714E4a8F75612F20a79720 (1000000.000000000000000000 ETH)
2024-06-18T19:26:44.925248Z  INFO reth::cli: Allocated Genesis Account: 13. 0xBcd4042DE499D14e55001CcbB24a551F3b954096 (1000000.000000000000000000 ETH)
2024-06-18T19:26:44.925284Z  INFO reth::cli: Allocated Genesis Account: 14. 0xbDA5747bFD65F08deb54cb465eB87D40e51B197E (1000000.000000000000000000 ETH)
2024-06-18T19:26:44.925320Z  INFO reth::cli: Allocated Genesis Account: 15. 0xcd3B766CCDd6AE721141F452C550Ca635964ce71 (1000000.000000000000000000 ETH)
2024-06-18T19:26:44.925356Z  INFO reth::cli: Allocated Genesis Account: 16. 0xdD2FD4581271e230360230F9337D5c0430Bf44C0 (1000000.000000000000000000 ETH)
2024-06-18T19:26:44.925392Z  INFO reth::cli: Allocated Genesis Account: 17. 0xdF3e18d64BC6A983f673Ab319CCaE4f1a57C7097 (1000000.000000000000000000 ETH)
2024-06-18T19:26:44.925428Z  INFO reth::cli: Allocated Genesis Account: 18. 0xf39Fd6e51aad88F6F4ce6aB8827279cffFb92266 (1000000.000000000000000000 ETH)
2024-06-18T19:26:44.925465Z  INFO reth::cli: Allocated Genesis Account: 19. 0xFABB0ac9d68B0B445fB7357272Ff202C5651694a (1000000.000000000000000000 ETH)
2024-06-18T19:26:44.925491Z  INFO reth::cli: configuring dev mining mode mode=FixedBlockTime
2024-06-18T19:26:44.926220Z DEBUG reth::cli: Spawning auto mine task
2024-06-18T19:26:44.926433Z  INFO reth::cli: Pruner initialized prune_config=PruneConfig { block_interval: 5, segments: PruneModes { sender_recovery: None, transaction_lookup: None, receipts: None, account_history: None, storage_history: None, receipts_log_filter: ReceiptsLogPruneConfig({}) } }
2024-06-18T19:26:44.926868Z  INFO reth::cli: Consensus engine initialized
2024-06-18T19:26:44.927135Z  INFO reth::cli: Engine API handler initialized
2024-06-18T19:26:44.927165Z DEBUG reth::cli: Reading JWT auth secret file path="/home/zhan4987/.local/share/reth/dev/jwt.hex"
2024-06-18T19:26:44.927417Z DEBUG reth::cli: Using RPC module config http=Some(Selection({Eth, Net, Web3})) ws=None
2024-06-18T19:26:44.937420Z  INFO reth::cli: RPC auth server started url=127.0.0.1:8551
2024-06-18T19:26:44.937433Z DEBUG reth_ipc::server: removed existing IPC endpoint file endpoint="/tmp/reth.ipc"
2024-06-18T19:26:44.937641Z  INFO reth::cli: RPC IPC server started path=/tmp/reth.ipc
2024-06-18T19:26:44.937657Z  INFO reth::cli: RPC HTTP server started url=127.0.0.1:8545
2024-06-18T19:26:44.940548Z  INFO reth::cli: Starting consensus engine
2024-06-18T19:26:44.940896Z DEBUG consensus::engine::hooks: Next hook is not ready hook="StaticFile"
2024-06-18T19:26:45.932957Z DEBUG consensus::auto: Sent fork choice update state=ForkchoiceState { head_block_hash: 0x25d1f803b6d822f1db1e5ebfb8e66b004c9f9ae1fd172df5cddecf7a929541af, safe_block_hash: 0x25d1f803b6d822f1db1e5ebfb8e66b004c9f9ae1fd172df5cddecf7a929541af, finalized_block_hash: 0x25d1f803b6d822f1db1e5ebfb8e66b004c9f9ae1fd172df5cddecf7a929541af }
2024-06-18T19:26:45.934374Z DEBUG blockchain_tree: Block hash not found in block indices block_hash=0x25d1f803b6d822f1db1e5ebfb8e66b004c9f9ae1fd172df5cddecf7a929541af
2024-06-18T19:26:45.934849Z DEBUG consensus::engine: Syncing to new target target=0x25d1f803b6d822f1db1e5ebfb8e66b004c9f9ae1fd172df5cddecf7a929541af
2024-06-18T19:26:45.934967Z DEBUG consensus::auto: Forkchoice update returned SYNCING, waiting for VALID fcu_response=OnForkChoiceUpdated { forkchoice_status: Syncing, fut: Left(Ready(Some(Ok(ForkchoiceUpdated { payload_status: PayloadStatus { status: Syncing, latest_valid_hash: None }, payload_id: None })))) }
2024-06-18T19:26:45.934992Z DEBUG consensus::auto: Sent fork choice update state=ForkchoiceState { head_block_hash: 0x25d1f803b6d822f1db1e5ebfb8e66b004c9f9ae1fd172df5cddecf7a929541af, safe_block_hash: 0x25d1f803b6d822f1db1e5ebfb8e66b004c9f9ae1fd172df5cddecf7a929541af, finalized_block_hash: 0x25d1f803b6d822f1db1e5ebfb8e66b004c9f9ae1fd172df5cddecf7a929541af }
2024-06-18T19:26:45.935462Z  INFO reth_node_events::node: Live sync in progress, downloading blocks remaining_blocks=1 target_block_hash=0x25d1f803b6d822f1db1e5ebfb8e66b004c9f9ae1fd172df5cddecf7a929541af
2024-06-18T19:26:45.935636Z  INFO reth_node_events::node: Received forkchoice updated message when syncing head_block_hash=0x25d1f803b6d822f1db1e5ebfb8e66b004c9f9ae1fd172df5cddecf7a929541af safe_block_hash=0x25d1f803b6d822f1db1e5ebfb8e66b004c9f9ae1fd172df5cddecf7a929541af finalized_block_hash=0x25d1f803b6d822f1db1e5ebfb8e66b004c9f9ae1fd172df5cddecf7a929541af
2024-06-18T19:26:45.936289Z DEBUG blockchain_tree: Appending block to canonical chain head=0x25d1f803b6d822f1db1e5ebfb8e66b004c9f9ae1fd172df5cddecf7a929541af parent=BlockNumHash { number: 0, hash: 0x2f980576711e3617a5e4d83dd539548ec0f7792007d505a3d2e9674833af2d7c }
2024-06-18T19:26:45.937278Z DEBUG reth_optimism_consensus::validation: receipts verification failed error=receipt root mismatch: got 0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421, expected 0x0000000000000000000000000000000000000000000000000000000000000000 receipts=[]
2024-06-18T19:26:45.937378Z  WARN consensus::engine: Failed to insert downloaded block err=Failed to insert block (hash=0x25d1f803b6d822f1db1e5ebfb8e66b004c9f9ae1fd172df5cddecf7a929541af, number=1, parent_hash=0x2f980576711e3617a5e4d83dd539548ec0f7792007d505a3d2e9674833af2d7c): receipt root mismatch: got 0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421, expected 0x0000000000000000000000000000000000000000000000000000000000000000
2024-06-18T19:26:45.937402Z  WARN consensus::engine: Marking block as invalid invalid_number=1 invalid_hash=0x25d1f803b6d822f1db1e5ebfb8e66b004c9f9ae1fd172df5cddecf7a929541af err=receipt root mismatch: got 0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421, expected 0x0000000000000000000000000000000000000000000000000000000000000000
2024-06-18T19:26:45.937462Z  WARN consensus::engine: Bad block with hash hash=0x25d1f803b6d822f1db1e5ebfb8e66b004c9f9ae1fd172df5cddecf7a929541af header=Header { parent_hash: 0x2f980576711e3617a5e4d83dd539548ec0f7792007d505a3d2e9674833af2d7c, ommers_hash: 0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347, beneficiary: 0x0000000000000000000000000000000000000000, state_root: 0xf09d8f7da5bc5036f8dd9536c953e2212390a46fb3e553ece2b7d419131537b1, transactions_root: 0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421, receipts_root: 0x0000000000000000000000000000000000000000000000000000000000000000, withdrawals_root: Some(0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421), logs_bloom: 0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000, difficulty: 2, number: 1, gas_limit: 30000000, gas_used: 0, timestamp: 1718738805, mix_hash: 0x0000000000000000000000000000000000000000000000000000000000000000, nonce: 0, base_fee_per_gas: Some(875000000), blob_gas_used: Some(0), excess_blob_gas: Some(0), parent_beacon_block_root: Some(0x0000000000000000000000000000000000000000000000000000000000000000), requests_root: None, extra_data: 0x }
2024-06-18T19:26:45.937693Z ERROR consensus::auto: Forkchoice update returned invalid response fcu_response=OnForkChoiceUpdated { forkchoice_status: Invalid, fut: Left(Ready(Some(Ok(ForkchoiceUpdated { payload_status: PayloadStatus { status: Invalid { validation_error: "links to previously rejected block" }, latest_valid_hash: Some(0x2f980576711e3617a5e4d83dd539548ec0f7792007d505a3d2e9674833af2d7c) }, payload_id: None })))) }
2024-06-18T19:26:46.930929Z DEBUG consensus::auto: Sent fork choice update state=ForkchoiceState { head_block_hash: 0x57700ab36067e231ccb16296e229615f2d3ad5e51d902f36c1c2bd01da5bbdbc, safe_block_hash: 0x57700ab36067e231ccb16296e229615f2d3ad5e51d902f36c1c2bd01da5bbdbc, finalized_block_hash: 0x57700ab36067e231ccb16296e229615f2d3ad5e51d902f36c1c2bd01da5bbdbc }
2024-06-18T19:26:46.931699Z DEBUG blockchain_tree: Block hash not found in block indices block_hash=0x57700ab36067e231ccb16296e229615f2d3ad5e51d902f36c1c2bd01da5bbdbc
2024-06-18T19:26:46.931968Z DEBUG consensus::engine: Syncing to new target target=0x57700ab36067e231ccb16296e229615f2d3ad5e51d902f36c1c2bd01da5bbdbc
2024-06-18T19:26:46.932033Z  INFO reth_node_events::node: Live sync in progress, downloading blocks remaining_blocks=1 target_block_hash=0x57700ab36067e231ccb16296e229615f2d3ad5e51d902f36c1c2bd01da5bbdbc
2024-06-18T19:26:46.932066Z  INFO reth_node_events::node: Received forkchoice updated message when syncing head_block_hash=0x57700ab36067e231ccb16296e229615f2d3ad5e51d902f36c1c2bd01da5bbdbc safe_block_hash=0x57700ab36067e231ccb16296e229615f2d3ad5e51d902f36c1c2bd01da5bbdbc finalized_block_hash=0x57700ab36067e231ccb16296e229615f2d3ad5e51d902f36c1c2bd01da5bbdbc
2024-06-18T19:26:46.932216Z DEBUG consensus::auto: Forkchoice update returned SYNCING, waiting for VALID fcu_response=OnForkChoiceUpdated { forkchoice_status: Syncing, fut: Left(Ready(Some(Ok(ForkchoiceUpdated { payload_status: PayloadStatus { status: Syncing, latest_valid_hash: None }, payload_id: None })))) }
2024-06-18T19:26:46.932433Z DEBUG consensus::auto: Sent fork choice update state=ForkchoiceState { head_block_hash: 0x57700ab36067e231ccb16296e229615f2d3ad5e51d902f36c1c2bd01da5bbdbc, safe_block_hash: 0x57700ab36067e231ccb16296e229615f2d3ad5e51d902f36c1c2bd01da5bbdbc, finalized_block_hash: 0x57700ab36067e231ccb16296e229615f2d3ad5e51d902f36c1c2bd01da5bbdbc }
2024-06-18T19:26:46.933480Z  WARN consensus::engine: Bad block with existing invalid ancestor hash=0x57700ab36067e231ccb16296e229615f2d3ad5e51d902f36c1c2bd01da5bbdbc invalid_ancestor=Header { parent_hash: 0x2f980576711e3617a5e4d83dd539548ec0f7792007d505a3d2e9674833af2d7c, ommers_hash: 0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347, beneficiary: 0x0000000000000000000000000000000000000000, state_root: 0xf09d8f7da5bc5036f8dd9536c953e2212390a46fb3e553ece2b7d419131537b1, transactions_root: 0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421, receipts_root: 0x0000000000000000000000000000000000000000000000000000000000000000, withdrawals_root: Some(0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421), logs_bloom: 0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000, difficulty: 2, number: 1, gas_limit: 30000000, gas_used: 0, timestamp: 1718738805, mix_hash: 0x0000000000000000000000000000000000000000000000000000000000000000, nonce: 0, base_fee_per_gas: Some(875000000), blob_gas_used: Some(0), excess_blob_gas: Some(0), parent_beacon_block_root: Some(0x0000000000000000000000000000000000000000000000000000000000000000), requests_root: None, extra_data: 0x }
2024-06-18T19:26:46.933907Z ERROR consensus::auto: Forkchoice update returned invalid response fcu_response=OnForkChoiceUpdated { forkchoice_status: Invalid, fut: Left(Ready(Some(Ok(ForkchoiceUpdated { payload_status: PayloadStatus { status: Invalid { validation_error: "links to previously rejected block" }, latest_valid_hash: Some(0x2f980576711e3617a5e4d83dd539548ec0f7792007d505a3d2e9674833af2d7c) }, payload_id: None })))) }
2024-06-18T19:26:47.930558Z DEBUG consensus::auto: Sent fork choice update state=ForkchoiceState { head_block_hash: 0x81e77a48931ecd1b005d237bd3b5da90b61c741544cff53b4729fdf846779a08, safe_block_hash: 0x81e77a48931ecd1b005d237bd3b5da90b61c741544cff53b4729fdf846779a08, finalized_block_hash: 0x81e77a48931ecd1b005d237bd3b5da90b61c741544cff53b4729fdf846779a08 }
2024-06-18T19:26:47.930764Z  INFO reth::cli: Status connected_peers=0 freelist=3 latest_block=0
2024-06-18T19:26:47.931213Z DEBUG blockchain_tree: Block hash not found in block indices block_hash=0x81e77a48931ecd1b005d237bd3b5da90b61c741544cff53b4729fdf846779a08
2024-06-18T19:26:47.931445Z DEBUG consensus::engine: Syncing to new target target=0x81e77a48931ecd1b005d237bd3b5da90b61c741544cff53b4729fdf846779a08
2024-06-18T19:26:47.931468Z  INFO reth_node_events::node: Live sync in progress, downloading blocks remaining_blocks=1 target_block_hash=0x81e77a48931ecd1b005d237bd3b5da90b61c741544cff53b4729fdf846779a08
2024-06-18T19:26:47.931509Z DEBUG consensus::auto: Forkchoice update returned SYNCING, waiting for VALID fcu_response=OnForkChoiceUpdated { forkchoice_status: Syncing, fut: Left(Ready(Some(Ok(ForkchoiceUpdated { payload_status: PayloadStatus { status: Syncing, latest_valid_hash: None }, payload_id: None })))) }
2024-06-18T19:26:47.931531Z DEBUG consensus::auto: Sent fork choice update state=ForkchoiceState { head_block_hash: 0x81e77a48931ecd1b005d237bd3b5da90b61c741544cff53b4729fdf846779a08, safe_block_hash: 0x81e77a48931ecd1b005d237bd3b5da90b61c741544cff53b4729fdf846779a08, finalized_block_hash: 0x81e77a48931ecd1b005d237bd3b5da90b61c741544cff53b4729fdf846779a08 }
2024-06-18T19:26:47.931562Z  INFO reth_node_events::node: Received forkchoice updated message when syncing head_block_hash=0x81e77a48931ecd1b005d237bd3b5da90b61c741544cff53b4729fdf846779a08 safe_block_hash=0x81e77a48931ecd1b005d237bd3b5da90b61c741544cff53b4729fdf846779a08 finalized_block_hash=0x81e77a48931ecd1b005d237bd3b5da90b61c741544cff53b4729fdf846779a08
2024-06-18T19:26:47.932924Z  WARN consensus::engine: Bad block with existing invalid ancestor hash=0x81e77a48931ecd1b005d237bd3b5da90b61c741544cff53b4729fdf846779a08 invalid_ancestor=Header { parent_hash: 0x2f980576711e3617a5e4d83dd539548ec0f7792007d505a3d2e9674833af2d7c, ommers_hash: 0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347, beneficiary: 0x0000000000000000000000000000000000000000, state_root: 0xf09d8f7da5bc5036f8dd9536c953e2212390a46fb3e553ece2b7d419131537b1, transactions_root: 0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421, receipts_root: 0x0000000000000000000000000000000000000000000000000000000000000000, withdrawals_root: Some(0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421), logs_bloom: 0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000, difficulty: 2, number: 1, gas_limit: 30000000, gas_used: 0, timestamp: 1718738805, mix_hash: 0x0000000000000000000000000000000000000000000000000000000000000000, nonce: 0, base_fee_per_gas: Some(875000000), blob_gas_used: Some(0), excess_blob_gas: Some(0), parent_beacon_block_root: Some(0x0000000000000000000000000000000000000000000000000000000000000000), requests_root: None, extra_data: 0x }
2024-06-18T19:26:47.933142Z ERROR consensus::auto: Forkchoice update returned invalid response fcu_response=OnForkChoiceUpdated { forkchoice_status: Invalid, fut: Left(Ready(Some(Ok(ForkchoiceUpdated { payload_status: PayloadStatus { status: Invalid { validation_error: "links to previously rejected block" }, latest_valid_hash: Some(0x2f980576711e3617a5e4d83dd539548ec0f7792007d505a3d2e9674833af2d7c) }, payload_id: None })))) }
2024-06-18T19:26:48.932243Z DEBUG consensus::auto: Sent fork choice update state=ForkchoiceState { head_block_hash: 0xe8f9572cc77eafe5b64c94d8c5bedfc35da249f5ce8dac089b3dc74b64b0755e, safe_block_hash: 0xe8f9572cc77eafe5b64c94d8c5bedfc35da249f5ce8dac089b3dc74b64b0755e, finalized_block_hash: 0xe8f9572cc77eafe5b64c94d8c5bedfc35da249f5ce8dac089b3dc74b64b0755e }
2024-06-18T19:26:48.933010Z DEBUG blockchain_tree: Block hash not found in block indices block_hash=0xe8f9572cc77eafe5b64c94d8c5bedfc35da249f5ce8dac089b3dc74b64b0755e
2024-06-18T19:26:48.933258Z DEBUG consensus::engine: Syncing to new target target=0xe8f9572cc77eafe5b64c94d8c5bedfc35da249f5ce8dac089b3dc74b64b0755e
2024-06-18T19:26:48.933341Z  INFO reth_node_events::node: Live sync in progress, downloading blocks remaining_blocks=1 target_block_hash=0xe8f9572cc77eafe5b64c94d8c5bedfc35da249f5ce8dac089b3dc74b64b0755e
2024-06-18T19:26:48.933369Z  INFO reth_node_events::node: Received forkchoice updated message when syncing head_block_hash=0xe8f9572cc77eafe5b64c94d8c5bedfc35da249f5ce8dac089b3dc74b64b0755e safe_block_hash=0xe8f9572cc77eafe5b64c94d8c5bedfc35da249f5ce8dac089b3dc74b64b0755e finalized_block_hash=0xe8f9572cc77eafe5b64c94d8c5bedfc35da249f5ce8dac089b3dc74b64b0755e
2024-06-18T19:26:48.933400Z DEBUG consensus::auto: Forkchoice update returned SYNCING, waiting for VALID fcu_response=OnForkChoiceUpdated { forkchoice_status: Syncing, fut: Left(Ready(Some(Ok(ForkchoiceUpdated { payload_status: PayloadStatus { status: Syncing, latest_valid_hash: None }, payload_id: None })))) }
2024-06-18T19:26:48.933417Z DEBUG consensus::auto: Sent fork choice update state=ForkchoiceState { head_block_hash: 0xe8f9572cc77eafe5b64c94d8c5bedfc35da249f5ce8dac089b3dc74b64b0755e, safe_block_hash: 0xe8f9572cc77eafe5b64c94d8c5bedfc35da249f5ce8dac089b3dc74b64b0755e, finalized_block_hash: 0xe8f9572cc77eafe5b64c94d8c5bedfc35da249f5ce8dac089b3dc74b64b0755e }
2024-06-18T19:26:48.934753Z  WARN consensus::engine: Bad block with existing invalid ancestor hash=0xe8f9572cc77eafe5b64c94d8c5bedfc35da249f5ce8dac089b3dc74b64b0755e invalid_ancestor=Header { parent_hash: 0x2f980576711e3617a5e4d83dd539548ec0f7792007d505a3d2e9674833af2d7c, ommers_hash: 0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347, beneficiary: 0x0000000000000000000000000000000000000000, state_root: 0xf09d8f7da5bc5036f8dd9536c953e2212390a46fb3e553ece2b7d419131537b1, transactions_root: 0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421, receipts_root: 0x0000000000000000000000000000000000000000000000000000000000000000, withdrawals_root: Some(0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421), logs_bloom: 0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000, difficulty: 2, number: 1, gas_limit: 30000000, gas_used: 0, timestamp: 1718738805, mix_hash: 0x0000000000000000000000000000000000000000000000000000000000000000, nonce: 0, base_fee_per_gas: Some(875000000), blob_gas_used: Some(0), excess_blob_gas: Some(0), parent_beacon_block_root: Some(0x0000000000000000000000000000000000000000000000000000000000000000), requests_root: None, extra_data: 0x }
2024-06-18T19:26:48.935430Z ERROR consensus::auto: Forkchoice update returned invalid response fcu_response=OnForkChoiceUpdated { forkchoice_status: Invalid, fut: Left(Ready(Some(Ok(ForkchoiceUpdated { payload_status: PayloadStatus { status: Invalid { validation_error: "links to previously rejected block" }, latest_valid_hash: Some(0x2f980576711e3617a5e4d83dd539548ec0f7792007d505a3d2e9674833af2d7c) }, payload_id: None })))) }
2024-06-18T19:26:49.490991Z DEBUG reth::cli: shutting down gracefully
2024-06-18T19:26:49.492094Z  INFO reth::cli: Wrote network peers to file peers_file="/home/zhan4987/.local/share/reth/dev/known-peers.json"
2024-06-18T19:26:49.494201Z DEBUG reth_tasks: gracefully shut down

Platform(s)

Linux (x86)

What version/commit are you on?

reth Version: 1.0.0-rc.2-dev
Commit SHA: d786b45
Build Timestamp: 2024-06-18T19:25:14.442219648Z
Build Features: jemalloc,optimism
Build Profile: debug

What database version are you on?

Current database version: 2

Which chain / network are you on?

dev

What type of node are you running?

Archive (default)

What prune config do you use, if any?

No response

If you've built Reth from source, provide the full command you used

cargo run --bin op-reth --features optimism -- node --dev --dev.block-time 1s

Code of Conduct

• I agree to follow the Code of Conduct

[Troublor]

After a quick inspection, it seems AutoSeal does not set the receipts_root in the block header:

reth/crates/consensus/auto-seal/src/lib.rs

Lines 403 to 416 in 2a5c93f

	let Block { mut header, body, .. } = block.block;
	let body = BlockBody { transactions: body, ommers, withdrawals, requests };
	trace!(target: "consensus::auto", ?execution_outcome, ?header, ?body, "executed block, calculating state root and completing header");
	// calculate the state root
	header.state_root = db.state_root(execution_outcome.state())?;
	trace!(target: "consensus::auto", root=?header.state_root, ?body, "calculated root");
	// finally insert into storage
	self.insert_new_block(header.clone(), body);
	// set new header with hash that should have been updated by insert_new_block
	let new_header = header.seal(self.best_hash);

[jsvisa]

I think it maybe related with

reth/crates/primitives/src/proofs.rs

Lines 43 to 71 in 7bfc604

	pub fn calculate_receipt_root_optimism(
	receipts: &[ReceiptWithBloom],
	chain_spec: &reth_chainspec::ChainSpec,
	timestamp: u64,
	) -> B256 {
	// There is a minor bug in op-geth and op-erigon where in the Regolith hardfork,
	// the receipt root calculation does not include the deposit nonce in the receipt
	// encoding. In the Regolith Hardfork, we must strip the deposit nonce from the
	// receipts before calculating the receipt root. This was corrected in the Canyon
	// hardfork.
	if chain_spec.is_fork_active_at_timestamp(reth_chainspec::Hardfork::Regolith, timestamp) &&
	!chain_spec.is_fork_active_at_timestamp(reth_chainspec::Hardfork::Canyon, timestamp)
	{
	let receipts = receipts
	.iter()
	.cloned()
	.map(|mut r| {
	r.receipt.deposit_nonce = None;
	r
	})
	.collect::<Vec<_>>();
	return ordered_trie_root_with_encoder(receipts.as_slice(), |r, buf| {
	r.encode_inner(buf, false)
	})
	}
	ordered_trie_root_with_encoder(receipts, |r, buf| r.encode_inner(buf, false))
	}

try to investigate and fix it

[jsvisa]

@Troublor you're right, the receipts_root were not calculated. I'm trying to fix it.

[Troublor]

@jsvisa FYI, I tried to added this code into build_and_execute function, it seems to work:

// calculate the receipts root
header.receipts_root = execution_outcome
    .receipts
    .optimism_root_slow(0, &chain_spec, header.timestamp)
    .expect("receipts root should never be none");

[jsvisa]

yeah, but this will fail for ethereum's dev node, we need to add a feature to seperate them.