Fix block processing bottleneck issues for networks with sub 1-second block times

[shahnami]

Describe the bug

The process_new_blocks function has a bottleneck when processing networks with sub-1-second block times (e.g., Arbitrum Sepolia). The system starts generating duplicate and out-of-order block detection errors when processing high-frequency blocks (>250 blocks/minute).

The BlockTracker::record_block method checks block order against the last recorded block in its history. However, with concurrent record_block calls, blocks may be added to the tracker's internal queue in a different order than they're numbered, causing the tracker to see "out-of-order" blocks that are actually just arriving concurrently.

Steps to reproduce

1. Set up Arbitrum Sepolia network with monitor arbitrum_sepolia_stylus.json
2. Use the default network configuration with block_time_ms: 250
3. Run the monitor and observe logs for several minutes
4. Errors should appear showing missed blocks and out-of-order detections

Application logs

It works fine when setting block time to 1 second:


2025-10-29T16:39:31.621934Z  INFO openzeppelin_monitor::services::blockwatcher::service: Started block watcher for network: arbitrum_sepolia
2025-10-29T16:39:31.621970Z  INFO openzeppelin_monitor: Service started. Press Ctrl+C to shutdown
2025-10-29T16:40:00.528261Z  INFO process_new_blocks: openzeppelin_monitor::services::blockwatcher::service: Processing blocks:
        Last processed block: 209736002
        Latest confirmed block: 209739569
        Start block: 209739496 (skipped 3493 blocks)
        Confirmations required: 12
        Max past blocks: 73 network: "arbitrum_sepolia"
2025-10-29T16:40:04.286409Z  INFO process_new_blocks: openzeppelin_monitor::services::blockwatcher::service: Processed 74 blocks in 4048ms network: "arbitrum_sepolia"


It fails when setting block time to 250ms:


2025-10-29T16:41:00.743050Z  INFO process_new_blocks: openzeppelin_monitor::services::blockwatcher::service: Processing blocks:
        Last processed block: 209739569
        Latest confirmed block: 209739806
        Start block: 209739570
        Confirmations required: 12
        Max past blocks: 253 network: "arbitrum_sepolia"
2025-10-29T16:41:03.136006Z ERROR process_new_blocks: openzeppelin_monitor::utils::logging::error: Error occurred, Missed block 209739695, trace_id: cb9dae3e-850d-4ca7-be3a-93e1946dd85c, timestamp: 2025-10-29T16:41:03.135989+00:00 network: "arbitrum_sepolia"
2025-10-29T16:41:03.136050Z ERROR process_new_blocks: openzeppelin_monitor::utils::logging::error: Error occurred, Missed block 209739696, trace_id: 36def967-01b4-42df-83b1-f418a74bb2ec, timestamp: 2025-10-29T16:41:03.136048+00:00 network: "arbitrum_sepolia"
2025-10-29T16:41:03.137356Z ERROR process_new_blocks: openzeppelin_monitor::utils::logging::error: Error occurred, Missed block 209739802, trace_id: 8dd45e42-90e5-46df-850c-752fd45aa5fd, timestamp: 2025-10-29T16:41:03.137355+00:00 network: "arbitrum_sepolia"
2025-10-29T16:41:03.137369Z ERROR process_new_blocks: openzeppelin_monitor::utils::logging::error: Error occurred, Missed block 209739803, trace_id: 54029899-8be8-4453-b213-6fd8405e4b7b, timestamp: 2025-10-29T16:41:03.137368+00:00 network: "arbitrum_sepolia"
2025-10-29T16:41:03.137380Z ERROR process_new_blocks: openzeppelin_monitor::utils::logging::error: Error occurred, Missed block 209739804, trace_id: ef53278a-bf16-4f67-8ba0-edb05416faaa, timestamp: 2025-10-29T16:41:03.137379+00:00 network: "arbitrum_sepolia"
2025-10-29T16:41:03.137392Z ERROR process_new_blocks: openzeppelin_monitor::utils::logging::error: Error occurred, Missed block 209739805, trace_id: 419e462b-4f5d-448b-8a41-3a592825bf7c, timestamp: 2025-10-29T16:41:03.137391+00:00 network: "arbitrum_sepolia"
2025-10-29T16:41:03.137426Z ERROR process_new_blocks: openzeppelin_monitor::utils::logging::error: Error occurred, Out of order or duplicate block detected: received 209739805 after 209739806, trace_id: e7236141-66c4-45dd-81a7-c8a5d22e0b57, timestamp: 2025-10-29T16:41:03.137424+00:00 network: "arbitrum_sepolia"
2025-10-29T16:41:03.137454Z ERROR process_new_blocks: openzeppelin_monitor::utils::logging::error: Error occurred, Out of order or duplicate block detected: received 209739804 after 209739805, trace_id: 94edd2b6-5e30-41d7-bbec-a972d0d567b0, timestamp: 2025-10-29T16:41:03.137452+00:00 network: "arbitrum_sepolia"
2025-10-29T16:41:03.137480Z ERROR process_new_blocks: openzeppelin_monitor::utils::logging::error: Error occurred, Out of order or duplicate block detected: received 209739803 after 209739804, trace_id: 9aa0b817-0534-466e-b859-7561a76eae7c, timestamp: 2025-10-29T16:41:03.137478+00:00 network: "arbitrum_sepolia"
2025-10-29T16:41:03.137516Z ERROR process_new_blocks: openzeppelin_monitor::utils::logging::error: Error occurred, Out of order or duplicate block detected: received 209739802 after 209739803, trace_id: c7d20418-e443-4ef7-9beb-969b5480735a, timestamp: 2025-10-29T16:41:03.137515+00:00 network: "arbitrum_sepolia"
2025-10-29T16:41:03.137544Z ERROR process_new_blocks: openzeppelin_monitor::utils::logging::error: Error occurred, Out of order or duplicate block detected: received 209739801 after 209739802, trace_id: 3144eae0-cec6-4a1d-b57e-36357d022e14, timestamp: 2025-10-29T16:41:03.137543+00:00 network: "arbitrum_sepolia"
2025-10-29T16:41:03.137571Z ERROR process_new_blocks: openzeppelin_monitor::utils::logging::error: Error occurred, Out of order or duplicate block detected: received 209739800 after 209739801, trace_id: a34ce50e-5a80-4a75-b1d0-9993ddc81762, timestamp: 2025-10-29T16:41:03.137569+00:00 network: "arbitrum_sepolia"
2025-10-29T16:41:03.137600Z ERROR process_new_blocks: openzeppelin_monitor::utils::logging::error: Error occurred, Out of order or duplicate block detected: received 209739799 after 209739800, trace_id: 6a72a2ec-f94a-4af9-a0b4-d342296e81c1, timestamp: 2025-10-29T16:41:03.137599+00:00 network: "arbitrum_sepolia"
2025-10-29T16:41:03.137627Z ERROR process_new_blocks: openzeppelin_monitor::utils::logging::error: Error occurred, Out of order or duplicate block detected: received 209739798 after 209739799, trace_id: 268f87ac-a366-4cea-bd03-686ff32c1f0d, timestamp: 2025-10-29T16:41:03.137625+00:00 network: "arbitrum_sepolia"
2025-10-29T16:41:03.137653Z ERROR process_new_blocks: openzeppelin_monitor::utils::logging::error: Error occurred, Out of order or duplicate block detected: received 209739797 after 209739798, trace_id: 34e35891-b695-44a4-a207-1262df87431c, timestamp: 2025-10-29T16:41:03.137651+00:00 network: "arbitrum_sepolia"
2025-10-29T16:41:03.137677Z ERROR process_new_blocks: openzeppelin_monitor::utils::logging::error: Error occurred, Out of order or duplicate block detected: received 209739796 after 209739797, trace_id: 7f416697-9aa7-4322-a980-e582b7299e7e, timestamp: 2025-10-29T16:41:03.137676+00:00 network: "arbitrum_sepolia"
2025-10-29T16:41:03.137707Z ERROR process_new_blocks: openzeppelin_monitor::utils::logging::error: Error occurred, Out of order or duplicate block detected: received 209739795 after 209739796, trace_id: 59c6409c-8e43-48a1-83e4-fc1bbccfb6f6, timestamp: 2025-10-29T16:41:03.137705+00:00 network: "arbitrum_sepolia"
2025-10-29T16:41:03.137729Z ERROR process_new_blocks: openzeppelin_monitor::utils::logging::error: Error occurred, Out of order or duplicate block detected: received 209739794 after 209739795, trace_id: 7c8cf84b-c767-4bbb-adc6-b1c08e8fc8d7, timestamp: 2025-10-29T16:41:03.137727+00:00 network: "arbitrum_sepolia"

Platform(s)

Mac (Apple Silicon)

Deployment Type

Built from source

Any other context that we need to know specific to the bug?

Use a private RPC provider as you'll get throttled quickly on public ones.

Version Information

1.1.0

Monitor Configuration

{
    "name": "Arbitrum Sepolia Stylus",
    "paused": false,
    "networks": [
        "arbitrum_sepolia"
    ],
    "addresses": [
        {
            "address": "0xDAF4A50D2281497737BC3F4B8cF5f4c2fA3C159c",
            "contract_spec": [
                {
                    "inputs": [],
                    "name": "retrieve",
                    "outputs": [
                        {
                            "internalType": "uint256",
                            "name": "",
                            "type": "uint256"
                        }
                    ],
                    "stateMutability": "view",
                    "type": "function"
                },
                {
                    "inputs": [
                        {
                            "internalType": "uint256",
                            "name": "num",
                            "type": "uint256"
                        }
                    ],
                    "name": "store",
                    "outputs": [],
                    "stateMutability": "nonpayable",
                    "type": "function"
                }
            ]
        }
    ],
    "match_conditions": {
        "functions": [
            {
                "signature": "store(uint256)",
                "expression": null
            }
        ],
        "events": [],
        "transactions": [
            {
                "status": "Success",
                "expression": null
            }
        ]
    },
    "trigger_conditions": [],
    "triggers": [
        "evm_large_transfer_usdc_slack"
    ]
}

Network Type

EVM

Build Command

No response

Code of Conduct

• I agree to follow the Code of Conduct