In-process tests and automatic builder variant tests #91

karim-agha · 2025-05-30T21:11:36Z

📝 Summary

This PR closes the following issues:

Few things happened here:

RBuilder running in-process

Now we have a new type called LocalInstance that runs an entire RBuilder instance in-process. It uses IPC transport instead of HTTP for RPC and EngineAPI. This LocalInstance represents op-rbuilder node and can be configured by giving it an optional NodeConfig or RBuilderArgs structures. Examples:

let fb_node = LocalInstance::flashblocks().await?;
let standard_node = LocalInstance::standard().await?;
let fb_with_args = LocalInstance::new::<FlashblocksBuilder>(OpRBuilderArgs { ... }).await?;
let standard_with_args_and_node_config = LocalInstance::new_with_config::<StandardBuilder>(
   OpRBuilderArgs { ... }, 
   NodeConfig<OpChainSpec> {...}
).await?;

Better `OpRbuilderArgs` defaults

The Default implementation of the cli arguments type and all its descendants will use the defaults specified by clap attributes instead of rust default values.

Chain Driver

When interacting with a LocalInstance we have now ChainDriver that is used to trigger production of new blocks, get latest blocks, etc. It can be either instantiated by ChainDriver::new(&local_instance) or local_instance.driver().

Better transaction pool events inspector

In LocalInstance all transaction pool events are recorded in a strongly typed manner inside TransactionPoolObserver. We record their entire state history, so we can later validate the correct order of transitions. The public API for this is:

driver.build_new_block().await?;
assert!(rbuilder.pool().is_dropped(*reverted_bundle.tx_hash()));
assert!(rbuilder.pool().is_pending(*reverted_bundle.tx_hash()));
// ...
rbuilder.pool().history(tx_hash).unwrap().contains(...)
rbuilder.pool().tx_status(tx_hash).

Transactions have states that are instance of TransactionEvent from the transaction-pool crate in reth. Namely:

pub enum TransactionEvent {
    Pending,
    Queued,
    Mined(B256),
    Replaced(TxHash),
    Discarded,
    Invalid,
    Propagated(Arc<Vec<PropagateKind>>),
}

Automatic unit tests generation for both standard and flashblocks builders.

When writing unit or integration tests instead of using #[tokio::test] async test_name() use the rb_test macro and let the LocalInstance be injected by the macro by the test infrastructure. In this mode a sample unit test would look as follows:

/// This test ensures that the transaction size limit is respected.
/// We will set limit to 1 byte and see that the builder will not include any transactions.
#[rb_test]
async fn data_availability_tx_size_limit(rbuilder: LocalInstance) -> eyre::Result<()> {
    let driver = rbuilder.driver().await?;

    // Set (max_tx_da_size, max_block_da_size), with this case block won't fit any transaction
    let call = rbuilder
        .provider()
        .await?
        .raw_request::<(i32, i32), bool>("miner_setMaxDASize".into(), (1, 0))
        .await?;
    assert!(call, "miner_setMaxDASize should be executed successfully");

    let unfit_tx = driver
        .transaction()
        .random_valid_transfer()
        .with_max_priority_fee_per_gas(50)
        .send()
        .await?;
    let block = driver.build_new_block().await?;

    // tx should not be included because we set the tx_size_limit to 1
    assert!(
        !block.includes(unfit_tx.tx_hash()),
        "transaction should not be included in the block"
    );

    Ok(())
}

then when you run cargo test it will automatically generate both variants of this test:

$ cargo test data_availability_tx_size_limit
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.29s
     Running unittests src/lib.rs (target/debug/deps/op_rbuilder-7194e393c77c3c5d)

running 2 tests
test tests::data_availability::data_availability_tx_size_limit_standard ... ok
test tests::data_availability::data_availability_tx_size_limit_flashblocks ... ok

test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 33 filtered out; finished in 3.60s

     Running unittests src/main.rs (target/debug/deps/op_rbuilder-df17a41dd57b9b91)

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 7 filtered out; finished in 0.00s

This macro can be configured to either run in only flashbots mode, only standard mode, use custom RBuilderArgs use custom NodeConfig, etc. Examples of using this macro are in the tests directory.

There is also a new macro called if_flashblocks! that allows you to specify a block of code that runs only under the flashblocks variant of a test. Example:

#[rb_test]
async fn test_name(rbuilder: LocalInstance) -> eyre::Result<()> {
  println!("This message will appear in both variants of the test");

  if_flashblocks! {
    println!("This message will appear only in flashblocks version of the test");
  }
}

Use this macro if the differences between variants are minimal and don't merit writing two versions of the test, otherwise use #[rb_test(standard)] and #[rb_test(flashbots)] to narrow down the generation to one builder.

Test tracing

You can now see detailed logs of your unit tests by setting the TEST_TRACE env variable. That will print out all reth and op-rbuilder logs to console when they're running. Usage:

$ TEST_TRACE=on cargo test
$ TEST_TRACE=error cargo test
$ TEST_TRACE=info cargo test
$ TEST_TRACE=debug cargo test
$ TEST_TRACE=trace cargo test

❌ Known issues ❌

Most flashblocks variants of tests are failing. When using standard send_transaction ETH api, transactions are not included in produced blocks. They are also not included in the flashblocks that get generated. Fixing those tests are outside of the scope of this PR and will happen in a follow-up PR. I've intentionally left the tests failing now so we are aware of this. The tests landscape now looks as follows:

cargo test
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.30s
     Running unittests src/lib.rs (target/debug/deps/op_rbuilder-7194e393c77c3c5d)

running 35 tests
test builders::generator::tests::test_job_deadline ... ok
test builders::generator::tests::test_block_cell_immediate_value ... ok
test builders::generator::tests::test_block_cell_update_value ... ok
test tx_signer::test::test_sign_transaction ... ok
test builders::generator::tests::test_block_cell_multiple_waiters ... ok
test builders::generator::tests::test_block_cell_wait_for_value ... ok
test tests::data_availability::data_availability_tx_size_limit_flashblocks ... ok
test tests::revert::revert_protection_bundle_flashblocks ... FAILED
test tests::revert::revert_protection_bundle_range_limits_standard ... ok
test tests::revert::revert_protection_allow_reverted_transactions_without_bundle_flashblocks ... FAILED
test tests::txpool::pending_pool_limit_flashblocks ... FAILED
test tests::data_availability::data_availability_block_size_limit_standard ... ok
test tests::revert::revert_protection_disabled_bundle_endpoint_error_standard ... ok
test tests::revert::revert_protection_disabled_bundle_endpoint_error_flashblocks ... ok
test tests::data_availability::data_availability_block_size_limit_flashblocks ... ok
test tests::data_availability::data_availability_block_fill_standard ... ok
test tests::revert::revert_protection_check_transaction_receipt_status_message_standard ... ok
test tests::revert::revert_protection_bundle_standard ... ok
test tests::data_availability::data_availability_block_fill_flashblocks ... FAILED
test tests::data_availability::data_availability_tx_size_limit_standard ... ok
test tests::revert::revert_protection_bundle_range_limits_flashblocks ... ok
test tests::revert::revert_protection_allow_reverted_transactions_without_bundle_standard ... ok
test tests::txpool::pending_pool_limit_standard ... ok
test tests::revert::revert_protection_disabled_flashblocks ... FAILED
test tests::revert::revert_protection_monitor_transaction_gc_standard ... ok
test tests::smoke::chain_produces_blocks_flashblocks ... FAILED
test tests::revert::revert_protection_check_transaction_receipt_status_message_flashblocks ... ok
test builders::generator::tests::test_payload_generator ... ok
test tests::ordering::fee_priority_ordering_standard ... ok
test tests::revert::revert_protection_disabled_standard ... ok
test tests::revert::revert_protection_monitor_transaction_gc_flashblocks ... FAILED
test tests::smoke::chain_produces_blocks_standard ... ok
test tests::smoke::produces_blocks_under_load_within_deadline_flashblocks ... ok
test tests::smoke::produces_blocks_under_load_within_deadline_standard ... ok
test tests::flashblocks::chain_produces_blocks_flashblocks ... FAILED

failures:
    tests::data_availability::data_availability_block_fill_flashblocks
    tests::flashblocks::chain_produces_blocks_flashblocks
    tests::revert::revert_protection_allow_reverted_transactions_without_bundle_flashblocks
    tests::revert::revert_protection_bundle_flashblocks
    tests::revert::revert_protection_disabled_flashblocks
    tests::revert::revert_protection_monitor_transaction_gc_flashblocks
    tests::smoke::chain_produces_blocks_flashblocks
    tests::txpool::pending_pool_limit_flashblocks

test result: FAILED. 27 passed; 8 failed; 0 ignored; 0 measured; 0 filtered out; finished in 31.17s

✅ I have completed the following steps:

Run make lint
Run make test
Added tests (if applicable)

…load

karim-agha · 2025-06-03T14:02:40Z

Update: Mosts tests are passing now.
Debugging last few failures:

running 35 tests
test builders::generator::tests::test_block_cell_immediate_value ... ok
test builders::generator::tests::test_job_deadline ... ok
test builders::generator::tests::test_block_cell_update_value ... ok
test tx_signer::test::test_sign_transaction ... ok
test builders::generator::tests::test_block_cell_wait_for_value ... ok
test builders::generator::tests::test_block_cell_multiple_waiters ... ok
test builders::generator::tests::test_payload_generator ... ok
test tests::revert::revert_protection_disabled_bundle_endpoint_error_flashblocks ... ok
test tests::revert::revert_protection_disabled_bundle_endpoint_error_standard ... ok
test tests::revert::revert_protection_bundle_range_limits_standard ... ok
test tests::data_availability::data_availability_block_size_limit_flashblocks ... ok
test tests::data_availability::data_availability_tx_size_limit_standard ... ok
test tests::data_availability::data_availability_block_size_limit_standard ... ok
test tests::data_availability::data_availability_block_fill_standard ... ok
test tests::revert::revert_protection_bundle_range_limits_flashblocks ... ok
test tests::data_availability::data_availability_tx_size_limit_flashblocks ... ok
test tests::data_availability::data_availability_block_fill_flashblocks ... FAILED
test tests::revert::revert_protection_check_transaction_receipt_status_message_flashblocks ... ok
test tests::txpool::pending_pool_limit_standard ... ok
test tests::revert::revert_protection_check_transaction_receipt_status_message_standard ... ok
test tests::txpool::pending_pool_limit_flashblocks ... ok
test tests::revert::revert_protection_bundle_flashblocks ... ok
test tests::revert::revert_protection_bundle_standard ... ok
test tests::revert::revert_protection_allow_reverted_transactions_without_bundle_standard ... ok
test tests::revert::revert_protection_allow_reverted_transactions_without_bundle_flashblocks ... ok
test tests::revert::revert_protection_disabled_standard ... ok
test tests::revert::revert_protection_disabled_flashblocks ... ok
test tests::ordering::fee_priority_ordering_standard ... ok
test tests::smoke::produces_blocks_under_load_within_deadline_standard ... ok
test tests::smoke::produces_blocks_under_load_within_deadline_flashblocks ... ok
test tests::revert::revert_protection_monitor_transaction_gc_standard ... ok
test tests::revert::revert_protection_monitor_transaction_gc_flashblocks ... ok
test tests::smoke::chain_produces_blocks_standard ... ok
test tests::smoke::chain_produces_blocks_flashblocks ... ok
test tests::flashblocks::chain_produces_blocks_flashblocks ... FAILED

failures:

---- tests::data_availability::data_availability_block_fill_flashblocks stdout ----

thread 'tests::data_availability::data_availability_block_fill_flashblocks' panicked at crates/op-rbuilder/src/tests/data_availability.rs:102:5:
unfit tx should not be in block
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

---- tests::flashblocks::chain_produces_blocks_flashblocks stdout ----

thread 'tests::flashblocks::chain_produces_blocks_flashblocks' panicked at crates/op-rbuilder/src/tests/framework/txs.rs:137:18:
Failed to get transaction count: Transport(BackendGone)


failures:
    tests::data_availability::data_availability_block_fill_flashblocks
    tests::flashblocks::chain_produces_blocks_flashblocks

test result: FAILED. 33 passed; 2 failed; 0 ignored; 0 measured; 0 filtered out; finished in 31.01s

karim-agha · 2025-06-03T14:15:46Z

All tests are passing now:

running 34 tests
test builders::generator::tests::test_block_cell_immediate_value ... ok
test builders::generator::tests::test_block_cell_update_value ... ok
test builders::generator::tests::test_job_deadline ... ok
test tx_signer::test::test_sign_transaction ... ok
test builders::generator::tests::test_block_cell_multiple_waiters ... ok
test builders::generator::tests::test_block_cell_wait_for_value ... ok
test builders::generator::tests::test_payload_generator ... ok
test tests::revert::revert_protection_disabled_bundle_endpoint_error_standard ... ok
test tests::revert::revert_protection_disabled_bundle_endpoint_error_flashblocks ... ok
test tests::data_availability::data_availability_tx_size_limit_standard ... ok
test tests::data_availability::data_availability_block_size_limit_flashblocks ... ok
test tests::data_availability::data_availability_tx_size_limit_flashblocks ... ok
test tests::data_availability::data_availability_block_fill_standard ... ok
test tests::revert::revert_protection_bundle_range_limits_flashblocks ... ok
test tests::revert::revert_protection_bundle_range_limits_standard ... ok
test tests::data_availability::data_availability_block_size_limit_standard ... ok
test tests::revert::revert_protection_check_transaction_receipt_status_message_flashblocks ... ok
test tests::revert::revert_protection_check_transaction_receipt_status_message_standard ... ok
test tests::txpool::pending_pool_limit_standard ... ok
test tests::txpool::pending_pool_limit_flashblocks ... ok
test tests::revert::revert_protection_bundle_standard ... ok
test tests::revert::revert_protection_bundle_flashblocks ... ok
test tests::revert::revert_protection_allow_reverted_transactions_without_bundle_standard ... ok
test tests::revert::revert_protection_allow_reverted_transactions_without_bundle_flashblocks ... ok
test tests::revert::revert_protection_disabled_standard ... ok
test tests::revert::revert_protection_disabled_flashblocks ... ok
test tests::ordering::fee_priority_ordering_standard ... ok
test tests::smoke::produces_blocks_under_load_within_deadline_standard ... ok
test tests::smoke::produces_blocks_under_load_within_deadline_flashblocks ... ok
test tests::revert::revert_protection_monitor_transaction_gc_standard ... ok
test tests::revert::revert_protection_monitor_transaction_gc_flashblocks ... ok
test tests::smoke::chain_produces_blocks_standard ... ok
test tests::smoke::chain_produces_blocks_flashblocks ... ok
test tests::flashblocks::chain_produces_blocks_flashblocks ... ok

test result: ok. 34 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 34.98s

     Running unittests src/main.rs (target/debug/deps/op_rbuilder-81f1f02591ba5b0a)

running 7 tests
test builders::generator::tests::test_job_deadline ... ok
test builders::generator::tests::test_block_cell_immediate_value ... ok
test builders::generator::tests::test_block_cell_update_value ... ok
test tx_signer::test::test_sign_transaction ... ok
test builders::generator::tests::test_block_cell_multiple_waiters ... ok
test builders::generator::tests::test_block_cell_wait_for_value ... ok
test builders::generator::tests::test_payload_generator ... ok

test result: ok. 7 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 4.66s

   Doc-tests op_rbuilder

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

karim-agha · 2025-06-06T14:35:08Z

Superseded by #132
#68 will be fixed in a follow-up PR

karim-agha added 16 commits May 28, 2025 15:59

First tests migrated and working

b4285ee

checkpoint

0bc2636

More tests are passing

5207e43

all revert tests are passing

2745a21

txpool tests are passing

18fbebf

All standard tests are passing

cb3b132

All standard tests are passing

a807fcb

All tests are passing

03475cb

Default values in local instance use cli defaults

1b77e79

Checkpoint before adding proc macros

7bbea21

Removed old test machinery

4615e25

rb-test macro works

402fe19

flattened tests directory tree

729c4ff

flattened tests directory tree

da0d602

Merged with main

dc25243

if_flashblocks macro

8c342fa

karim-agha requested review from SozinM, avalonche and ferranbt as code owners May 30, 2025 21:11

karim-agha added 4 commits June 3, 2025 13:47

Giving the block builder time to build a block between FCU and GetPay…

13605ab

…load

merge conflicts resolved

f1a6e1b

merge conflicts resolved

2654a3f

Most tests passing

fb4a7c9

all tests are passing except 1

44939ef

karim-agha added 2 commits June 3, 2025 14:17

all tests are passing

3a53fc9

lint

b1501fa

karim-agha closed this Jun 6, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

In-process tests and automatic builder variant tests #91

In-process tests and automatic builder variant tests #91

Uh oh!

karim-agha commented May 30, 2025 •

edited

Loading

Uh oh!

karim-agha commented Jun 3, 2025

Uh oh!

karim-agha commented Jun 3, 2025

Uh oh!

karim-agha commented Jun 6, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

In-process tests and automatic builder variant tests #91

In-process tests and automatic builder variant tests #91

Uh oh!

Conversation

karim-agha commented May 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📝 Summary

RBuilder running in-process

Better OpRbuilderArgs defaults

Chain Driver

Better transaction pool events inspector

Automatic unit tests generation for both standard and flashblocks builders.

Test tracing

❌ Known issues ❌

✅ I have completed the following steps:

Uh oh!

karim-agha commented Jun 3, 2025

Uh oh!

karim-agha commented Jun 3, 2025

Uh oh!

karim-agha commented Jun 6, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

karim-agha commented May 30, 2025 •

edited

Loading

Better `OpRbuilderArgs` defaults