Improve Test Framework

[0xForerunner]

We currently have a pretty good array of integration tests, but I think it's worth exploring some more options.

#146 is a good start to getting more robust e2e tests through the kurtosis devnet. I think it's worth while ideating on how we can leverage the kurtosis devnet to probe more specific test cases.

The current integration test suite has a few shortcomings:

• tendency to randomly fail at times and I'm not particularly sure what the root cause of this is.
• It's also probably not best practice to use user installed dependencies like is currently being done with op-reth.
• Relying on log parsing to make assertions is pretty fragile.

We could implement the current test suite entirely within a kurtosis devnet which would likely give us a bit more reliability. Additionally many of the current integration tests would work quite well using mocks instead.

I don't currently have the best solution at hand, but I'll take some time to investigate to see if I can find a good path forwards.

[ferranbt]

Hey, I appreciate the thoughtful issue about testing. Let me share some context about our current approach that might help frame the discussion:

While Kurtosis offers a comprehensive environment, we've found it comes with tradeoffs:

• Setup times are lengthy (minutes rather than seconds)
• Non-deterministic port allocation makes automation challenging
• We've experienced reliability issues, particularly with rollup-boost integration

Our current integration test philosophy is to:

• The tests are run like unit tests
• Make them as deterministic as possible
• Allow precise control over test conditions (e.g., simulating builder failures, builder sends invalid blocks)
• Make tests highly reliable and reproducible

The fine-grained control in our current tests (like mocking the CL node) gives us deterministic behavior that might be difficult to achieve in a full Kurtosis environment without a non-trivial amount of hacks. I am open to have this thought challenged though.

That said, I'd be interested to hear more about the specific test cases you're looking to cover. Understanding your testing goals could help us determine if we need to adjust our approach.

I do see value in complementing our current suite with long-lived smoke/chaos tests using Kurtosis for full end-to-end validation. But, this would complement rather than replace the current integration tests.

For your specific questions about the framework:

tendency to randomly fail at times and I'm not particularly sure what the root cause of this is.

The integration tests have become more stable lately. We still occasionally see failures on the first block, but this has improved significantly. I'd be interested to hear about specific GitHub Actions runs where you're seeing failures. That said, I'm not convinced Kurtosis would necessarily solve these stability issues - it might introduce different ones given its complexity.

It's also probably not best practice to use user installed dependencies like is currently being done with op-reth.

This depends on what problem we're trying to solve. While I understand the concern about external dependencies, I don't necessarily consider our current approach bad practice in this context.

Relying on log parsing to make assertions is pretty fragile.

I agree log parsing isn't ideal, but this challenge would exist regardless of whether we use Kurtosis or our current setup. At some point, we need to capture specific events in the system. This is more about how we instrument our tests rather than the environment they run in.

Additionally many of the current integration tests would work quite well using mocks instead.

Isn't this what we're already doing with the CL mocker?

[avalonche]

I see the current integration tests to be complimentary to the kurtosis setup. I think the two serves different use cases:

As Ferran mentioned, the current integration tests:

• are much quicker to run
• easier to setup test conditions
• more reliable and deterministic

It's useful for testing new features against the live builder without excessive mocks with external systems. I think there are valid points for improvement for the integration framework:

It's also probably not best practice to use user installed dependencies like is currently being done with op-reth.

I agree that this makes running integration tests locally not very reliable due to potentially mismatched versioning with user installed dependencies. I believe this can be solved by extracting the validation logic to a function in /tester.

Relying on log parsing to make assertions is pretty fragile.

Since the purpose of the integration test is to be more reliable / deterministic I agree we should try move away from log parsing assertions to metrics / chain state assertions.

The kurtosis devnet setup gives us:

• a comprehensive e2e setup
• test behaviour outside of block building e.g. forwarded / multiplexed calls
• performance / compatibility with op-node

which tests a different aspect of the system and doesn't necessarily have to test the same properties we want in the integration test framework.

The design decision I see here is between kurtosis vs another devnet test framework (e.g. https://github.com/flashbots/builder-playground) to simulate real network conditions. We've encountered problems with kurtosis being slow to setup, and also a lot of the system internals are rather opaque and hard to triage as it is all written in starlark.

[0xForerunner]

Admittedly, this is quite a challenging environment to test in, and kurtosis may not be the final answer haha. I think the current integration tests do a pretty good job all things considered. One rather easy improvement would be to replace local dependencies with containerized deps. See testcontainers for a potentially good alternative.

Additionally many of the current integration tests would work quite well using mocks instead.

Isn't this what we're already doing with the CL mocker?

Ah yeah what I meant was that some of the current integration tests could be replaced with mocked unit tests, without the heavy integration test dependencies. This is a rather small issue though.

[ferranbt]

See testcontainers for a potentially good alternative.

This is a good note. Actually, I used something similar in Go a while back.

[0xKitsune]

Closing as completed in #165