feat(pytest): add two commands that consume blockchain test fixtures

[danceratopz]

🗒️ Description

This PR:

1. Adds a new command: consume, which acts as a base command with sub-commands corresponding to the different way clients can consume json test fixtures.
2. Adds a consume direct command that takes a directory containing JSON fixture files and an evm tool as command-line arguments (currently geth evm tool only).
   For example:

   fill --output fixtures-no-paris -n auto -k "not Paris"  # geth master evm tool expects "Merge" not "Paris"
   consume direct --input=fixtures-no-paris -n auto
   

3. Adds a consume rlp command that implements the hive/simulators/ethereum/consensus simulator as a pytest plugin. If a hive server is started in --dev mode, such as:

   ./hive --dev --client go-ethereum --client-file configs/cancun.yaml --docker.output --sim.loglevel 4
   

   then the python port of the consensus simulator can be executed (not the HIVE_SIMULATOR evn var must be set) as: example:

   export HIVE_SIMULATOR=http://127.0.0.1:3000
   # then, e.g., run on a release with the `consume_via_rlp` simulator:
   consume rlp --input=https://github.com/ethereum/execution-spec-tests/releases/download/v2.1.1/fixtures.tar.gz -k selfdestruct -n auto
   # or, e.g., run fill and then execute the result in the `consume_via_rlp` simulator:
   fill --output=fixtures --fork Shanghai -n auto -k "not hive and warm_coinbase";
   consume rlp --input=fixtures -v -n auto
   # or
   fill --output=stdout --fork Shanghai -k "not hive and warm_coinbase" | consume rlp -v  # piping doesn't support -n auto
   

The skeleton is there to add a re-write of the "pyspec" (consume-via-engine-api) simulator, but this is currently not implemented.

consume direct

This is another pytest-based command that dynamically generates test cases for every fixture in the fixture files located in the fixture directory. Each test case executes the blocktest command using --single-test (see ethereum/go-ethereum#28421) to execute the tests on a per-fixture level:

If --run/--single-fixture is not available in blocktest, it falls back to the usual blocktest behaviour of executing all fixtures within the provided json file:

Further ideas:

• Write metadata to fixtures' _info sections using fill that can be used when generating the tests in fill, for example, which eips are required to run the test; xfail if not supported.
• In CI/CD this can be used with pytest's --junit-xml=test_results.xml option and the publish-test-results Github Action, for example. Can also be combined with xdist to get parallelisation out of the box.

This is how consume fits in within execution-spec-tests and alongside other ways of consuming fixtures:

consume rlp

Requires: https://github.com/danceratopz/hive.py/tree/fix/client/files-def-in-post-with-files

A re-write of the hive/simulators/ethereum/consensus simulator as a pytest plugin.

Lots to talk about and still lots to fix 😄

Currently works with go-ethereum and besu. This is also possible:

./hive --dev --client go-ethereum,besu

then consume rlp will parametrize each blockchain test by client_type.

Dockerized Usage with Hive

The simulator be ran (without --dev mode) in Hive with:

./hive --sim ethereum/eest-block-rlp  --client-file configs/cancun.yaml --client go-ethereum --docker.output

using this Dockerfile:
https://github.com/danceratopz/hive/blob/mek-it-snek/simulators/ethereum/eest-block-rlp/Dockerfile#L1-L15

This currently has no testing and no docs.

TODO:

• Update doc. This will require improving our language as we'll have two commands that "test": fill and consume. Need to make it clear that fill generates test cases and consume executes them as a module test.
• Add fill | consume direct and merge feat(pytest): enable piping of fill's json to consume rlp danceratopz/execution-spec-tests#26
• Try and get all clients to work with consume rlp.
• Try to understand why xdist with -n > 1hangs (sometimes).

🔗 Related Issues

✅ Checklist

• All: Set appropriate labels for the changes.
• All: Considered squashing commits to improve commit history.
• All: Added an entry to CHANGELOG.md.
• All: Considered updating the online docs in the ./docs/ directory.
• Tests: Included the type and version of evm t8n tool used to locally execute test cases: e.g., ref with commit hash or geth 1.13.1-stable-3f40e65.
• Tests: Ran mkdocs serve locally and verified the auto-generated docs for new tests in the Test Case Reference are correctly formatted.

[spencer-tb]

Really appreciate this PR - its going to be very useful in the future once every client has a blocktest-like sub-command! Let's get this CI up and running :D

All further ideas are great as well! We should add the coverage to this weekly testing report for each client team

[marioevz]

Few comments, but overall it's really impressive!

[marioevz]

LGTM! I think we should be able to merge directly to main.