feature: use pytest to collect and execute test fillers

[danceratopz]

This PR (work in progress) ports the test-filling process implemented in the custom tf framework to the pytest framework. Pytest becomes responsible for collecting fillers (test functions) and managing their execution to generate fixtures from the spec defined within them.

This is achieved by:

1. Defining a test spec in a test function (as before),
2. Storing them from within the test function in a test-function-scoped helper pyest fixture, and,
3. Executing the stored TestSpec/filler (and writing the fixtures) in pytest's pytest_runtest_call() hook.

Pytest is a widely used Python framework and offers several advantages out-of-the-box:

• Separation of test filler collection and execution; pytest --collect-only can be used to explore which fillers are available. Use pytest --collect-only -v for a list with docstrings.
• Pytest will continue execution if a filler fails (tf stops upon error).
• Parametrization of tests (the ethereum fork is also parametrized) which gives more context to test filler errors.
• Easier filtering and specification of a subset of test fillers during development (e.g. pytest -k warm_coinbase or pytest -k Shanghai).
• Easily drop into the debugger via pytest --pdb upon test filler error.
• pytest-xdist can be used to execute test fillers concurrently.

Some demos and screenshots of pytest with execution-spec-tests can be found here (but note that these were generated from a different and now very out-of-date branch).

Quick start for working with this branch:

git clone git@github.com:danceratopz/execution-spec-tests.git
git checkout feature/use-pytest-to-collect-and-hydrate-test-fillers
python -m venv ./venv
source venv/bin/activate
pip install -e .
pytest --collect-only
pytest -v
# or run a newly ported filler explicitly
pytest -v fillers/eips/eip3651.py

To do list for framework related tasks:

• Port ReferenceSpec version checking - pytest: port eip spec version checking to a pytest plugin danceratopz/execution-spec-tests#12
• Remove unused tf-related framework code (e.g., src/ethereum_test_filling_tool/filler.py)
• Edit tf the entry point to run pytest.
• Add --latest-fork command-line option - pytest: add latest fork parameter and exit gracefully upon an invalid fork danceratopz/execution-spec-tests#6
• Update README and docs docs: mega endgame execution spec test doc revamp danceratopz/execution-spec-tests#16
• Add VS Code project settings config
• Convert direct parametrizations of fork to a fixture.
• ...

To do list for porting test fillers to pytest format:

• fillers/eips/eip3651.py
• fillers/eips/eip3855.py
• fillers/eips/eip3860.py - Pytest: convert eip-3860 fillers to pytest format (mark 2) danceratopz/execution-spec-tests#13
• fillers/eips/eip4844/datahash_opcode.py - fillers/eip-4844: update blob tx tests for pytest danceratopz/execution-spec-tests#7
• fillers/eips/eip4844/excess_data_gas.py - fillers/eip-4844: update blob tx tests for pytest danceratopz/execution-spec-tests#7
• fillers/example/acl_example.py
• fillers/example/yul_example.py
• fillers/security/selfdestruct_balance_bug.py - fillers: convert remaining fillers to pytest format danceratopz/execution-spec-tests#14
• fillers/vm/chain_id.py
• fillers/vm/dup.py - fillers: convert remaining fillers to pytest format danceratopz/execution-spec-tests#14
• fillers/withdrawals/withdrawals.py - fillers: convert remaining fillers to pytest format danceratopz/execution-spec-tests#14

[lightclient]

I think this is a great idea! My only request is to try and keep the core dev who doesn't write python on a regular basis in the back of your mind as you develop the filler interface. One problem with the retesteth fillers IMO is that they became a DSL that only those who were regularly writing them could use / understand. One goal of this project should be for core devs to i) come in and fairly easily understand what is and isn't being tested and ii) write tests for a proposal if your team is busy with other things or their proposal isn't yet a priority.

[marioevz]

LGTM!

[spencer-tb]

LGTM :D