Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(fw): run evm blocktest on generated fixtures #325

Conversation

danceratopz
Copy link
Member

@danceratopz danceratopz commented Oct 3, 2023

🗒️ Description

This PR optionally enables the execution of evm blocktest on each generated JSON fixture file. The check is ran as a function in the FixtureCollector in a module-scope pytest fixture (so is not executed per test function or per test-case).

If --t8n-dump-dir is additionally specified, the output of the evm blocktest command is also written there along with a verify_fixtures.sh script to re-run the command.

There's a couple of other changes to the t8n-dump directories:

  • The JSON fixtures always get copied to the corresponding t8n-dump directory.
  • The t8n-dump directory structure:
    • Adds an additional directory level for the output of the evm test command and a copy of the json fixtures.
    • Stays as flat as possible, but the base directory is now named to match the location of the test case within ./tests.
        /tmp/evmone/berlin__eip2930_access_list__test_acl__test_access_list/
        ├── fork_Berlin
        │   ├── 0
        │   │   ├── args.py
        │   │   ├── input
        │   │   │   ├── alloc.json
        │    ....
        │   ├── test_access_list.json
        │   ├── verify_fixtures_args.py
        │   ├── verify_fixtures_returncode.txt
        │   ├── verify_fixtures.sh
        │   ├── verify_fixtures_stderr.txt
        │   └── verify_fixtures_stdout.txt
         ...
        ```
    

Examples:

  1. No fixture checking performed
    fill
    
  2. Use first "evm" binary in path as a t8n tool and as an evm *test tool (same binary used)
    fill --verify-fixtures
    
  3. Use a different evm binary as an evm test tool (first evm binary in path used as t8n tool; --verify-fixtures not necessary)
    fill --verify-fixtures-bin=~/code/github/marioevz/go-ethereum/build/bin/evm
    
  4. Explicitly set both t8n and test tools; dump debug data
    fill --evm-bin=~/code/github/ethereum/evmone/build/bin/evmone-t8n --verify-fixtures-bin=~/code/github/marioevz/go-ethereum/build/bin/evm  --evm-dump-dir=/tmp/evmone
    

🔗 Related Issues

✅ Checklist

  • All: Set appropriate labels for the changes.
  • All: Considered squashing commits to improve commit history.
  • All: Added an entry to CHANGELOG.md.
  • All: Considered updating the online docs in the ./docs/ directory.
  • Tests: Included the type and version of evm t8n tool used to locally execute test cases: e.g., ref with commit hash or geth 1.13.1-stable-3f40e65.
  • Tests: Ran mkdocs serve locally and verified the auto-generated docs for new tests in the Test Case Reference are correctly formatted.

@danceratopz danceratopz added scope:evm Scope: evm_transition_tool package type:feat type: Feature scope:fw Scope: Framework (evm|tools|forks|pytest) labels Oct 3, 2023
Copy link
Member

@marioevz marioevz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks very good! Just a few comments to keep in mind.

src/evm_transition_tool/transition_tool.py Outdated Show resolved Hide resolved
src/evm_transition_tool/transition_tool.py Outdated Show resolved Hide resolved
src/pytest_plugins/test_filler/test_filler.py Outdated Show resolved Hide resolved
Copy link
Member Author

@danceratopz danceratopz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes after discussing with Mario.

src/pytest_plugins/test_filler/test_filler.py Outdated Show resolved Hide resolved
src/pytest_plugins/test_filler/test_filler.py Outdated Show resolved Hide resolved
src/evm_transition_tool/transition_tool.py Show resolved Hide resolved
src/pytest_plugins/test_filler/test_filler.py Outdated Show resolved Hide resolved
Copy link
Member

@marioevz marioevz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM :)

@danceratopz danceratopz changed the title feat(fw): run evm statetest or blocktest on generated fixtures feat(fw): run evm blocktest on generated fixtures Oct 12, 2023
@danceratopz danceratopz merged commit 35b6f34 into ethereum:main Oct 12, 2023
5 checks passed
@danceratopz danceratopz deleted the feat/run-evm-statetest-or-blocktest-on-generated-fixtures branch October 12, 2023 08:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
scope:evm Scope: evm_transition_tool package scope:fw Scope: Framework (evm|tools|forks|pytest) type:feat type: Feature
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants