Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor: Abstract fixture type collection; Parametrize tests over fixture formats; Per-format sub-folder output #358

Merged

Conversation

marioevz
Copy link
Member

@marioevz marioevz commented Dec 8, 2023

🗒️ Description

Modifies the fixture collector to collect and dump any kind of test (BlockchainTest, StateTest, or TransactionTest).

The fixtures are now dumped in separate subfolders depending on the fixture type.
E.g. BlockchainTest tests are dumped at fixtures/blockchain_tests, with current test folder output as previously.

🔗 Related Issues

None

✅ Checklist

  • All: Set appropriate labels for the changes.
  • All: Considered squashing commits to improve commit history.
  • All: Added an entry to CHANGELOG.md.
  • All: Considered updating the online docs in the ./docs/ directory.
  • Tests: Included the type and version of evm t8n tool used to locally execute test cases: e.g., ref with commit hash or geth 1.13.1-stable-3f40e65.
  • Tests: Ran mkdocs serve locally and verified the auto-generated docs for new tests in the Test Case Reference are correctly formatted.

@marioevz marioevz added the scope:fw Scope: Framework (evm|tools|forks|pytest) label Dec 8, 2023
@marioevz marioevz marked this pull request as ready for review December 8, 2023 05:08
@marioevz marioevz force-pushed the feat/statetest-fixture-collector branch from f38a6f6 to d749226 Compare December 9, 2023 16:57
@marioevz
Copy link
Member Author

marioevz commented Dec 9, 2023

Fixtures for test types are now also generated automatically (blockchain_test and state_test), thanks to this fantastic trick described in this comment: pytest-dev/pytest#2424 (comment)

@marioevz
Copy link
Member Author

Ready for review @spencer-tb @danceratopz , let me know what you think!

Copy link
Member

@danceratopz danceratopz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks awesome so far, still going through it 🙂.

One suggestion would be marioevz#12.

danceratopz and others added 4 commits December 12, 2023 11:06
* feat(pytest): allow filtering by fixture format via pytest mark

* feat(fw): give unset fixture format a more verbose name
* feat(pytest): remove items which are pre-merge and have hive format

* chore: make exception string clearer; fix whitespace
@marioevz marioevz changed the title feature: Refactor Fixture Collector to use abstract BaseFixture class to dump fixtures refactor: Abstract fixture type collection; Parametrize tests over fixture formats Dec 13, 2023
@marioevz marioevz changed the title refactor: Abstract fixture type collection; Parametrize tests over fixture formats refactor: Abstract fixture type collection; Parametrize tests over fixture formats; Per-format sub-folder output Dec 13, 2023
@marioevz marioevz merged commit c63094b into ethereum:feat/statetest Dec 13, 2023
5 checks passed
marioevz added a commit that referenced this pull request Dec 22, 2023
…xture formats; Per-format sub-folder output (#358)

* tools: Move fixture collector to spec

* refactor: Fixtures collector

* all: remove `BaseTestConfig`

* fix: all filler tests

* docs: update

* tox: spelling

* feat: Parametrize the supported fixture types for each test type

* fix: many tests

* fix: remaining tests

* fix: fixture name case

* fix: Indirect parametrize using fixture format

* feat: Dynamic generation of spec test fixtures

* fix: type-checking

* fix: imports

* feat(pytest): allow filtering by fixture format via pytest mark (#12)

* feat(pytest): allow filtering by fixture format via pytest mark

* feat(fw): give unset fixture format a more verbose name

* feat(pytest): only paramatrize post-merge forks with hive format (#13)

* feat(pytest): remove items which are pre-merge and have hive format

* chore: make exception string clearer; fix whitespace

* fix(pytest): test_filler: remove `enable_hive` completely

* fix: Remove unused `STATE_TEST_HIVE`

---------

Co-authored-by: danceratopz <danceratopz@gmail.com>
marioevz added a commit that referenced this pull request Jan 6, 2024
…xture formats; Per-format sub-folder output (#358)

* tools: Move fixture collector to spec

* refactor: Fixtures collector

* all: remove `BaseTestConfig`

* fix: all filler tests

* docs: update

* tox: spelling

* feat: Parametrize the supported fixture types for each test type

* fix: many tests

* fix: remaining tests

* fix: fixture name case

* fix: Indirect parametrize using fixture format

* feat: Dynamic generation of spec test fixtures

* fix: type-checking

* fix: imports

* feat(pytest): allow filtering by fixture format via pytest mark (#12)

* feat(pytest): allow filtering by fixture format via pytest mark

* feat(fw): give unset fixture format a more verbose name

* feat(pytest): only paramatrize post-merge forks with hive format (#13)

* feat(pytest): remove items which are pre-merge and have hive format

* chore: make exception string clearer; fix whitespace

* fix(pytest): test_filler: remove `enable_hive` completely

* fix: Remove unused `STATE_TEST_HIVE`

---------

Co-authored-by: danceratopz <danceratopz@gmail.com>
marioevz added a commit that referenced this pull request Jan 10, 2024
…xture formats; Per-format sub-folder output (#358)

* tools: Move fixture collector to spec

* refactor: Fixtures collector

* all: remove `BaseTestConfig`

* fix: all filler tests

* docs: update

* tox: spelling

* feat: Parametrize the supported fixture types for each test type

* fix: many tests

* fix: remaining tests

* fix: fixture name case

* fix: Indirect parametrize using fixture format

* feat: Dynamic generation of spec test fixtures

* fix: type-checking

* fix: imports

* feat(pytest): allow filtering by fixture format via pytest mark (#12)

* feat(pytest): allow filtering by fixture format via pytest mark

* feat(fw): give unset fixture format a more verbose name

* feat(pytest): only paramatrize post-merge forks with hive format (#13)

* feat(pytest): remove items which are pre-merge and have hive format

* chore: make exception string clearer; fix whitespace

* fix(pytest): test_filler: remove `enable_hive` completely

* fix: Remove unused `STATE_TEST_HIVE`

---------

Co-authored-by: danceratopz <danceratopz@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
scope:fw Scope: Framework (evm|tools|forks|pytest)
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants