Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(pytest): add pytest-html plugin for html reports #537

Merged
merged 20 commits into from
May 7, 2024

Conversation

danceratopz
Copy link
Member

@danceratopz danceratopz commented May 1, 2024

🗒️ Description

  • Adds the pytest-html plugin to EEST's dependencies so that html test reports can be generated for fill executions.
  • Generates html report by default. Default location is the root of fill's output directory (--output, default is ./fixtures).
  • Adds fill command-line options and t8n/solc versions to the report Environment section.
  • Adds links to the JSON fixture and --evm-dump-dir for each test (see screengrab).

Demo screengrab of report below.

TODO

  • 1. Add relevant config/metadata (solc version, t8n version) to "Environment" Table (see screenshot).
  • 2. Enable by default and write to fixture output directory (we could use click to specify output directory, but perhaps the plugin has an option to help manage this).
  • 3.a. Add new "Test fixture" column to the "Summary" table (or in "Links" column?). Add link to the locally generated json fixture, if test is pass.
  • 3.b. Add new "Debug output dir" column to the "Summary" table with link (or in "Links" column?), if option is enabled.

Other tasks beyond the scope of this PR:

  • 4. Enable for framework tests?
  • 5. Use in CI/CD?

Considerations for use in CI/CD:

  • Filling with Cancun generates an html report of 5.9 M, even if all tests pass.
  • There seems to be negligible impact on runtimes with reporting enabled.

HTML Report Features Demo

fill_html_reports

🔗 Related Issues

None

✅ Checklist

  • All: Set appropriate labels for the changes.
  • All: Considered squashing commits to improve commit history.
  • All: Added an entry to CHANGELOG.md.
  • All: Considered updating the online docs in the ./docs/ directory.
  • Tests: All converted JSON/YML tests from ethereum/tests have been added to converted-ethereum-tests.txt.
  • Tests: Included the type and version of evm t8n tool used to locally execute test cases: e.g., ref with commit hash or geth 1.13.1-stable-3f40e65.
  • Tests: Ran mkdocs serve locally and verified the auto-generated docs for new tests in the Test Case Reference are correctly formatted.

@danceratopz danceratopz added scope:pytest Scope: Pytest plugins type:feat type: Feature labels May 1, 2024
@danceratopz danceratopz marked this pull request as ready for review May 6, 2024 21:43
@danceratopz danceratopz requested a review from marioevz May 6, 2024 21:43
Copy link
Member

@marioevz marioevz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very nice! thanks!

@danceratopz danceratopz merged commit 25d4755 into main May 7, 2024
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
scope:pytest Scope: Pytest plugins type:feat type: Feature
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants