-
Notifications
You must be signed in to change notification settings - Fork 157
flowey: vmm_tests: introduce step to verify all tests run on at least one runner #2094
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull Request Overview
This PR introduces infrastructure to verify test coverage by creating a step that parses nextest JUnit XML files to collect test names and ensures all tests run on at least one runner. The implementation adds nextest list command generation, JSON artifact publishing, and a new verification node for comparing test results.
- Adds a new verification step to parse JUnit XML files and extract test names
- Extends nextest integration to generate list commands and publish JSON artifacts
- Updates test result publishing to include nextest list JSON output
Reviewed Changes
Copilot reviewed 12 out of 12 changed files in this pull request and generated 5 comments.
Show a summary per file
File | Description |
---|---|
flowey/flowey_lib_hvlite/src/_jobs/verify_all_tests_run.rs | New node that parses JUnit XML files to extract test names for verification |
flowey/flowey_lib_common/src/run_cargo_nextest_run.rs | Extended to generate nextest list commands and capture JSON output |
flowey/flowey_lib_common/src/gen_cargo_nextest_list_cmd.rs | New module for generating cargo nextest list commands |
flowey/flowey_lib_common/src/publish_test_results.rs | Updated to publish nextest list JSON as artifacts |
.github/workflows/openvmm-pr.yaml | Generated workflow updates reflecting the new artifact publishing |
.github/workflows/openvmm-pr-release.yaml | Generated workflow updates reflecting the new artifact publishing |
Very cool idea, but I think it needs a bunch of refactoring to clean up the cases where steps have been combined that really should be separate. |
This PR resolves #2010
With #1991 it is possible (however unlikely) for a contributor to introduce a test that doesn't end up running on any of the vmm_test jobs in CI. This PR introduces a step that verifies the list of all tests built as part of a CI run are run on at least one vmm_test job.
Here are a list of changes that make this possible:
nextest list
command which dumps the tests that are built as part of the nextest archive.nextest list
and the corresponding junit xml that was produced from all vmm_test runs. This node then gathers a full set of tests from both artifacts and compares them to make sure the sets are equal.