Skip to content

Summarize dgoss results after execution completes#123

Merged
ianpittwood merged 8 commits intomainfrom
dgoss-result-summaries
Mar 21, 2025
Merged

Summarize dgoss results after execution completes#123
ianpittwood merged 8 commits intomainfrom
dgoss-result-summaries

Conversation

@ianpittwood
Copy link
Contributor

@ianpittwood ianpittwood commented Feb 27, 2025

Closes #102
Follows #122

  • Model Goss's JSON output as a series of Pydantic models.
  • Summarizes dgoss results in a rich.Table by image name, version, and type with the total test duration and number of successful, failed, and skipped tests.
  • Print condensed summary of any failed tests split up by their associated unique image identifier.
  • Modify error identification and collection behavior in dgoss
    • No longer propagate errors via raise, instead return them as part of a tuple with the test results.
    • If dgoss exits non-zero, assume it's due to test failures if the command produced a valid JSON output.
    • If dgoss exits non-zero and output cannot be parsed as JSON, consider the command to have experienced a runtime error. Create and collect a BakeryToolRuntimeError to return with the commmand.

This approach has some shortcomings we'll need to consider:

  • This implementation locks us into JSON outputs to the end user. goss cannot produce multiple result outputs. If we wish to allow other result outputs, we will need to heavily modify our implementation to expect a variety of different output formats or we will need to convert the JSON into other formats ourselves. I assume team's may want a more common standard reported back, such as JUnit, so we will need to consider how/if we want to perform that conversion or if we should just ignore summarizing results in the CLI.
  • Result summary output is not suppressed by --quiet. We may want to consider if it should be.

@github-actions
Copy link

github-actions bot commented Feb 27, 2025

Test Results

301 tests   299 ✅  1m 9s ⏱️
  1 suites    2 💤
  1 files      0 ❌

Results for commit 8be45c2.

♻️ This comment has been updated with latest results.

@bschwedler
Copy link
Contributor

Thank you for doing this! The output of bakery run dgoss is much cleaner output is much easier to look through now:
image

@bschwedler
Copy link
Contributor

It could be useful to include the list the skipped tests when the --verbose flag is set.

This also makes me think about whether we should break up the tests into sections or files that are included conditionally instead of peppering this everywhere

    skip: {% raw %}{{ if eq .Env.IMAGE_TYPE "min" }}true{{ else }}false{{ end }}{% endraw %}

Base automatically changed from dgoss-error-collection to main March 21, 2025 17:33
Fix one more failing snyk  test
Add test_failures retrieval property to GossJsonReportCollection
- Print summary of failed tests if there are failures
- Print runtime errors if any commands fail to execute
- If any tests or commands fail, exit 1
@ianpittwood ianpittwood force-pushed the dgoss-result-summaries branch from d8b76a0 to 5100d6f Compare March 21, 2025 18:04
- Show success in bold green if there are no failed tests.
- Show failures in bold red if there are failures. Use bright black italics for zero.
- Show skips in bold yellow if tests were skipped. Use bright black italics for zero.
- Add a total tests column.
@ianpittwood ianpittwood merged commit d0539c2 into main Mar 21, 2025
3 checks passed
@ianpittwood ianpittwood deleted the dgoss-result-summaries branch March 21, 2025 19:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Summarize dgoss test results for all images tested

2 participants