Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

As a GEOPM contributor who makes pull requests, I would like test failures to be reported in an easily-discoverable location #2401

Open
dannosliwcd opened this issue Jun 23, 2022 · 0 comments
Labels
feature Description of a new feature as a story test

Comments

@dannosliwcd
Copy link
Contributor

Is your feature request related to a problem? Please describe.
Some of our tests are automatically executed on every pull request. If the suite fails, it requires non-intuitive steps to figure out why:

  1. Figure out which test(s) failed. Use GitHub's virtual search bar (not your browser's search, because the log file has a virtual scrollbar) to look for FAIL: . Put the failing test name in your clipboard, or remember it for later.
  2. Scroll down and click the dropdown button near the end of the build logs to reveal the contents of the test logs
  3. Use the virtual search bar to find the test identified in step 1. This where the failure context can usually be found

Describe the solution you'd like
I would like to see all test failures in an easy-to-find location. It looks like GitHub recently added support for job summaries, and this seems to be one of the intended use cases.

One way this may be achieved is by writing pass/fail summaries to a file, e.g.:

run: make check | tee ${RUNNER_TEMP}/check_service_summary.txt
...
run: make check | tee ${RUNNER_TEMP}/check_basedir_summary.txt

Then converting those summaries to markdown in a later step. E.g.:

    - name: Summarize tests
      shell: bash
      run: |
           echo "### Test Summary" >> $GITHUB_STEP_SUMMARY
           echo "**Passing tests**: $(grep '^PASS: ' ${RUNNER_TEMP}/check_service_summary.txt ${RUNNER_TEMP}/check_basedir_summary.txt | wc -l)" >> $GITHUB_STEP_SUMMARY
           for testname in $(awk -e '/^FAIL: / { print "service/" $2 }' ${RUNNER_TEMP}/check_service_summary.txt) $(awk -e '/^FAIL: / { print $2 }' ${RUNNER_TEMP}/check_basedir_summary.txt); do
             echo "**Failed test**: $testname" >> $GITHUB_STEP_SUMMARY
             echo '```' >> $GITHUB_STEP_SUMMARY
             cat ${testname}.log >> $GITHUB_STEP_SUMMARY || true
             echo '```' >> $GITHUB_STEP_SUMMARY
           done

Describe alternatives you've considered
Alternatively, it looks like there are off-the-shelf GitHub actions we can import into the project, that do a lot of fancy test reporting. I think those would require that we configure our test runners to output machine-readable reports. I imagine that's possible for the Python and C++ test libraries we use, but it would take a little more investigation.

Additional context
The proposed solution and the alternative aren't perfect. The report location is still on a separate "Summary" page, which isn't the one that appears when you click the big red X on a failing suite. But it is at least all located on a single page and only two clicks away from the PR after the proposal.

The proposal renders something like (with some fake injected failures):
image

@dannosliwcd dannosliwcd added test feature Description of a new feature as a story labels Jun 23, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature Description of a new feature as a story test
Projects
None yet
Development

No branches or pull requests

1 participant