You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem? Please describe.
Some of our tests are automatically executed on every pull request. If the suite fails, it requires non-intuitive steps to figure out why:
Figure out which test(s) failed. Use GitHub's virtual search bar (not your browser's search, because the log file has a virtual scrollbar) to look for FAIL: . Put the failing test name in your clipboard, or remember it for later.
Scroll down and click the dropdown button near the end of the build logs to reveal the contents of the test logs
Use the virtual search bar to find the test identified in step 1. This where the failure context can usually be found
Describe the solution you'd like
I would like to see all test failures in an easy-to-find location. It looks like GitHub recently added support for job summaries, and this seems to be one of the intended use cases.
One way this may be achieved is by writing pass/fail summaries to a file, e.g.:
run: make check | tee ${RUNNER_TEMP}/check_service_summary.txt
...
run: make check | tee ${RUNNER_TEMP}/check_basedir_summary.txt
Then converting those summaries to markdown in a later step. E.g.:
Describe alternatives you've considered
Alternatively, it looks like there are off-the-shelf GitHub actions we can import into the project, that do a lot of fancy test reporting. I think those would require that we configure our test runners to output machine-readable reports. I imagine that's possible for the Python and C++ test libraries we use, but it would take a little more investigation.
Additional context
The proposed solution and the alternative aren't perfect. The report location is still on a separate "Summary" page, which isn't the one that appears when you click the big red X on a failing suite. But it is at least all located on a single page and only two clicks away from the PR after the proposal.
The proposal renders something like (with some fake injected failures):
The text was updated successfully, but these errors were encountered:
Is your feature request related to a problem? Please describe.
Some of our tests are automatically executed on every pull request. If the suite fails, it requires non-intuitive steps to figure out why:
FAIL:
. Put the failing test name in your clipboard, or remember it for later.Describe the solution you'd like
I would like to see all test failures in an easy-to-find location. It looks like GitHub recently added support for job summaries, and this seems to be one of the intended use cases.
One way this may be achieved is by writing pass/fail summaries to a file, e.g.:
Then converting those summaries to markdown in a later step. E.g.:
Describe alternatives you've considered
Alternatively, it looks like there are off-the-shelf GitHub actions we can import into the project, that do a lot of fancy test reporting. I think those would require that we configure our test runners to output machine-readable reports. I imagine that's possible for the Python and C++ test libraries we use, but it would take a little more investigation.
Additional context
The proposed solution and the alternative aren't perfect. The report location is still on a separate "Summary" page, which isn't the one that appears when you click the big red X on a failing suite. But it is at least all located on a single page and only two clicks away from the PR after the proposal.
The proposal renders something like (with some fake injected failures):
The text was updated successfully, but these errors were encountered: