GitHub Action
Publish Test Report
This action allows us to publish reports as github-actions check and produce comments on Pull requests with a summary of the report.
By identifying comments through comment-header
(which will be transformed to an invisible html
comment, containing the comment-header
to identify comments controlled by this action), we can update an existing comment, adding more information as new reports get available over the run of a workflow.
This action is a composite action, it uses the following actions:
Action |
---|
name: My Workflow
...
permissions:
actions: write # Necessary to cancel workflow executions
checks: write # Necessary to write reports
pull-requests: write # Necessary to comment on PRs
...
This snippet will post a comment with the given comment-header
to the PR that triggered the workflow execution.
Any existing comments with the same comment-header
will be hidden as OUTDATED
:
jobs:
...
recreate-comment:
runs-on: ubuntu-latest
steps:
- name: Publish Report
uses: turing85/publish-report@v2
with:
checkout: 'true'
comment-header: my-comment-header
comment-message-recreate: Hello
recreate-comment: true
...
The report will be generated from the files selected by the glob pattern in pattern-path
.
The name of the report will be report-name
+ " Report"
.
If all tests succeeded, the message comment-message-success
will be appended to the comment with the given comment-header
.
If tests failed, the message in comment-message-failure
will be appended to the comment with the given comment-header
.
We set the execution of the Publish Report
step to if: ${{ always() }}
so that it is also executed when tests fail (and we get a report).
jobs:
...
test:
runs-on: ubuntu-latest
steps:
- name: Git checkout
uses: actions/checkout@v4
...
- name: Run Tests
...
continue-on-error: true # the "publish" step will fail, so we get a report when tests failed as well
...
...
- name: Publish Report
uses: turing85/publish-report@v2
if: ${{ always() }}
with:
# cancel-workflow-on-error: 'false' # If we do not want to cancel the whole workflow execution on error
# checkout: 'true' # not needed; project is already checked out
comment-header: my-comment-header
comment-message-success: |
YAY! {0} passed!
{1} tests were successful, {2} tests failed, {3} test were skipped.
The report can be found [here]({4}).
comment-message-failure: |
On no! {0} failed!
{1} tests were successful, {2} tests failed, {3} test were skipped.
The report can be found [here]({4}).
report-fail-on-error: true # to fail when tests failed
report-name: Tests
report-path: '**/target/surefire-reports/TEST*.xml'
report-reporter: java-junit
...
When we have a scenario where we cannot or do not want the report generation the same job as the test, we can upload the test artifacts via actions/upload-artifact
.
We can then execute report generation in a separate step, downloading said test artifacts.
Notice that the name
and path
used in action actions/upload-artifact
correlates with download-artifact-name
and report-path
in action turing/publish-report
.
We should configure the test
job so that it does not fail when tests fail.
This guarantees that the test-report
job is executed, and can fail (as long as report-fail-on-error
is not set to 'false'
).
...
jobs:
...
test:
runs-on: ubuntu-latest
steps:
- name: Git checkout
uses: actions/checkout@v4
...
- name: Run Tests
...
...
- name: Upload test artifacts
uses: actions/upload-artifact@v4
if: ${{ always() }}
with:
name: test-report
path: '**/target/*-reports/TEST*.xml'
if-no-files-found: error
retention-days: 2
test-report:
runs-on: ubuntu-latest
needs:
...
- test
...
steps:
- name: Publish Report
uses: turing85/publish-report@v2
with:
...
checkout: true
...
download-artifact-name: test-report
report-path: '**/target/surefire-reports/TEST*.xml'
...
...
When forks provide PRs, the corresponding workflow runs are limited in what they can do. This is done for security reasons. For details, please read "Keeping your GitHub Actions and workflows secure Part 1: Preventing pwn requests" (securitylab.github.com
). Two consequences are that we cannot
- comment on PRs, and
- upload test reports.
This is where the pull_request_target
and workflow_run
events come into play.
The action is designed to be used with the pull_request_target
and workflow_run
events. Often times, the same workflow file is used to build the default branch, as well as PRs. For a good user experience, the extension tries to "figure out" when the jobs that comment on PRs should be run.
Reports are generated and uploaded whenever recreate-comment
is not 'true'
. The action tries to update the pull request comment if and only if:
inputs.comment-enabled
is'true'
, andinputs.recreate-comment
is not'true'
, and- either
github.event_name
is'pull_request'
or'pull_request_target'
, - or
github.event.workflow_run.event
is'pull_request'
- either
The recommendation is to create three workflows:
- one workflow for the initial comment,
- the main workflow for the actual build, and
- one workflow to update the comment wit the test results.
The initial workflows may look something like this:
name: Comment on PR
on:
pull_request_target:
...
permissions:
pull-requests: write
jobs:
comment:
runs-on: ubuntu-latest
steps:
- name: (Re)create comment
uses: turing85/publish-report@v2
with:
github-token: ${{ github.token }}
comment-message-recreate: |
## 🚦Reports 🚦
Reports will be posted here as they get available.
comment-message-pr-number: ${{ github.event.number }}
recreate-comment: true
You might notice that we override the comment-message-recreate
. We will discuss this later.
The workflow to update the comment could look like this:
name: Build report
on:
workflow_run:
workflows:
- "build"
types:
- completed
permissions:
actions: write
checks: write
pull-requests: write
jobs:
report:
if: ${{ github.event.workflow_run.event == 'pull_request' }}
runs-on: ubuntu-latest
steps:
- name: Download PR number
uses: actions/download-artifact@v4
with:
github-token: ${{ github.token }}
name: pr-number
run-id: ${{ github.event.workflow_run.id }}
- name: Set PR number
id: get-pr-number
run: |
echo "pr-number=$(cat pr-number.txt)" | tee "${GITHUB_OUTPUT}"
rm -rf pr-number.txt
- name: Publish reports
uses: turing85/publish-report@feature/run-id-and-pr-number
with:
comment-message-pr-number: ${{ steps.get-pr-number.outputs.pr-number }}
download-artifact-name: test-reports
download-artifact-run-id: ${{ github.event.workflow_run.id }}
report-name: My Tests
report-path: '**/target/**/TEST*.xml'
We use an artifact named pr-number
here. Since we use a workflow_run
, we do not know anything of the pull request. Thus, we need some support from the actual build pipeline. It must create an artifact pr-number
that contains a file pr-number.txt
. the content of this file should be the number of the pull request.
The necessary steps to generate this artifact in the actual build workflow may look like this:
name: Build
on:
pull_request:
branches:
- main
...
push:
branches:
- main
...
permissions:
actions: write
checks: write
pull-requests: write
jobs:
build:
runs-on: ubuntu-latest
steps:
...
(build the application, generate the test artifacts)
...
- name: Get PR number
id: get-pr-number
if: ${{ always() }}
run: |
echo "${{ github.event.number }}" > "pr-number.txt"
- name: Upload PR number
uses: actions/upload-artifact@v4
if: ${{ always() }}
with:
name: ${{ env.PR_NUMBER_ARTIFACT_NAME }}
path: pr-number.txt
Now on the point why we override the comment-message-recreate
. The default value of this variable is:
## 🚦Reports for run [#${{ github.run_number }}](${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }})🚦
Reports will be posted here as they get available.
This text contains a link to the workflow run associated with this comment. While very convenient, we are not able to update the comment from the main workflow. The workflow for the initial comment runs in parallel to the main workflow, so there is no way to "figure out" the run id or run number.
The workflow that updates the comment could of course add the link to the comment. However, at this point in time, the run is already over
For a complex example please take a look at the workflow of github.com/turing85/advent-of-code-2022
Name | semantics | Required? | default |
---|---|---|---|
General Inputs | |||
|
The github-token to use. | ✅ |
|
|
Whether the entire current workflow should be cancelled on error (i.e. when tests failed). | ✅ |
|
|
Whether a checkout should be performed | ✅ |
|
Comment-related Inputs | |||
|
Overrides the comment on a PR. | ✅ |
|
|
Triggers the (re-)creation of the comment in a PR, that is updated with the reports. | ✅ |
|
|
Whether a comment on the PR should be posted. | ✅ |
|
|
The header to identify the PR comment. This is an invisible tag on the comment. | ✅ |
|
|
Message appended to the comment posted on the PR after the tests failed.
The message can be templated for replacement. The format feature of github-expressions is used to replace placeholders. The following placeholder-mapping applies:
|
✅ |
<details>
<summary><h3>😔 {0} failed</h3></summary>
| Passed | Failed | Skipped |
|--------|--------|---------|
| ✅ {1} | ❌ {2} | ⚠️ {3} |
You can see the report [here]({4}).
</details> |
|
The new comment message. Notice that the old comment will be completely overridden. | ✅ |
## 🚦Reports for run [#${{ github.run_number }}](${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }})🚦
Reports will be posted here as they get available. |
|
Initial text for the comment posted on the PR. Subsequent messages will be appended. | ✅ |
## 🚦Reports for run [#${{ github.run_number }}](${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }})🚦
Reports will be posted here as they get available. |
|
Message appended to the comment posted on the PR after the tests succeed.
The message can be templated for replacement. The format feature of github-expressions is used to replace placeholders. The following placeholder-mapping applies:
|
✅ |
<details>
<summary><h3>🥳 {0} passed</h3></summary>
| Passed | Failed | Skipped |
|--------|--------|---------|
| ✅ {1} | ❌ {2} | ⚠️ {3} |
You can see the report [here]({4}).
</details> |
|
The PR number to which the comment should be written. | ✅ |
|
Artifact-related Inputs | |||
|
The name of the artifact to download. | ✅ |
|
|
The pattern of the artifact to download. | ✅ |
|
|
If artifacts should be merged if multiple artifacts are downloaded. | ✅ |
|
|
The run-id for which the artifact should be downloaded. | ✅ |
|
Report-related Inputs | |||
|
Whether an error in a test should fail the step. | ✅ |
|
|
Limits which test suites are listed. Supported options: - all - failed | ✅ |
|
|
Limits which test cases are listed. Supported options: - all - failed - none | ✅ |
|
|
The name of the report. The Text "Report" will be appended to form the report name that is attached to the check. So if we pass "JUnit" as report-name, the corresponding report will be called "JUnit Report". | ✅ |
|
|
Allows you to generate only the summary.
If enabled, the report will contain a table listing each test results file and the number of passed, failed, and skipped tests. Detailed listing of test suites and test cases will be skipped. |
✅ |
|
|
A glob path to the report files. | ✅ |
|
|
Format of test results. Supported options: - dart-json - dotnet-trx - flutter-json - java-junit - jest-junit - mocha-json - mochawesome-json | ✅ |
|
Name | Semantics | type |
---|---|---|
|
URL to the report in workflow checks. |
|
|
Number of tests failed. |
|
|
Number of tests passed. |
|
|
Number of tests skipped. |
|
Screenshots are taken from this comment and this workflow run.
Initial comment on a PR |
First report added |
Second report added |
JUnit Report |
OWASP report |
Thanks goes to these wonderful people (emoji key):
Marco Bungart 💻 🚧 |
Greg Duckworth 💻 |
Lorenzo Bettini 💬 🤔 |
This project follows the all-contributors specification. Contributions of any kind welcome!
This project is licensed under the Apache License 2.0. The license file can be found here.