Skip to content
upload

GitHub Action

Publish Test Report

v2.1.1 Latest version

Publish Test Report

upload

Publish Test Report

Action to publish a Test Report. The report is published as check and can be posted as comment to a PR.

Installation

Copy and paste the following snippet into your .yml file.

              

- name: Publish Test Report

uses: turing85/publish-report@v2.1.1

Learn more about this action in turing85/publish-report

Choose a version

GitHub license file

Github action to publish reports

This action allows us to publish reports as github-actions check and produce comments on Pull requests with a summary of the report.

By identifying comments through comment-header (which will be transformed to an invisible html comment, containing the comment-header to identify comments controlled by this action), we can update an existing comment, adding more information as new reports get available over the run of a workflow.

Components

This action is a composite action, it uses the following actions:

Action

actions/checkout@v4

marocchino/sticky-pull-request-comment@v2

actions/download-artifact@v4

phoenix-actions/test-reporting@v15

andymckay/cancel-action@0.5

Quick reference

Permission setup (for personal repositories)

name: My Workflow
...
permissions:
  actions: write       # Necessary to cancel workflow executions
  checks: write        # Necessary to write reports
  pull-requests: write # Necessary to comment on PRs
...

Post initial comment

This snippet will post a comment with the given comment-header to the PR that triggered the workflow execution. Any existing comments with the same comment-header will be hidden as OUTDATED:

jobs:
  ...
  recreate-comment:
    runs-on: ubuntu-latest

    steps:
      - name: Publish Report
        uses: turing85/publish-report@v2
        with:
          checkout: 'true'
          comment-header: my-comment-header
          comment-message-recreate: Hello
          recreate-comment: true
  ...

Generate test report and append message to existing comment

The report will be generated from the files selected by the glob pattern in pattern-path. The name of the report will be report-name + " Report".

If all tests succeeded, the message comment-message-success will be appended to the comment with the given comment-header. If tests failed, the message in comment-message-failure will be appended to the comment with the given comment-header.

We set the execution of the Publish Report step to if: ${{ always() }} so that it is also executed when tests fail (and we get a report).

jobs:
  ...
  test:
    runs-on: ubuntu-latest

    steps:
      - name: Git checkout
        uses: actions/checkout@v4
      ...
      - name: Run Tests
        ...
        continue-on-error: true # the "publish" step will fail, so we get a report when tests failed as well
        ...
      ...
      - name: Publish Report
        uses: turing85/publish-report@v2
        if: ${{ always() }}
        with:
          # cancel-workflow-on-error: 'false' # If we do not want to cancel the whole workflow execution on error
          # checkout: 'true' # not needed; project is already checked out 
          comment-header: my-comment-header
          comment-message-success: |
            YAY! {0} passed!  
            
            {1} tests were successful, {2} tests failed, {3} test were skipped.
            
            The report can be found [here]({4}).

          comment-message-failure: |
            On no! {0} failed!  

            {1} tests were successful, {2} tests failed, {3} test were skipped.

            The report can be found [here]({4}).
          report-fail-on-error: true # to fail when tests failed
          report-name: Tests
          report-path: '**/target/surefire-reports/TEST*.xml'
          report-reporter: java-junit
  ...

Generate test report from a downloaded artifact and append message to existing comment

When we have a scenario where we cannot or do not want the report generation the same job as the test, we can upload the test artifacts via actions/upload-artifact. We can then execute report generation in a separate step, downloading said test artifacts.

Notice that the name and path used in action actions/upload-artifact correlates with download-artifact-name and report-path in action turing/publish-report.

We should configure the test job so that it does not fail when tests fail. This guarantees that the test-report job is executed, and can fail (as long as report-fail-on-error is not set to 'false').

...
jobs:
  ...
  test:
    runs-on: ubuntu-latest

    steps:
      - name: Git checkout
        uses: actions/checkout@v4
      ...
      - name: Run Tests
        ...
      ...
      - name: Upload test artifacts
        uses: actions/upload-artifact@v4
        if: ${{ always() }}
        with:
          name: test-report
          path: '**/target/*-reports/TEST*.xml'
          if-no-files-found: error
          retention-days: 2

  test-report:
    runs-on: ubuntu-latest
    
    needs:
      ...
      - test
      ...

    steps:
      - name: Publish Report
        uses: turing85/publish-report@v2
        with:
          ...
          checkout: true
          ...
          download-artifact-name: test-report
          report-path: '**/target/surefire-reports/TEST*.xml'
          ...
  ...

Support for fork PRs

When forks provide PRs, the corresponding workflow runs are limited in what they can do. This is done for security reasons. For details, please read "Keeping your GitHub Actions and workflows secure Part 1: Preventing pwn requests" (securitylab.github.com). Two consequences are that we cannot

  • comment on PRs, and
  • upload test reports.

This is where the pull_request_target and workflow_run events come into play.

The action is designed to be used with the pull_request_target and workflow_run events. Often times, the same workflow file is used to build the default branch, as well as PRs. For a good user experience, the extension tries to "figure out" when the jobs that comment on PRs should be run. Reports are generated and uploaded whenever recreate-comment is not 'true'. The action tries to update the pull request comment if and only if:

  • inputs.comment-enabled is 'true', and
  • inputs.recreate-comment is not 'true', and
    • either github.event_name is 'pull_request' or 'pull_request_target',
    • or github.event.workflow_run.event is 'pull_request'

The recommendation is to create three workflows:

  • one workflow for the initial comment,
  • the main workflow for the actual build, and
  • one workflow to update the comment wit the test results.

The initial workflows may look something like this:

name: Comment on PR

on:
  pull_request_target:
    ...

permissions:
  pull-requests: write

jobs:
  comment:
    runs-on: ubuntu-latest

    steps:
      - name: (Re)create comment
        uses: turing85/publish-report@v2
        with:
          github-token: ${{ github.token }}
          comment-message-recreate: |
            ## 🚦Reports 🚦
            Reports will be posted here as they get available.
          comment-message-pr-number: ${{ github.event.number }}
          recreate-comment: true

You might notice that we override the comment-message-recreate. We will discuss this later.

The workflow to update the comment could look like this:

name: Build report

on:
  workflow_run:
    workflows:
      - "build"
    types:
      - completed

permissions:
  actions: write
  checks: write
  pull-requests: write

jobs:
  report:
    if: ${{ github.event.workflow_run.event == 'pull_request' }}
    runs-on: ubuntu-latest

    steps:
      - name: Download PR number
        uses: actions/download-artifact@v4
        with:
          github-token: ${{ github.token }}
          name: pr-number
          run-id: ${{ github.event.workflow_run.id }}

      - name: Set PR number
        id: get-pr-number
        run: |
          echo "pr-number=$(cat pr-number.txt)" | tee "${GITHUB_OUTPUT}"
          rm -rf pr-number.txt

      - name: Publish reports
        uses: turing85/publish-report@feature/run-id-and-pr-number
        with:
          comment-message-pr-number: ${{ steps.get-pr-number.outputs.pr-number }}
          download-artifact-name: test-reports
          download-artifact-run-id: ${{ github.event.workflow_run.id }}
          report-name: My Tests
          report-path: '**/target/**/TEST*.xml'

We use an artifact named pr-number here. Since we use a workflow_run, we do not know anything of the pull request. Thus, we need some support from the actual build pipeline. It must create an artifact pr-number that contains a file pr-number.txt. the content of this file should be the number of the pull request.

The necessary steps to generate this artifact in the actual build workflow may look like this:

name: Build

on:
  pull_request:
    branches:
      - main
    ...
  push:
    branches:
      - main
    ...

permissions:
  actions: write
  checks: write
  pull-requests: write


jobs:
  build:
    runs-on: ubuntu-latest

    steps:
      ...
      (build the application, generate the test artifacts)
      ...
      - name: Get PR number
        id: get-pr-number
        if: ${{ always() }}
        run: |
          echo "${{ github.event.number }}" > "pr-number.txt"

      - name: Upload PR number
        uses: actions/upload-artifact@v4
        if: ${{ always() }}
        with:
          name: ${{ env.PR_NUMBER_ARTIFACT_NAME }}
          path: pr-number.txt

Now on the point why we override the comment-message-recreate. The default value of this variable is:

## 🚦Reports for run [#${{ github.run_number }}](${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }})🚦
Reports will be posted here as they get available.

This text contains a link to the workflow run associated with this comment. While very convenient, we are not able to update the comment from the main workflow. The workflow for the initial comment runs in parallel to the main workflow, so there is no way to "figure out" the run id or run number.

The workflow that updates the comment could of course add the link to the comment. However, at this point in time, the run is already over

Complex example

For a complex example please take a look at the workflow of github.com/turing85/advent-of-code-2022

Inputs

NamesemanticsRequired?default
General Inputs

github-token

The github-token to use.

${{ github.token }}

cancel-workflow-on-error

Whether the entire current workflow should be cancelled on error (i.e. when tests failed).

'false'

checkout

Whether a checkout should be performed

'false'

Comment-related Inputs

override-comment

Overrides the comment on a PR.

'false'

recreate-comment

Triggers the (re-)creation of the comment in a PR, that is updated with the reports.

'false'

comment-enabled

Whether a comment on the PR should be posted.

'true'

comment-header

The header to identify the PR comment. This is an invisible tag on the comment.

reports

comment-message-failure

Message appended to the comment posted on the PR after the tests failed.

The message can be templated for replacement. The format feature of github-expressions is used to replace placeholders. The following placeholder-mapping applies:

  • {0} is inputs.report-name
  • {1} is the number of successful tests
  • {2} is the number of failed tests
  • {3} is the number of skipped tests
  • {4} is the URL to the HTML-Report
<details>
  <summary><h3>😔 {0} failed</h3></summary>

  | Passed | Failed | Skipped |
  |--------|--------|---------|
  | ✅ {1} | ❌ {2} | ⚠️ {3}   |

  You can see the report [here]({4}).
</details>

comment-message-override

The new comment message. Notice that the old comment will be completely overridden.
## 🚦Reports for run [#${{ github.run_number }}](${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }})🚦
Reports will be posted here as they get available.

comment-message-recreate

Initial text for the comment posted on the PR. Subsequent messages will be appended.
## 🚦Reports for run [#${{ github.run_number }}](${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }})🚦
Reports will be posted here as they get available.

comment-message-success

Message appended to the comment posted on the PR after the tests succeed.

The message can be templated for replacement. The format feature of github-expressions is used to replace placeholders. The following placeholder-mapping applies:

  • {0} is inputs.report-name
  • {1} is the number of successful tests
  • {2} is the number of failed tests
  • {3} is the number of skipped tests
  • {4} is the URL to the HTML-Report
<details>
  <summary><h3>🥳 {0} passed</h3></summary>

  | Passed | Failed | Skipped |
  |--------|--------|---------|
  | ✅ {1} | ❌ {2} | ⚠️ {3}   |

  You can see the report [here]({4}).
</details>

comment-message-pr-number

The PR number to which the comment should be written.

${{ github.event.number }}

Artifact-related Inputs

download-artifact-name

The name of the artifact to download.

''

download-artifact-pattern

The pattern of the artifact to download.

''

download-artifact-merge-multiple

If artifacts should be merged if multiple artifacts are downloaded.

'false'

download-artifact-run-id

The run-id for which the artifact should be downloaded.

${{ github.run_id }}

Report-related Inputs

report-fail-on-error

Whether an error in a test should fail the step.

'true'

report-list-suites

Limits which test suites are listed. Supported options: - all - failed

all

report-list-tests

Limits which test cases are listed. Supported options: - all - failed - none

all

report-name

The name of the report. The Text "Report" will be appended to form the report name that is attached to the check. So if we pass "JUnit" as report-name, the corresponding report will be called "JUnit Report".

JUnit

report-only-summary

Allows you to generate only the summary.

If enabled, the report will contain a table listing each test results file and the number of passed, failed, and skipped tests.

Detailed listing of test suites and test cases will be skipped.

false

report-path

A glob path to the report files.

**/*.xml

report-reporter

Format of test results. Supported options: - dart-json - dotnet-trx - flutter-json - java-junit - jest-junit - mocha-json - mochawesome-json

java-junit

Outputs

NameSemanticstype

report-url

URL to the report in workflow checks.

string

tests-failed

Number of tests failed.

number

tests-passed

Number of tests passed.

number

tests-skipped

Number of tests skipped.

number

Showcase

Screenshots are taken from this comment and this workflow run.

initial comment

Initial comment on a PR

first report

First report added

second report

Second report added

junit report

JUnit Report

owasp report

OWASP report

Contributors ✨

Thanks goes to these wonderful people (emoji key):

Marco Bungart
Marco Bungart

💻 🚧
Greg Duckworth
Greg Duckworth

💻
Lorenzo Bettini
Lorenzo Bettini

💬 🤔

This project follows the all-contributors specification. Contributions of any kind welcome!

License

This project is licensed under the Apache License 2.0. The license file can be found here.