Test Reporter fails on workflow_run

[snovak7]

I have two step

...
      - uses: actions/upload-artifact@v4
        if: success() || failure()
        with:
          name: test-results
          path: "**/test-results*.trx"
...

and

name: report
on:
  workflow_run:
    workflows: 
      - build
    types:
      - completed
permissions: 
  contents: read
  actions: read
  checks: write
jobs:
  report:
    runs-on: ubuntu-latest
    steps:
      - uses: dorny/test-reporter@v1
        with:
          artifact: test-results
          name: dotnet tests
          path: '**/test-results*.trx'
          reporter: dotnet-trx

trx file is successfully uploaded, inspecting the URL of test reporter action, it offers me to download the file, but the action fails

Report fails with error
HTTPError: Response code 400 (Authentication information is not given in the correct format. Check the value of Authorization header.)

Additional info: it worked, but after I bumped the version of the upload-artifacts (among other), somehow stopped working...

[abelbraaksma]

I noticed something similar. Here's one of my public repos where it goes wrong (with a similar config as you have): fsprojects/FSharp.Control.TaskSeq#209. I.e., it looks as if it is hanging, but it simply doesn't run anymore (there have not been any changes in the workflow config):

I suspect actions/upload-artifact@v3 being updated to actions/upload-artifact@v4 to be at the root of the issue. I'll try it with reverting that update and will report back.

EDIT, we are not alone in this, see this long thread: actions/upload-artifact#472

[abelbraaksma]

Yep, this confirms it. The new upload-artifacts@v4 either has bugs, or other changes, that prevents it from working with test-reporter@v1. I tested it here (fsprojects/FSharp.Control.TaskSeq#212) with v3, which "just works", after all PRs with v4 did not work:

[jozefizso]

Yes, this caused by different behavior in artifacts v4: https://github.blog/changelog/2023-12-14-github-actions-artifacts-v4-is-now-generally-available/

[abelbraaksma]

@jozefizso are you aware of any way to fix this? Would it require bumping the major version here as well, or could it be made compatible with either?

[SapiensAnatis]

FYI I was able to make this work using a similar workflow_run setup by manually invoking the download-artifact action at v4:

on:
  workflow_run:
    workflows: ["test"]
    types:
      - completed

jobs:
  report:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Download artifacts
        uses: actions/download-artifact@v4
        with:
          github-token: ${{ secrets.PERSONAL_ACCESS_TOKEN }}
          pattern: test-result-*
          merge-multiple: true
          run-id: ${{ github.event.workflow_run.id }}
      - name: Generate report
        uses: dorny/test-reporter@v1
        with:
          name: test report
          path: "*.trx"
          reporter: dotnet-trx

You have to generate your own secrets.PERSONAL_ACCESS_TOKEN for the download-artifact action to be able to download artifacts from outside its own workflow. (see: https://github.com/actions/download-artifact#download-artifacts-from-other-workflow-runs-or-repositories) The workflow to download from is specified using ${{ github.event.workflow_run.id }}

[fernandopasik]

The github token @SapiensAnatis mentions needs to have the permission actions:read on target repo.

@jozefizso @abelbraaksma Would this mean that the fix for this problem in the action will need a token as well?

[abelbraaksma]

@fernandopasik I'm not sure, but it might very well be? If you want me to test a fix (I assume you'd do that in a new version as it'll introduce an incompatibility, I guess), let me know and I can run a test through fsprojects/FSharp.Control.TaskSeq#245).

[brianjmurrell]

I tried downgrading my (test) artifact upload from v4 to v3:

-        uses: actions/upload-artifact@v4
+        # Don't bump this to v4 due to https://github.com/dorny/test-reporter/issues/343
+        uses: actions/upload-artifact@v3

but now this action is complaining that it cannot find any matching artifacts:

Run dorny/test-reporter@v1.9.0
Action was triggered by pull_request: using SHA from head of source branch
Check runs will be created with SHA=9972855826a6700db87ab9fa52432366297e5934
Fetching list of tracked files from GitHub
Found 4106 files tracked by GitHub
Using test report parser 'java-junit'
Warning: No artifact matches /(Functional on .*) test-results/
Error: No test report files were found

Yet when I uploaded with v4 it was able to find an artifact but not download it as reported here:

Run dorny/test-reporter@v1.9.0
Action was triggered by pull_request: using SHA from head of source branch
Check runs will be created with SHA=e46e72b700b960c19aa8495a43cc9471fff9fae9
Fetching list of tracked files from GitHub
Found 4106 files tracked by GitHub
Using test report parser 'java-junit'
Downloading artifact Functional on EL 9 test-results.zip
Error: Response code 400 (Authentication information is not given in the correct format. Check the value of Authorization header.)

And the artifact RE that it's complaining there were no matches on would certainly match uploaded artifacts:

Functional on EL 8.8 test-results
Functional on EL 9 test-results

So is simply downgrading the actions/upload-artifact action to v3 not enough to work around this issue?

[brianjmurrell]

@jozefizso said previously:

Yes, this caused by different behavior in artifacts v4: github.blog/changelog/2023-12-14-github-actions-artifacts-v4-is-now-generally-available

So given that it's 4 months on since the above comment, what's the plan to fix this?

How is this action in any way usable given this issue? I.e. isn't is completely broken at this point now? Or does this issue only affect certain use cases of this action? If so, which use cases are still working?

[abelbraaksma]

I.e. isn't is completely broken at this point now?

It is actually artifacts@v4 that is broken. They chose to, deliberately, make it backwards incompatible. And the guide to fix it isn't all that useful.

So is simply downgrading the actions/upload-artifact action to v3 not enough to work around this issue?

Simply downgrading should be enough. See the breaking changes here: https://github.com/actions/upload-artifact?tab=readme-ov-file#breaking-changes

Even if you applied fixes, it looks like any changes you did to accommodate v4, would work with v3. I think the main issue is same-named artifacts, which don't work anymore.

[brianjmurrell]

Simply downgrading should be enough.

It is not. As I previously said, it produces this failure:

Warning: No artifact matches /(Functional on .*) test-results/

when the artifact list clearly has artifacts that match the RE:

Functional on EL 8.8 test-results
Functional on EL 9 test-results

Even if you applied fixes, it looks like any changes you did to accommodate v4, would work with v3. I think the main issue is same-named artifacts, which don't work anymore.

They don't have the same name though. One has 8.8 in it where the other has 9.

[djbrown]

See also #363

[JojOatXGME]

I was running into this issue today as well, although I don't have a repository using this action myself. The following configuration seems to work:

Show working example for test-report workflow (according to my tests)

name: Test Report
run-name: >
  Test Report for ${{ github.event.workflow.name }}
  #${{ github.event.workflow_run.run_number }}:
  ${{ github.event.workflow_run.display_title }}
on:
  workflow_run:
    workflows: [Build]
    types: [completed]

# Permissions only have to be specified when set to be restricted by default.
# Otherwise, the workflow will have all necessary permissions by default.
# See https://docs.github.com/en/actions/security-guides/automatic-token-authentication#permissions-for-the-github_token
permissions:
  checks: write
  # The documentation mentions that the following permissions is necessary to access other workflow runs.
  # However, the workflow seems to work even without this permission.
  # Maybe there is a special case which always allows access to the workflow run which triggered this run.
  actions: read

jobs:
  report:
    name: Test Report
    runs-on: ubuntu-latest

    steps:

      # We have to clone the repo as the test-reporter otherwise runs into an error.
      # See https://github.com/dorny/test-reporter/issues/131#issuecomment-2093880211
      - name: Fetch Sources
        uses: actions/checkout@v4

      # We have to manually download the artifact because dorny/test-reporter@v1
      # does not support artifacts uploaded by actions/upload-artifact@v4.
      # See https://github.com/dorny/test-reporter/issues/343
      - name: Download Test Reports
        uses: actions/download-artifact@v4
        with:
          pattern: test-results-*
          run-id: ${{ github.event.workflow_run.id }}
          github-token: ${{ github.token }}

      - name: Publish Test Reports as Annotations
        uses: dorny/test-reporter@v1
        with:
          name: "Unit Tests Report"
          path: "test-results-*/test-results/*/TEST-*.xml"
          reporter: java-junit

You have to generate your own secrets.PERSONAL_ACCESS_TOKEN for the download-artifact action to be able to download artifacts from outside its own workflow.
— #343 (comment)

The github token @SapiensAnatis mentions needs to have the permission actions:read on target repo.
— #343 (comment)

You can use the token provided by GitHub Actions (github.token or secrets.GITHUB_TOKEN). See my example above. The token has all the necessary permissions, actions/download-artifact@v4 just doesn't use the token by default. Interestingly, even if you restrict permissions by default, you don't have to specify the permission actions: read. At least according to my tests. Maybe GitHub has implemented a special case for workflows triggered by workflow_run, so that they can always access the previous run that triggered this run.

[JojOatXGME]

FYI, I created a pull request, which I think should fix this issue.

• Fix problematic retransmission of authentication token #436

[JojOatXGME]

FIY again, I also created a feature request for support of raw_details in annotations created from workflows. I hope that would eventually make the need for a separate workflow obsolete.

• Support `raw_details` parameter for annotations generated by GitHub Actions community/community#122234