Jobs need a way to reference all outputs of matrix jobs

[ghost]

This need has already been discussed in multiple threads on github.community, so I’m quoting a great summary that was posted in one of those threads by consideratio:

Issue

Currently, a job with strategy.matrix runs multiple times with variations based on the matrix. The issue is that only a single value is available for future jobs that needs this job’s output, because even if the output is set by multiple matrix variations of the job, only one is retained.

Workaround

There is a workaround of defining separate outputs for a job with strategy.matrix by letting the job variations set different outputs.

A new observation

Limitation of the workaround

While it is possible to define a fixed list of outputs for a job with strategy.matrix running with a fixed list of jobs, it is not possible to handle a situation when you have declared the strategy.matrix dynamically (using output of a previous job), and would need a dynamic amount of outputs, for example one output per job variation.

This limitation can be understood more directly by the documented context availability which doesn’t lists the needs context available when setting job.<job_id>.outputs but only when setting a specific output such as job.<job_id>.outputs.<output_id> .

Practically I’ve also verified this limitation.

  test-set-dynamic-outputs:
    runs-on: ubuntu-20.04
    needs: test-set-dynamic-outputs-init
    outputs: ${{ toJson(needs.test-set-dynamic-outputs-init.dynamic_outputs) }}

The workflow is not valid. .github/workflows/dynamic_outputs.yaml (Line: 66, Col: 14): Unrecognized named-value: 'needs'. Located at position 8 within expression: toJson(needs.test-set-dynamic-outputs-init.dynamic_outputs) .github/workflows/dynamic_outputs.yaml (Line: 66, Col: 14): Unexpected value '${{ toJson(needs.test-set-dynamic-outputs-init.dynamic_outputs) }}'

For more details, see this workflow definition .

Two feature requests

I think both these feature requests could be relevant, but the first feature is the best feature from a user perspective and possibly even also from a GitHub developers perspective as it could actually be simpler to implement and maintain long term.

Add a new context: jobs.<job_id>.matrix-outputs

Could GitHub add new context? It would be used for those that want to see the distinct output values for a given output id, as set by different matrix job variations.

I propose a new context ( jobs.<job_id>.matrix-outputs ) instead of a change of the existing context ( jobs.<job_id>.outputs ) for a two main reasons:

1. It wouldn’t be a breaking change to add something new.
2. It would be relevant to distinguish and use both jobs.<job_id>.outputs and jobs.<job_id>.matrix-outputs separately.

Allow dynamic jobs.<job_id>.outputs definitions

Would it be possible to allow context for jobs.<job_id>.outputs to be set, and not only for the individual output ids?

In other words, to allow this to function:

  test-set-dynamic-outputs:
    runs-on: ubuntu-20.04
    needs: init
    outputs: ${{ toJson(needs.init.dynamic_outputs) }}

And to not be restricted where only this is functioning:

In other words, to allow this to function:

  test-set-dynamic-output-ids:
    runs-on: ubuntu-20.04
    needs: init
    outputs:
      a: ${{ toJson(needs.init.dynamic_output_a) }}
      b: ${{ toJson(needs.init.dynamic_output_b) }}

GitHub team, please take a look at the above feature requests and please consider implementing them! The current situation, wherein each successive job instance overwrites the outputs of the prior instance, is really not great. Thank you.

[ringerc]

I explained this another way at https://github.community/t/roll-up-matrix-outputs/253564

This is a huge limitation of the current interface IMO.

I like the idea of a matrix-outputs. But I'd like a simple way to flatten that out to a list of individual runs of a matrix job, too, for use in a downstream matrix or similar. I'd like to be able to access:

• An array of matrix job instance ids that can be passed as a matrix input for a downstream job
• For each matrix job instance, a json object containing all its matrix inputs - the matrix variable as that job saw its value
• For each matrix job instance, a json object containing all that matrix job's outputs

[MemphisMeng]

DId you remove your discussion? Your link went to nothing.

[ghost]

@MemphisMeng see https://github.com/orgs/community/discussions/17245#discussioncomment-3421929

[airtower-luna]

The forum was moved from github.community to here in summer, and most threads without a selected answer weren't migrated. That's probably why the link is broken. 😕

[0x53A]

An alternative might be stop hacking around with dynamic matrixes and be able to run dynamic workflows.

In gitlab there's an equivalent feature where you can generate and run a child pipeline. The advantage of generating a whole pipeline/workflow is that at this point you can just dynamically hardcode everything exactly like you need.

[radiomix]

💀 Sad Github changes discussion links under the hood destroying contex 💣 !
The above URL posted as https://github.community/t/bug-jobs-output-should-return-a-list-for-a-matrix-job/128626/32 does not link to the discussion any more! (Holds true for all the links to github.github.community in this thread). Not the first time I see github discussion links broken, especially on feature-requests a lot of people voted for.

It seams an intended Github feature to prevent people from accumulating comparable discussions/threads.

[ghost]

Agreed.

The community engagement platform churn is nearly akin to Google’s messaging apps churn.

[dmurchison]

Verry informative and helpful post... Thank you!

[cheukribbon]

Is there any update on this? Is github working on this?

[mikz]

I've worked around this by using artifacts to generate the matrix.

jobs:
  matrix:
    runs-on: ubuntu-22.04
    strategy:
      fail-fast: false
      matrix:
        include:
          - target: mafiapigs-macos
            metaId: 2177361
            otherId: 2177360
          - target: project-windows
            metaId: 2177362
            otherId: 2177360
    steps:
      - run: |
          matrix=$((
            echo '${{ toJSON(matrix) }}'
          ) | jq -c .)
          echo "$matrix" > matrix
          echo "::set-output name=matrix::$matrix"
        id: matrix
        if: github.event_name == 'workflow_dispatch' && (github.event.inputs.build_target == matrix.target || github.event.inputs.build_target == '*' || github.event.inputs.build_target == '')
      - uses: actions/upload-artifact@v3
        with:
          name: ${{ hashFiles('matrix') || 'none' }}
          path: matrix
          if-no-files-found: warn

  check-matrix:
    runs-on:  ubuntu-22.04
    needs: [ matrix ]
    outputs:
      matrix: ${{ steps.matrix.outputs.matrix }}
    steps:
      - uses: actions/download-artifact@v3
      - run: |
          matrix="$(cat */matrix | jq -c --slurp .)"
          echo "::set-output name=matrix::$matrix"
        id: matrix

  steam:
    name: "Release ${{ matrix.includes.target }}"
    needs: [check-matrix]
    runs-on: ubuntu-22.04
    strategy:
      fail-fast: false
      matrix:
        includes: ${{ fromJSON(needs.check-matrix.outputs.matrix) }}
    steps:

[theonlysinjin]

Thanks @mikz, this seems to be the workaround.
Thankfully we went from a bunch of files with just new line separators, so we were also able the parse all the files at once with

matrix="$(cat */matrix | jq -ncR '[inputs]')"

I wondered if a possible solution might be to be able to call another workflow from within a step,
this means you could also get around the problem by chaining multiple workflows together.
This might just be another workaround and not work for someone else.

[piotrekkr]

Using artifacts seems like a possible workaround but there can be some edge cases where artifact get corrupted when multiple jobs write to same artifact at same time. There is a warning in upload action docs:

Warning: Be careful when uploading to the same artifact via multiple jobs as artifacts may become corrupted. When uploading a file with an identical name and path in multiple jobs, uploads may fail with 503 errors due to conflicting uploads happening at the same time. Ensure uploads to identical locations to not interfere with each other.

Probably very rare to happen but it may so watch for this

[xriu]

I'm implementing this workaround, if you want to prevent artifact corruption during concurrent executions of the same workflow. You can use the following:

concurrency:
  group: ${{ github.event.pull_request.number }}
  cancel-in-progress: true

Basically this will cancel previous executions and continue with the new one

[piotrekkr]

@xriu There are two problems here:

1. concurrent execution of same workflow that updates artifact, (e.g. someone push twice to same PR almost at same time)
2. concurrent execution of matrix jobs that update same artifact within same workflow run

For first case your workaround will probably work. For second one it may get artifacts corrupted because matrix jobs will concurrently write results to same artifact. In second case you could set concurrency on job level but this would defeat whole purpose of matrix which is concurrent execution.

Ugly but probably working solution would be to use different artifacts for each matrix job. For example:

1. you have matrix like this:

   jobs:
     example_matrix:
       strategy:
         matrix:
           os: [ubuntu-22.04, ubuntu-20.04]
           version: [10, 12, 14]
   

2. each matrix job uploads artifact with special name like matrix-outputs-${{ github.run_id }}-${{ github.run_attempt }}-HASH_OF_MATRIX_OBJECT containing json file with outputs like:

   {
     "${{ matrix.os }}": {
       "${{ matrix.version }}": {
         "outputs": {....}
       }
     }
   }
   

3. job that need matrix output will:
   1. ask for artifacts using github API
   2. download
   3. unpack
   4. deep merge json data from unpacked artifacts into one object e.g. jq -s 'reduce .[] as $item ({}; . * $item)' *.json > outputs-combined.json

As you can see pretty ugly workaround but should be resistant to artifact corruption.

[sigprof]

Note that actions/upload-artifact@v4.0.0 now disallows multiple uploads to the same artifact, so if you are using this workaround, you must ensure that each instance of the matrix job uses an unique artifact name. If you try to use the same artifact names for all instances, you will get an error like this:

Failed to CreateArtifact: Received non-retryable error: Failed request: (409) Conflict: an artifact with this name already exists on the workflow run

In my testing ${{strategy.job-index}} seems to be enough to provide distinction between multiple instances of the same matrix job.

Changing the job to use multiple artifacts may also cause another issue — actions/download-artifact still does not support a glob or any other filter for the artifact name (it can download either a single artifact specified by the exact name, or all artifacts for the workflow run), so it may download too much if the workflow also uses artifacts for other purposes (like passing the intermediate build results between jobs).

There is also a new limit on the number of artifacts:

Each job in a workflow run now has a limit of 10 artifacts.

However, looks like each instance of a matrix job is treated as a distinct job when this limit is applied (at least I tried a test with 25 jobs in the matrix, and it passed).

[cj81499]

"Support matrix context in output keys" merged 2 days ago!

actions/runner#2477

[srosell-ut]

I've tested this (it was released in the v2.303.0 release and at least what in that issue is described we couldn't make it work (I've added a comment in the PR to see if I can get some response). Anyone has make it work? 🤔

[martincostello]

It notes in the PR that:

Please note that: Server side changes are still in-progess

I guess there's changes in GitHub itself required to fully support the scenario.

[srosell-ut]

yeap, make sense I guess, I don't really understand what they refer with that side note but I guess is it not finished then 🤷
Thanks for the comment 🙇

[csaba-almasi]

An FYI for anyone struggling with this and hacking their way around the limitations: the PR for the runner landed to support the new context, but the team working on the necessary server-side implementation has been completely laid off, leaving this feature in purgatory. See this comment from the contributor on the Runner issue.

[Loki-Afro]

although it has been already asked in the pr comments but .. are there any updates?

[dburman23]

2024 here and this is still an issue facepalm
Only workaround for now is to upload what is to be your outputs, as artifacts from one job and download them into the other job, per parallel execution of the matrix.