Dynamically generating GitHub Actions matrix to avoid timeouts when emulating architectures

[mwilliamson]

Hello! I maintain some Python bindings for jq, and use cibuildwheel to build its binary wheels. When trying to make a release recently, I ran into trouble since building the aarch64 wheels took more than six hours due to the slowness of the emulation.

To get around this, it seemed like the best approach would be to have multiple GitHub Actions jobs running in parallel, rather than one for each architecture on each OS. However, I didn't want to hard-code Python versions and similar into the workflow: I want cibuildwheel to take care of things like that for me.

Fortunately, GitHub Actions supports dynamically generating matrices, so I wrote a script that integrates with cibuildwheel to get the list of builds, and it then generates a matrix with one job per wheel.

• GitHub Action workflow: https://github.com/mwilliamson/jq.py/blob/5e0756dada43e11d83469d5123ca25cd1fbe397e/.github/workflows/build.yml
• Script to generate the matrix: https://github.com/mwilliamson/jq.py/blob/5e0756dada43e11d83469d5123ca25cd1fbe397e/build-wheels-matrix.py

Is this something that anyone else has done? Are there better ways of doing the same thing? Is this something that others would find useful as an example or somesuch? Or is this just a terrible idea?

[henryiii]

🤯 I didn’t know you could do that… this would be a handy composite action! I’d like some way to do a little grouping too, I think (like by Python version), but not critical.

This seems really potentially useful. I wonder if we can simplify this a bit in a generic way? Maybe a way to format print-build-identifiers?

[henryiii]

Continuing the idea a bit, I think this is a little simpler (and avoids sed in favor of jq again):

- id: set-matrix
  run: |
    MATRIX_INCLUDE=$(
      {
        cibuildwheel --print-build-identifiers --platform linux \
         | jq -Rc '{"only": inputs, "os": "ubuntu-latest"}'
        && cibuildwheel --print-build-identifiers --platform macos \
         | jq -Rc '{"only": inputs, "os": "macos-latest"}'
        && cibuildwheel --print-build-identifiers --platform windows \
         | jq -Rc '{"only": inputs, "os": "windows-latest"}'
      } | jq -sc
    )
    echo ::set-output name=include::"$MATRIX_INCLUDE"

build-wheels:
  name: Build ${{ matrix.only }}
  strategy:
    matrix:
      include: ${{ fromJson(needs.build_wheels_matrix.outputs.include) }}
  runs-on: ${{ matrix.os }}
  steps:
      # checkout and stuff here
    - uses: pypa/cibuildwheel@eb39da0b10a9f48015e84973545691981949b13d
      with:
        only: ${{ matrix.only }}

Edit: using include instead of matrix to give user more options on the matrix, like only using this for part of it or only injecting emulated runs this way.

Setting the arch's is a bit tricky for Windows, since cibuildwheel --print-build-identifiers --platform windows doesn't print anything unless you force the arch; auto64 isn't picked up correctly since it's resolving to the host instead of the target. Edit: fixed in #1266.

[henryiii]

Here's a blend of "grouped" inputs and generated ones for emulated archs:

- id: set-matrix
  run: |
    MATRIX_INCLUDE=$(
      {
        cibuildwheel --print-build-identifiers --platform linux \
         | jq -Rc '{"only": inputs, "os": "ubuntu-latest"}'
      } | jq -sc
    )
    echo ::set-output name=include::"$MATRIX_INCLUDE"
  with:
    CIBW_ARCHS: aarch64

build-wheels:
  name: Build ${{ matrix.os }} ${{ matrix.only }}
  strategy:
    matrix:
      os: [ubuntu-latest, macos-latest, windows-latest]
      only: [""]
      include: ${{ fromJson(needs.build_wheels_matrix.outputs.include) }}
  runs-on: ${{ matrix.os }}
  steps:
      # checkout and stuff here
    - uses: pypa/cibuildwheel@eb39da0b10a9f48015e84973545691981949b13d
      with:
        only: ${{ matrix.only }}

[ddelange]

@henryiii fyi in the jq example above (and other docs?), inputs should be changed to . ref MagicStack/asyncpg#993

[henryiii]

I think this is the one I’ve kept up with fixes: https://iscinumpy.dev/post/cibuildwheel-2-10-0/

[joerick]

Btw I'd still be open to a --printf-builds option with expandable {identifier}, {platform}, and {arch} tokens

[henryiii]

By the way, if the only reason to do this is emulated timeouts, you could look into the various platforms that provide native runners (Azure maybe, Cirrus CI, Travis, maybe others). But I also like it due to faster builds, and you can still get timeouts due to combining manylinx + musllinux + pypy, etc.

[mwilliamson]

Even though they're slow, emulated builds are still preferable for me just because I don't want to have invest time in learning another CI platform.