How to use matrix for multi-platform builds? #846
Comments
|
I was able to handle individual jobs by using the method posted here. Reference: #671 (comment) |
|
That's very interesting! I wonder if it would be possible to have buildx pushing different platforms without the need of being a single call. Currently if I push a single --platform, it seems to override the previously pushed one. |
|
@crazy-max that's really cool but... last time I tried to use upload-artifact for the job, it took 20-30 minutes only to upload and download it. It was definitely a showstopper for me. Do you believe it has improved? Maybe I should try it again. |
You were uploading all cache export which can be quite expensive to compress > upload > download > decompress. In https://docs.docker.com/build/ci/github-actions/multi-platform/#distribute-build-across-multiple-runners we are just uploading the resulting image tarball. If you want to use cache in the first job you should consider the |
|
I will do some testing. Thanks a lot! |
|
This is how I'm adding GHA caching on top of the provided example: - name: Prepare
run: |
mkdir -p /tmp/images
platform=${{ matrix.platform }}
platform=${platform//\//-}
echo "TARFILE=${platform}.tar" >> $GITHUB_ENV
echo "TAG=${{ env.TMP_LOCAL_IMAGE }}:${platform}" >> $GITHUB_ENV
echo "SCOPE=${{ env.GITHUB_REF_NAME }}-${platform}" >> $GITHUB_ENV
- name: Build
uses: docker/build-push-action@v4
with:
context: .
platforms: ${{ matrix.platform }}
tags: ${{ env.TAG }}
outputs: type=docker,dest=/tmp/images/${{ env.TARFILE }}
cache-from: type=gha,scope=${{ env.SCOPE }}
cache-to: type=gha,scope=${{ env.SCOPE }},mode=maxThe test is running now. However, I wonder how can I integrate the push phase of the example with |
|
Hi, great to see that approach. I have been using this as well for a couple of months now. And can even adopt some of the commands to mine. I did not find a solution, however, to annotate labels like |
Following https://docs.docker.com/build/ci/github-actions/multi-platform/#distribute-build-across-multiple-runners, I wonder if we could instead reuse the build push action with a temp dockerfile. Not tested but here is the idea: push:
runs-on: ubuntu-latest
needs:
- build
services:
registry:
image: registry:2
ports:
- 5000:5000
steps:
-
name: Download images
uses: actions/download-artifact@v3
with:
name: images
path: /tmp/images
-
name: Load images
run: |
for image in /tmp/images/*.tar; do
docker load -i $image
done
-
name: Push images to local registry
run: |
docker push -a ${{ env.TMP_LOCAL_IMAGE }}
-
name: Set up Docker Buildx
uses: docker/setup-buildx-action@v2
with:
driver-opts: network=host
-
name: Login to Docker Hub
uses: docker/login-action@v2
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
-
name: Temp Dockerfile
run: |
mkdir -p /tmp/dkfilectx
echo "FROM ${{ env.TMP_LOCAL_IMAGE }}" > /tmp/dkfilectx/Dockerfile
-
name: Docker meta
id: meta
uses: docker/metadata-action@v4
with:
images: ${{ env.REGISTRY_IMAGE }}
-
name: Push
uses: docker/build-push-action@v4
with:
context: /tmp/dkfilectx
push: true
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
-
name: Inspect image
run: |
docker buildx imagetools inspect ${{ env.REGISTRY_IMAGE }}:${{ steps.meta.outputs.version }} |
|
It's failing with: I think it's because the "Push" stage is missing the platforms. I'm trying with it now. |
|
Oh no. That's not it. It's because we build the images with the platforms as tags, then later we try to access with :latest. I think I know how to fix it. Trying now. |
|
After several attempts this is where I stopped: It does not make sense, it looks like TARGETPLATFORM is not being injected as the buildx docs says so. My PR is TECH7Fox/asterisk-hass-addons#251 in case you want to have a look. |
|
Anyway, this is a LOT of complication for such a simple task.
@crazy-max do you think it would be possible for buildx to support such a thing? I can open an issue there if you say so. |
Can you try with?: push:
runs-on: ubuntu-latest
needs:
- build
services:
registry:
image: registry:2
ports:
- 5000:5000
steps:
-
name: Download images
uses: actions/download-artifact@v3
with:
name: images
path: /tmp/images
-
name: Load images
run: |
for image in /tmp/images/*.tar; do
docker load -i $image
done
-
name: Push images to local registry
run: |
docker push -a ${{ env.TMP_LOCAL_IMAGE }}
-
name: Create manifest list and push to local registry
run: |
docker buildx imagetools create -t ${{ env.TMP_LOCAL_IMAGE }}:latest \
$(docker image ls --format '{{.Repository}}:{{.Tag}}' '${{ env.TMP_LOCAL_IMAGE }}' | tr '\n' ' ')
-
name: Set up Docker Buildx
uses: docker/setup-buildx-action@v2
with:
driver-opts: network=host
-
name: Login to Docker Hub
uses: docker/login-action@v2
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
-
name: Temp Dockerfile
run: |
mkdir -p /tmp/dkfilectx
echo "FROM ${{ env.TMP_LOCAL_IMAGE }}:latest" > /tmp/dkfilectx/Dockerfile
-
name: Docker meta
id: meta
uses: docker/metadata-action@v4
with:
images: ${{ env.REGISTRY_IMAGE }}
-
name: Push
uses: docker/build-push-action@v4
with:
context: /tmp/dkfilectx
push: true
tags: ${{ steps.meta.outputs.tags }}
platforms: linux/amd64,linux/arm/v6,linux/arm/v7,linux/arm64
labels: ${{ steps.meta.outputs.labels }}
-
name: Inspect image
run: |
docker buildx imagetools inspect ${{ env.REGISTRY_IMAGE }}:${{ steps.meta.outputs.version }}
Yes as said in docker/docs#17180 (comment) we could provide a composite action to ease the integration in your workflow. |
|
@crazy-max you have some references to LOCAL_IMAGE and TMP_LOCAL_IMAGE. Were they all supposed to be TMP_LOCAL_IMAGE like in the document (https://docs.docker.com/build/ci/github-actions/multi-platform/#distribute-build-across-multiple-runners)? |
|
Never mind, I think the answer is no. I'm testing here. |
That's a typo my bad, should be |
|
@crazy-max it worked! Thanks a lot! Just for information:
1 and 2 can be shaved to less than 1 minute if we switch from upload-artifact to actions cache, here is one example: TECH7Fox/asterisk-hass-addons#236 3 and 4 maybe can save 1 minute in total by leveraging some parallelism with GNU parallel. However, another approach that could potentially save time is to instead of using a local registry for the job, we could push the temporary images to GHCR itself with an unique tag and have a cleanup job that deletes the temporary images after. But still, nothing would beat both the speed and simplicity of: name: ci
on:
push:
branches:
- "main"
jobs:
build-and-push:
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
platform:
- linux/amd64
- linux/386
- linux/arm/v6
- linux/arm/v7
- linux/arm64
steps:
-
name: Checkout
uses: actions/checkout@v3
-
name: Set up QEMU
uses: docker/setup-qemu-action@v2
-
name: Set up Docker Buildx
uses: docker/setup-buildx-action@v2
-
name: Login to Docker Hub
uses: docker/login-action@v2
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
-
name: Build and push
uses: docker/build-push-action@v4
with:
context: .
platforms: ${{ matrix.platform }}
push: true
tags: user/app:latestIf |
Yeah, I realized. No worries! |
Made some changes to our example if you want to try: docker/docs#17305 |
|
I'll try for sure! I'll let you know soon. |
|
I think it can be sharpened a little bit by moving the metadata-action to its own job, like in here: Another thing, since I push images even for pull requests with the I just don't know how it would work with this push by digest stuff. For example, if I enable inline cache and push it by digest, how to I consume it back? Will they be retained when running the |
|
@crazy-max the result is amazing. A full build (with cache) now takes less than a minute: https://github.com/TECH7Fox/asterisk-hass-addons/actions/runs/4960678447/jobs/8876490603 Thanks a lot! |
I agree, this approach is great. When I implemented it, it ensured, that the test suites running during the build phase are successfully. When building all images with one runner, they tend to fail due to timeouts. @crazy-max I still have a problem with getting labels into the final manifest: I configured the workflow pretty much like you have posted in the last link. My workflow file is here, the relevant part is from line 446 to the end. Not sure if I miss something. Thanks in advance already! |
You are missing metadata-action in your build job. Double-check the example, metadata-action is ran twice, both in build and then in push. In build, you also need to supply the labels input. |
|
It's missing the labels input. |
|
Oh, I thought they are detected automatically.. I will double check. Thanks for the hint. |
|
Thanks again, the single
https://github.com/sando38/eturnal/pkgs/container/eturnal/92872612?tag=edge I included the labels. In the push job, I can also see that labels are included in the |
|
Hm... happened for me too: |
I know I can use the same runner to build all the platforms at the same time, but this causes my builds to take 2 hours instead of 20 minutes if I split to different runners.
I was able to achieve something similar with:
The problem is that it takes 10 minutes to upload the cache and then more 5 minutes to download the cache again.
Is there any suggestion to circumvent this?
The text was updated successfully, but these errors were encountered: