ci: Build ARM images for core components

[kimwnasptd]

This PR extends the GH Actions to also build the "core" images in kubeflow/kubeflow for ARM. This is for kubeflow/manifests#2472 and #2337

The last step is to then modify the build system and makefiles for the example notebook images https://github.com/kubeflow/kubeflow/tree/master/components/example-notebook-servers

[hwang9u]

I'm a beginner at Kuberflow, so I didn't know how to promote it. Thank you for your help! Have a nice day 🤩

[kimwnasptd]

@thesuperzapper can I get a review on this ASAP? The CI is broken right now, due to the multi-arch command having a redundant comma https://github.com/kubeflow/kubeflow/actions/runs/5717274226/job/15490659878

This PR should both fix that issue and also introduce ARM support for most of core components, aside from the notebook server images.

[thesuperzapper]

@kimwnasptd I want to clarify that we have actually tested the building with ARM64? Because there is no guarantee they will all work, and we probably don't want to push busted images.

---

Also, in my experience, building complex ARM64 images under QEMU emulation can take a LONG time (and might never finish in extreme cases). In the past, I mitigated this problem using the docker buildx registry cache feature in two ways.

1: Use a separate "ci" GitHub Container Registry (which is local to the runners, and fully supports caching, unlike ECR):

CI_REGISTRY_IMAGE="ghcr.io/<IMAGE_OWNER>/ci/<IMAGE_TITLE>

docker buildx build \
  --cache-from=type=registry,ref=${CI_REGISTRY_IMAGE} \
  --cach-to=type=registry,ref=${CI_REGISTRY_IMAGE},mode=max \
  ./path/to/Dockerfile

2: For builds that never finish, manually populate the cache on an ARM device (like a MacBook):

CI_REGISTRY_IMAGE="ghcr.io/<IMAGE_OWNER>/ci/<IMAGE_TITLE>"

docker buildx build \
  --cache-to=type=registry,ref=${CI_REGISTRY_IMAGE},mode=max \
  ./path/to/Dockerfile

---

If we are going to do lots of cross-architecture builds I think we must implement the cache feature, and it should be relatively straightforward to host the "ci" image cache on GitHub Container Registry (GHCR).

PS: they are a bit out of date now, but a while ago, I mirrored all the old images onto the Kubeflow Org's GHCR. We can make "ci" versions of all of them in a similar way (and possibly also push the non-ci image there too): https://github.com/orgs/kubeflow/packages

[kimwnasptd]

@thesuperzapper thanks for the review!

@kimwnasptd I want to clarify that we have actually tested the building with ARM64? Because there is no guarantee they will all work, and we probably don't want to push busted images.

Yes, I can confirm they can be build. I've tested all of them in an M2 MacBook and all the above components can be successfully built and run.

If we are going to do lots of cross-architecture builds I think we must implement the cache feature, and it should be relatively straightforward to host the "ci" image cache on GitHub Container Registry (GHCR).

Regarding the cache, I'd say let's evaluate this once we have some numbers first and the performance is indeed very poor. Looking at other projects like KServe they are also building for both x86 and ARM architectures and the builds are quite snappy.

https://github.com/kserve/kserve/actions/runs/5719086923/job/15496326007
https://github.com/kserve/kserve/blob/master/.github/workflows/sklearnserver-docker-publish.yml
https://hub.docker.com/layers/kserve/sklearnserver/latest/images/sha256-c010cf2e69fa014062b1b424014e0cc9cc58e170287caba827216480c02f1d37?context=explore

So I'd suggest to move forward with this and if we see that indeed the actions never complete we can re-evaluate going down a more complicated route afterwards.

[kimwnasptd]

/hold

Let me actually fully flesh this out in a local fork, to make sure we have some numbers and we can re-evaluate

[kimwnasptd]

/hold cancel

@thesuperzapper indeed most the build jobs finish in less than 30 mins. So I we should be good to go for now and not require a cache. You can see a run of GH workflows in the tests from this commit of my fork kimwnasptd@1e02d1e

Note that the 3 crud-web-apps are failing to build, but this is on ppc64le platform and is being tracked in #7226

I've verified that these three images can be built, and also tested them in my cluster. You can always check if an image for any architecture can be build locally with:

cd components/crud-web-apps/jupyter
ARCH=linux/arm64/v8 make docker-build-multi-arch

Since docker buildx is using qemu to emulate the different platforms https://docs.docker.com/build/building/multi-platform/

[kimwnasptd]

Note, I had to do a bump in the python and gevent versions in order for these images to be able to be built for ARM, due to gevent/gevent#1721

[thesuperzapper]

@kimwnasptd so are we planning to get this merged before 1.8, because I would like to!

However, I see that there is now a merge conflict.

[thesuperzapper]

@kimwnasptd Also, I see that in kimwnasptd@1e02d1e you are also updating the example notebook images, I cant see the update to those images in this PR, did I just miss it?

[thesuperzapper]

@kimwnasptd we really need to move this forward so that Kubeflow 1.8 works on ARM!

[pingsutw]

+1, we need the image for ARM.

[kimwnasptd]

@thesuperzapper ping to take a look at the current state. All the tests are passing, but I have a commit that triggered all of them which I'll drop once I get an ACK.

The changes at this point are:

1. New workflows for building for multi-arch during a PR (this way the integration tests run way faster)
2. The workflows that build/publish images will build for all archs

With the above we will always be testing on each PR that images can be built accordingly

[kimwnasptd]

Had a sync with @thesuperzapper. We are OK to merge for now, although there are a couple of improvements to make afterwards, like:

1. Have a common action for a couple of our workflows
2. Use a matrix for building each image for each different platform, to build in parallel
3. Move over to ghcr.io to host our images and also utilise a cache

@thesuperzapper I'm moving on with removing the DROP commits so that we can merge

[kimwnasptd]

@thesuperzapper pushed the changes. The actions for the three webapp got triggered since there is a commit for changing some versions, to make sure the apps work in ARM/ppc64le.

Once the tests are good we can merge

[kimwnasptd]

@thesuperzapper should be ready to merge

[thesuperzapper]

/lgtm
/approve

[google-oss-prow]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: thesuperzapper

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• .github/OWNERS [thesuperzapper]
• components/crud-web-apps/OWNERS [thesuperzapper]
• releasing/OWNERS [thesuperzapper]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment