GitHub-hosted arm64 runners

[Qard]

Many devs have migrated to M1 laptops and there's a growing movement of migrating servers to arm64 linux. It would be great if there were official arm64 runners. Custom runners is not a great solution for something that is rapidly becoming the default case.

[rochdev]

It's worth noting that most competitors like for example CircleCI already support this. Heck, even Travis supports this.

[a-h]

The Github Actions home page makes it look like ARM is supported as a 1st class feature, when it clearly isn't.

https://github.com/features/actions

I don't even see ARM or ARM64 on the roadmap at https://github.com/orgs/github/projects/4247/views/1?filterQuery=ARM

[jonesmz]

This also confused and annoyed me. Its the whole reason I'm even on this post because I was trying to figure out why I was lied to by the section heading on their advertising copy.

[mustafaergul]

Travis CI has supported the Arm64 for some time, I think worth to try:
https://docs.travis-ci.com/user/multi-cpu-architectures

[lukasz-mitka]

You can try this https://github.com/docker/setup-qemu-action in the meantime.
Won't be very efficient and may not work for everything, but you should be able to build arm docker image for example.

[a-h]

Thanks, I'll give that a shot. If fixes it, then the problem must be on the cdk side, perhaps not appropriately setting the platform or not being aware of buildx or something.

Even if it works, I'm worried about the relative complexity and fragility of the solution, since I don't know what else could be broken.

[edhemphill]

Could you not just run the aarch64 github runner in a VM Linux running on M-series Mac?

[lukasz-mitka]

The pricing for Mac runners is ridiculous
https://docs.github.com/en/billing/managing-billing-for-github-actions/about-billing-for-github-actions#per-minute-rates

Linux 2 $0.008
macOS 6 (M1) $0.16

Even after adjusting for number of cores it's over 6 times more expensive.
So I guess if qemu is slower than that it could work. Did anyone try it?

[JanneKiiskila]

You really do not need an Apple/Mac for Arm64/ aarch64 jobs. You could use some arm64-server instead, which would be very, very cost-efficient (due to way smaller power consumption than Intel/AMD alternatives).

• AWS has the Gravitron instances, they are very cost-effective.
• Those are of course AWS specific, but you can get for example the Ampere solutions.
• Azure (by Microsoft) already has those and since GitHub belongs to Microsoft nowadays as well, they should be able to leverage those? Ref: https://azure.microsoft.com/en-us/blog/azure-virtual-machines-with-ampere-altra-arm-based-processors-generally-available/

Would be very neat to get some update from someone in GitHub and/or Microsoft to comment on what's the hold-up here?

[Qard]

You do need a mac target if you're trying to test for M* Apple devices. Given their ubiquity with developers there should definitely be runners for that.

[maybedino]

Almost every cloud provider, including Azure, has arm64 servers now. But building arm64 container images on Github still requires QEMU which is so slow, it is close to being unusable. A ~2 minute amd64 build can take 30+ minutes on arm64. Native arm64 runners would be great, as more people are switching to arm64 servers and Github is lagging behind here.

[lukasz-mitka]

Well, arm64 doesn't have AI buzz to it so GitHub doesn't care :(

[JanneKiiskila]

Well, Arm64 should be perfectly fine for that. You can run amazing stuff even on the Arm microcontrollers.

• tinyML etc.
• https://alifsemi.com/ does also the Ensemble chips with AI accelerators.

[edhemphill]

All next generation AI purpose built chips will use aarch64 CPUs as part of the entire unit. i.e. you wont need an x86 CPU in there. Example: https://www.anandtech.com/show/20001/nvidia-unveils-gh200-grace-hopper-gpu-with-hbm3e-memory

GH200 uses an Arm Neoverse v2 CPU core. Google, Amazon doing the same stuff. All public info.

[janeklb]

Is it possible to update this discussion's title to be "Github-hosted arm64 runners"? I think that's what most people are asking for (since it's been possible to run self-hosted arm64 runners for a good while now). cc @Qard

[lukasz-mitka]

Arm-based hosted runners are coming to GitHub Actions!
[...]
These runners will be entering private beta in January 2024.

https://github.blog/changelog/2023-10-30-accelerate-your-ci-cd-with-arm-based-hosted-runners-in-github-actions/

Finally!

[debu99]

just private beta? i guess still need wait until middle of 2024 to use

[jbrsb]

Hi, does anyone have a date or any news on this feature ?

[relistan]

Really want this ASAP

[matti]

just private beta? i guess still need wait until middle of 2024 to use

@debu99 middle of 2024 might be too optimistic

[Jolg42]

I think this can be closed now? 🤔

EDIT: For macOS only:

See https://github.blog/changelog/2024-01-30-github-actions-introducing-the-new-m1-macos-runner-available-to-open-source/

Today, GitHub is excited to announce the launch of a new M1 macOS runner! This runner is available for all plans, free in public repositories, and eligible to consume included free plan minutes in private repositories. The new runner executes Actions workflows with a 3 vCPU, 7 GB RAM, and 14 GB of storage VM, which provides the latest Mac hardware Actions has to offer. The new runner operates exclusively on macOS 14 and to use it, simply update the runs-on key in your YAML workflow file to macos-14.

See https://github.blog/changelog/2024-01-30-github-actions-macos-14-sonoma-is-now-available/

The macOS 14 runner image is now available for GitHub hosted runners. Workflows executed on this image will run exclusively on the 3 vCPU M1 runner announced earlier today. To use the runner, simply update the runs-on: key in your YAML workflow file to macos-14, macos-14-xlarge, or macos-14-large.

I tried it and it works very well ✨
Thanks to the people at GitHub, this was probably a monumental effort behind the scenes 🙌🏼

[Feichtmeier]

this is about linux arm runners

[Jolg42]

Right, it was confusing, because it was mentioning M1 😄

[WeiChungHsu]

Can I use macos-14 runner with the service container such as debian or ubuntu?
I want to build arm64 linux kernel and/or arm64 architecture specific codes, but NOT to limited in macos.

I search github document and it says the service container only works with ubuntu runner. Does it mean macos-14 arm64 runner can't work with service container?

[nebuk89]

Hey! Yes ARM is currently in private beta while we work on ensuring we have an image that provides a good set of tools and we get our capacity all sorted :) it will be nearer the end of this year that we get our new default labels supporting ARM - but we are working on it actively <3

[nebuk89]

as @lukasz-mitka highlighted :D https://github.blog/changelog/2023-10-30-accelerate-your-ci-cd-with-arm-based-hosted-runners-in-github-actions/

If y'all sign up there we are batching folks in to what we have today (folks == Orgs/enterprises only currently as we are relying on the central runner groups)

[jacobsvante]

@nebuk89 When can we expect to see a wider beta release?

[nebuk89]

Good question, at the Org level we are aiming for June/July. For individual accounts later in the year (hopefully no later than end of Nov but depends on capacity and how many people use them!)

[jacobsvante]

June/July sounds promising!

[LouisBoudreau]

Thanks @nebuk89 for your reply!

[relistan]

I'm personally looking for linux ARM64 runners, so macOS runners don't do much for me at the moment.

We're using ARM64 on everyone's desktop and ARM64 in production and GitHub can't build images for that chain without emulating ARM64 on AMD64.

[crohr]

The RunsOn tool supports both arm64 and x64 and is a drop-in replacement. Might be worth a look.

[robwafle]

How can someone get included in the private beta? I could use an ARM64 Ubuntu runner today.

[lukasz-mitka]

There's a link at the end of this post
https://github.blog/changelog/2023-10-30-accelerate-your-ci-cd-with-arm-based-hosted-runners-in-github-actions/

[filipRisteski]

I went with ARM deployment for production, and I am very surprised that Github doesn't support this.

[kei1987]

Arm64 for macos runner is available but docker is not installed, and nested virtualization is not possible. No way to build a docker image in arm64 architecture as of now?

[patrick-stephens]

This is clarified in the statement that comes along with using the beta explaining that it is not included. You can install docker like any other ubuntu server though - we did that to verify the builds work fine. My understanding is that public beta will likely ensure this is included.

[kei1987]

yep, we are actually using ubuntu-latest with QEMU instead. It can build an image for arm64 arch but takes over 15 minutes to complete, which is not ideal for CI. Let's wait for the public beta

[patrick-stephens]

Ah sorry, you are talking about macOS runners not the Ubuntu ARM ones.
We're just building ARM images on the Ubuntu ones, not macOS. Containers on mac always needs more effort so never bother :)

[luong-komorebi]

Instead of waiting for Github hosted version of the runner, one can always rely on some third party runner service offering like https://www.warpbuild.com/ and get your ARM builds on native arm machine right now

[SanderVocke]

While we wait for publicly available Linux arm runners, I have made a marketplace Action which can run entire workflow runs transparently in a QEMU-emulated container on top of a x86_64 runner, for a "close-to-built-in" experience that doesn't require VMs. It has limitations for sure, but could be a solution for some projects:

https://github.com/SanderVocke/setup-qemu-container

[maybedino]

For docker, if your language supports cross-compilation (like go), or produces platform-independent code (like c# or ts), you can use multi stage builds to build on the runner architecture and just copy the result into the final image like this:
https://docs.docker.com/build/building/multi-platform/#cross-compilation
Which will be much faster.

More details: https://www.docker.com/blog/faster-multi-platform-builds-dockerfile-cross-compilation-guide/

[alec-pinson]

For docker, if your language supports cross-compilation (like go), or produces platform-independent code (like c# or ts), you can use multi stage builds to build on the runner architecture and just copy the result into the final image like this: https://docs.docker.com/build/building/multi-platform/#cross-compilation Which will be much faster.

More details: https://www.docker.com/blog/faster-multi-platform-builds-dockerfile-cross-compilation-guide/

Thanks @maybedino this gave a massive improvement for me!

before

#32 [linux/arm64 build_base 10/10] RUN cd cmd/frontend &&     go build -o /build/out/my-app .
#32 DONE 292.0s

after

#27 [linux/amd64->arm64 build_base 10/10] RUN cd cmd/frontend &&     GOOS=linux GOARCH=arm64 go build -o /build/out/my-app .
#27 DONE 41.8s

[SanderVocke]

We've just been using https://github.com/docker/setup-qemu-action for quite a while now but the issue is whilst this works it is super slow on emulated targets.

In many cases that may be the more straightforward choice. In my case I want to build a complicated Python package in a manylinux container, where the entire container should be ARM. In particular, I want to run multiple GHA workflow steps in the same emulated container, which I think is not possible with setup-qemu-action without explicitly using qemu in every step.

I don't want to build my software as a Docker image, as that would involve migrating a complex set of Github Actions scripts to a Dockerfile.

[crohr]

@SanderVocke is your repo open-source? In which case you can setup RunsOn for free and benefit from the fastest ARM processors around - https://runs-on.com/reference/benchmarks-gha-providers/#arm64-runners

[SanderVocke]

@SanderVocke is your repo open-source? In which case you can setup RunsOn for free and benefit from the fastest ARM processors around - https://runs-on.com/reference/benchmarks-gha-providers/#arm64-runners

Wow, thanks, I'll check it out! Honestly, I had been looking for something like this before making the emulation wrapper. But I couldn't find any runner hosts that offered free plans for open-source projects. Great to know that it is out there and I will definitely give it a try.