Include unit/integration tests for Linux/ARM64 in GitHub Actions CI

[odidev]

Summary

Hi Team,

I am working with ARGO-CD on the Linux/ARM64 platform.

ARGO-CD already releases Linux/ARM64 binaries and the docker images. However, there are unit tests and end-to-end tests running via GitHub Actions for Linux/AMD64 via ci-build.yaml file. It would be great if these tests were also available for Linux/ARM64 via GitHub Actions CI.

I have tried running these tests for Linux/ARM64 using self-hosted ARM64 GitHub runners in my forked repo, but the tests are failing.

Unit tests are failing for bad credentials, as below:

error listing repositories for argoproj: GET https://api.github.com/orgs/argoproj/repos?per_page=100: 401 Bad credentials [] 

The failure logs are same for Linux/AMD64 in my forked repo’s GA job.

e2e tests are also failing for Linux/ARM64, to start k3s service on runners. However, tests are failing for my forked repo’s Linux/AMD64 jobs with the logs below:

Error: applicationset_test.go:402: Waiting 30 seconds to give the cluster a chance to delete the pods. 

May I know, do you have plans to include integration tests for Linux/ARM64 in the GitHub Actions CI?

Kindly share your opinion on the same.

[crenshaw-dev]

CI is already fairly slow/crowded, so I'm hesitant to add anything else without really compelling reasons.

How would you feel about adding checks that run only a subset of tests on ARM64? As bugs are discovered, we could identify tests that are platform-sensitive and add them to the test list.

[odidev]

@crenshaw-dev

Thank you for the response.

How would you feel about adding checks that run only a subset of tests on ARM64?

Yes, that would be a reasonable compromise.

May I ask if there are any subset tests that you think should be enabled currently?

[crenshaw-dev]

I'm fairly new to the code base, so I can't think of any tests off-hand. But we could start with the ones that are failing for you and just add more as problems appear.

[odidev]

But we could start with the ones that are failing for you and just add more as problems appear.

Actually, my forked repo isn’t failing for platform-specific tests. Unit tests are failing commonly for both AMD64 and ARM64 for bad credentials as explained above. I don’t think this is a platform related error.

And e2e tests are failing for ARM64 to start k3s service. This is not a platform related issue since k3s supports ARM64. This is for some reason that k3s isn’t getting started on my ARM64 self-hosted runner. Since runner uses our local systems to configure jobs to GitHub Actions, the results may vary from runner to runner.

[34fathombelow]

@odidev As @crenshaw-dev mentioned the CI process is crowded and slow. I have been playing around with LXC containers on GitHub runners. I might be able to get unit tests and e2e tests running through a LXC container running on qemu and emulate the arm64 architecture. This is going to be much much slower than running on a native architecture. If the test times are somewhat reasonable, we might be able to run the test nightly or periodically. Let me know if this seems to a reasonable compromise or if you have any feedback.

[odidev]

@34fathombelow,

Yes, this looks like a reasonable approach to running tests. It would be great if the tests are executed periodically for ARM64, considering there are time/resource constraints.

[odidev]

@34fathombelow , may I know, is there any next step that I should start working on?

[34fathombelow]

@odidev I'm currently trying several different options. I have been busy for the past few days. Hope to have more information by the weekend, feel free to find me on the Argo slack channels if you want to discuss this in further detail.

[odidev]

Hi Team, a gentle reminder!!! Are there any plans/possibilities to add testing for Linux/ARM64 via CI in the near future?

[dneary]

Hi! This seems like a good ticket to raise an option: We (Ampere Computing) are sponsoring a pilot with the CNCF to offer hosted runners using Actuated to run Arm64 GitHub Actions on Equinix Metal. The changes to CI are very minor (you add "actuated(plus some metadata on # of cores and amount of RAM the runner needs)" to your projects' "runs-on:".

@crenshaw-dev, if you're interested, I can connect you to Alex Ellis to on-board you, at the very least, this should accelerate your image builds for Arm64 dramatically. At best, I hope that you will be able to run the full suite of tests on the Arm builds also. More information about the pilot project here: https://actuated.dev/blog/arm-ci-cncf-ampere

[alexellis]

Hi folks, a quick update.

Actuated has been installed on the argoproj organisation by @crenshaw-dev and we identified a few initial repos/builds on a recent call together.

containerd just completed their migration for unit and e2e tests, and will be targeting their release process next.

Michael, when do you expect to get started with one of the builds?

Alex