support for arm64

[Prashanth684]

with the developer preview of ARM available for 4.9, adding support in it for SNC so we can work toward having crc support for macs

[openshift-ci]

Hi @Prashanth684. Thanks for your PR.

I'm waiting for a code-ready member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[gbraad]

We do not have the equipment to confirm this works. Can you provide tests, or a means to ensure consistency/stability for follow-up changes we make to the general SNC codebase.

[praveenkumar]

/ok-to-test

[Prashanth684]

We do not have the equipment to confirm this works. Can you provide tests, or a means to ensure consistency/stability for follow-up changes we make to the general SNC codebase.

so "we" as in the multi-arch team at RedHat use ARM systems that we loan from beaker. That being said long term, the plan is to have CI on ARM starting mainly with AWS but we could also look to extend it to have a CI setup for crc which would be necessary. Let me bring this up for further discussion in our forums.

[Prashanth684]

/test e2e-snc

[cfergeau]

If these are the only changes needed to have snc running successfully on aarch64 hardware, then I'm happy to see it's so straightforward :)
Only remaining question is how we do test this.

Fwiw, if no additional changes are needed, I'd be fine with merging this even if aarch64 is unsupported from our end.

[praveenkumar]

@Prashanth684 Have you tested this PR, like generated bundle for arm and able to run those even on beaker infra?

[Prashanth684]

@Prashanth684 Have you tested this PR, like generated bundle for arm and able to run those even on beaker infra?

good point @praveenkumar . the single node cluster itself came up fine, but the createdisk.sh command to create the images pulls amd64 images from quay which will not work. can you point us to how to build these images so we can build them manifest listed for amd64 and arm64? these are the images in question:

quay.io/crcont/dnsmasq:latest
quay.io/crcont/gvisor-tap-vsock:v4
quay.io/crcont/routes-controller:latest

[cfergeau]

The dnsmasq image is there https://github.com/code-ready/crc/tree/master/images/dnsmasq
The gvisor-tap-vsock one is https://github.com/containers/gvisor-tap-vsock/tree/main/images
The routes-controller one is https://github.com/code-ready/routes-controller/tree/master/images

Some of these images can be built on osbs, I'd have to check if it can be used to generate aarch64 images.

[praveenkumar]

@Prashanth684 The images are built on osbs (Thanks to @cfergeau) and uploaded to quay so it should pull platform specific.

• https://quay.io/repository/crcont/dnsmasq?tab=tags
• https://quay.io/repository/crcont/routes-controller?tab=tags
• https://quay.io/repository/crcont/gvisor-tap-vsock

[Prashanth684]

many thanks again @cfergeau ! i will use these and run tests again and let you know.

[Prashanth684]

ok. updated and rebased. i was able to generate a crc bundle. @praveenkumar @cfergeau any docs on how i can run crc with a custom bundle?

[cfergeau]

any docs on how i can run crc with a custom bundle?

@Prashanth684 if you build crc from git, crc setup; crc start -b $custom_bundle should be all you need. Couple of caveats:

• even if crc builds on aarch64, I don't know how well the preflights will work there
• at this point we support OpenShift 4.8, not sure how well OpenShift 4.9 or newer would run with crc from git

[cfergeau]

Looks good to me, a few questions/suggestions, but nothing blocking.

[cfergeau]

oh, oc became strict about parameter order?

[Prashanth684]

yes looks like it....this used to work for me before too

[cfergeau]

The commands from ostree_hash= to here could actually be wrapped in a if [ -n "${SNC_GENERATE_MACOS_BUNDLE}" ] as it's not going to be useful on linux/Windows.
I don't think hyperkit (which crc uses on macOS) works on M1 macs, so it might be a bit early to generate macOS bundles by default in snc.
Could you at least expand the commit log a bit, is this causing an error if we omit the -r? Which additional files do we need from /boot/ostree which are in subdirs?

[Prashanth684]

so for aarch64 the kernel pulls in DTB - https://src.fedoraproject.org/rpms/kernel/blob/f34/f/kernel.spec#_1533 as part of the bootloading process which is the extra directory that requires the -r.

Yes, so looks like hyperkit is not yet available for arm64 so I would just want to get these changes in for now so that in the future when it is available we can start on the CRC side of things.

[Prashanth684]

changed to have this block under the conditional.

[cfergeau]

/retest

[Prashanth684]

any docs on how i can run crc with a custom bundle?

@Prashanth684 if you build crc from git, crc setup; crc start -b $custom_bundle should be all you need. Couple of caveats:

• even if crc builds on aarch64, I don't know how well the preflights will work there
• at this point we support OpenShift 4.8, not sure how well OpenShift 4.9 or newer would run with crc from git

So i did try and the pre-flight check failed with finding the virtualization device which doesn't seem to be present on any hardware we have today. So, i think for now I would just like to get this in so we at least have support to create bundles so we can investigate more when we get an ARM system with virtualization hardware or hopefully an M1 which we can test with. But again as @cfergeau pointed out hyperkit is not supported for arm64 so we will also have to wait for that.

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: cfergeau, Prashanth684

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [cfergeau]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[Prashanth684]

/retest

[openshift-ci]

@Prashanth684: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Rerun command
ci/prow/e2e-snc	f98a397	link	/test e2e-snc

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.