Add support for RISC-V #7778
Add support for RISC-V #7778
Conversation
|
Also note that this uses a prepackaged k3s-root we have manually compiled |
|
I'm not sure we're ready to do this until we have some hardware on our side to support native building and testing, for the CI runners. There are some talks occurring between suse and a risc-v ISV to acquire development hardware. We will also need to begin mirroring images for this platform; the rancher image mirror scripts currently filter it out, even if upstream does offer them - which many do not. |
|
Updated with a new commit that uses k3s-root v0.13.0 with RISC-V support. I also had to use a patched version of runc so it allows images built for Next, I will look into integrating the build script (which has been moved into the |
Yes, I'm looking into this. It looks like the only practical issue is that Dockerfile.dapper is based on Alpine that does not have all the requirements for cross-compiling available. My plan is to start with another Dockerfile, based on Debian (I have only tried Ubuntu up to now). |
Hmm, that might be a challenge, as we intentionally build with alpine and musl libc. We wouldn't want to ship binaries built on a different platform for just one arch. I think alpine has some helper scripts that are used to set up a cross build environment, see https://github.com/alpinelinux/aports/blob/master/scripts/bootstrap.sh |
Ok. I will try using that. Thanks. |
|
I just committed a version that uses Alpine to cross-build for RISC-V instead of Ubuntu. I implemented a I have also integrated cross-building with the Dapper environment, meaning that I successfully built a binary on my laptop, by just running: ARCH=riscv64 SKIP_IMAGE=true SKIP_VALIDATE=true SKIP_AIRGAP=true make
|
Codecov ReportPatch coverage has no change and project coverage change:
Additional details and impacted files@@ Coverage Diff @@
## master #7778 +/- ##
==========================================
+ Coverage 47.05% 51.37% +4.31%
==========================================
Files 143 143
Lines 14561 14561
==========================================
+ Hits 6852 7481 +629
+ Misses 6616 5890 -726
- Partials 1093 1190 +97
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
|
The pipeline produces a working RISC-V binary. Running in QEMU: root@ubuntu:~# wget https://k3s-ci-builds.s3.us-east-1.amazonaws.com/k3s-riscv64-604a9b056440205ad253ac7eb7c50133e7c380f7
...
root@ubuntu:~# cp k3s-riscv64-* /usr/local/bin/k3s
root@ubuntu:~# chmod +x /usr/local/bin/k3s
root@ubuntu:~# curl -sfL https:/INSTALL_K3S_EXEC="server --disable traefik,metrics-server --pause-image carvicsforth/pause-riscv:v3.9-v1.27.2" INSTALL_K3S_SKIP_DOWNLOAD="true" bash k3s-install.sh
[INFO] Skipping k3s download and verify
[INFO] Skipping installation of SELinux RPM
[INFO] Creating /usr/local/bin/kubectl symlink to k3s
[INFO] Creating /usr/local/bin/crictl symlink to k3s
[INFO] Creating /usr/local/bin/ctr symlink to k3s
[INFO] Creating killall script /usr/local/bin/k3s-killall.sh
[INFO] Creating uninstall script /usr/local/bin/k3s-uninstall.sh
[INFO] env: Creating environment file /etc/systemd/system/k3s.service.env
[INFO] systemd: Creating service file /etc/systemd/system/k3s.service
[INFO] systemd: Enabling k3s unit
Created symlink /etc/systemd/system/multi-user.target.wants/k3s.service → /etc/systemd/system/k3s.service.
[INFO] systemd: Starting k3s
root@ubuntu:~# kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
ubuntu Ready control-plane,master 50s v1.27.3+k3s-604a9b05 10.0.2.15 <none> Ubuntu 22.04.2 LTS 5.19.0-1012-generic containerd://1.7.1-k3s1
root@ubuntu:~# kubectl apply -f https://raw.githubusercontent.com/CARV-ICS-FORTH/kubernetes-riscv64/main/ubuntu.yaml
pod/ubuntu created
root@ubuntu:~# kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
ubuntu 1/1 Running 0 61s 10.42.0.4 ubuntu <none> <none>
root@ubuntu:~# kubectl exec -it ubuntu -- cat /etc/os-release
PRETTY_NAME="Ubuntu 22.04.1 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.1 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy
root@ubuntu:~# |
|
Rebased. |
Signed-off-by: Antony Chazapis <chazapis@ics.forth.gr>
…d script in scripts folder Signed-off-by: Antony Chazapis <chazapis@ics.forth.gr>
Signed-off-by: Antony Chazapis <chazapis@ics.forth.gr>
Signed-off-by: Antony Chazapis <chazapis@ics.forth.gr>
Signed-off-by: Antony Chazapis <chazapis@ics.forth.gr>
Signed-off-by: Antony Chazapis <chazapis@ics.forth.gr>
|
Rebased, updated runc with the upstream version that supports RISC-V, and resolved all discussions. The current status is:
If there was a way to define the required container images per architecture, the RISC-V version would work out of the box. Now, you can either patch running Pods to use the correct images or use a prebuilt binary with the Sample output from QEMU: root@ubuntu:~# kubectl get nodes
NAME STATUS ROLES AGE VERSION
ubuntu Ready control-plane,master 9h v1.27.3+k3s-9d376dfb-dirty
root@ubuntu:~# kubectl get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system helm-install-traefik-crd-49rhb 0/1 Completed 0 9h
kube-system helm-install-traefik-psnrr 0/1 Completed 2 9h
kube-system svclb-traefik-c0171ed3-9fjr9 2/2 Running 2 (20m ago) 9h
kube-system traefik-8657d6b9f4-j2znf 1/1 Running 1 (20m ago) 9h
kube-system coredns-97b598894-nf45g 1/1 Running 1 (20m ago) 9h
kube-system metrics-server-7c55d89d5d-nppqs 1/1 Running 0 9h
kube-system local-path-provisioner-6d44f4f9d7-gvmv6 1/1 Running 2 (19m ago) 9h
root@ubuntu:~# kubectl top pods -A
NAMESPACE NAME CPU(cores) MEMORY(bytes)
kube-system coredns-97b598894-nf45g 27m 14Mi
kube-system local-path-provisioner-6d44f4f9d7-gvmv6 5m 6Mi
kube-system metrics-server-7c55d89d5d-nppqs 41m 17Mi
kube-system svclb-traefik-c0171ed3-9fjr9 0m 2Mi
kube-system traefik-8657d6b9f4-j2znf 23m 27Mi |
Signed-off-by: Antony Chazapis <chazapis@ics.forth.gr>
|
I was going to try to rebase/squash this onto current master, but you didn't set the PR to allow maintainers to contribute to the branch. |
|
Rebased/squashed PR looks good on my cluster of Unmatched boards. I just can't run anything on it because none of our images currently support risc64; I'm working on building images for that. unmatched01:~ # kubectl get node -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
unmatched01.lan.khaus Ready control-plane,etcd,master 27m v1.28.1+k3s-b44521db 10.0.1.182 <none> openSUSE Tumbleweed 6.4.12-1-default containerd://1.7.3-k3s2
unmatched02.lan.khaus Ready control-plane,etcd,master 19m v1.28.1+k3s-b44521db 10.0.1.212 <none> openSUSE Tumbleweed 6.4.12-1-default containerd://1.7.3-k3s2
unmatched03.lan.khaus Ready control-plane,etcd,master 14m v1.28.1+k3s-b44521db 10.0.1.188 <none> openSUSE Tumbleweed 6.4.12-1-default containerd://1.7.3-k3s2
unmatched01:~ # kubectl describe node unmatched01.lan.khaus
Name: unmatched01.lan.khaus
Roles: control-plane,etcd,master
Labels: beta.kubernetes.io/arch=riscv64
beta.kubernetes.io/instance-type=k3s
beta.kubernetes.io/os=linux
kubernetes.io/arch=riscv64
kubernetes.io/hostname=unmatched01.lan.khaus
kubernetes.io/os=linux
node-role.kubernetes.io/control-plane=true
node-role.kubernetes.io/etcd=true
node-role.kubernetes.io/master=true
node.kubernetes.io/instance-type=k3s
Annotations: etcd.k3s.cattle.io/node-address: 10.0.1.182
etcd.k3s.cattle.io/node-name: unmatched01.lan.khaus-eaad4846
flannel.alpha.coreos.com/backend-data: {"VNI":1,"VtepMAC":"2a:55:42:6d:2c:da"}
flannel.alpha.coreos.com/backend-type: vxlan
flannel.alpha.coreos.com/kube-subnet-manager: true
flannel.alpha.coreos.com/public-ip: 10.0.1.182
k3s.io/hostname: unmatched01.lan.khaus
k3s.io/internal-ip: 10.0.1.182
k3s.io/node-args:
["server","--token","********","--cluster-init","true","--disable-servicelb","true","--disable","coredns","--disable","traefik","--disable...
k3s.io/node-config-hash: WSHDY5WIYDGUS2ZMBAHTM43P2NZGQLSFI2O6OJYZJ4UFH5TL2VAQ====
k3s.io/node-env: {"K3S_DATA_DIR":"/var/lib/rancher/k3s/data/80d685a8843169e5974bea73582cc22a05f19746bdb4ecc381eeb29d1c691e2e"}
node.alpha.kubernetes.io/ttl: 0
volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp: Mon, 11 Sep 2023 18:45:03 +0000
Taints: <none>
Unschedulable: false
Lease:
HolderIdentity: unmatched01.lan.khaus
AcquireTime: <unset>
RenewTime: Mon, 11 Sep 2023 19:12:55 +0000
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
MemoryPressure False Mon, 11 Sep 2023 19:12:47 +0000 Mon, 11 Sep 2023 19:02:34 +0000 KubeletHasSufficientMemory kubelet has sufficient memory available
DiskPressure False Mon, 11 Sep 2023 19:12:47 +0000 Mon, 11 Sep 2023 19:02:34 +0000 KubeletHasNoDiskPressure kubelet has no disk pressure
PIDPressure False Mon, 11 Sep 2023 19:12:47 +0000 Mon, 11 Sep 2023 19:02:34 +0000 KubeletHasSufficientPID kubelet has sufficient PID available
Ready True Mon, 11 Sep 2023 19:12:47 +0000 Mon, 11 Sep 2023 19:02:34 +0000 KubeletReady kubelet is posting ready status. AppArmor enabled
Addresses:
InternalIP: 10.0.1.182
Hostname: unmatched01.lan.khaus
Capacity:
cpu: 4
ephemeral-storage: 961544732Ki
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 16359916Ki
pods: 110
Allocatable:
cpu: 4
ephemeral-storage: 935390714556
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 16359916Ki
pods: 110
System Info:
Machine ID: e655aba1732d4799bd8bbcbaf8a3f4ab
System UUID: e655aba1732d4799bd8bbcbaf8a3f4ab
Boot ID: 0ddbcbc0-1f30-4b4c-ab4d-240eedba6496
Kernel Version: 6.4.12-1-default
OS Image: openSUSE Tumbleweed
Operating System: linux
Architecture: riscv64
Container Runtime Version: containerd://1.7.3-k3s2
Kubelet Version: v1.28.1+k3s-b44521db
Kube-Proxy Version: v1.28.1+k3s-b44521db
PodCIDR: 10.42.0.0/24
PodCIDRs: 10.42.0.0/24
ProviderID: k3s://unmatched01.lan.khaus
Non-terminated Pods: (0 in total)
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits Age
--------- ---- ------------ ---------- --------------- ------------- ---
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits
-------- -------- ------
cpu 0 (0%) 0 (0%)
memory 0 (0%) 0 (0%)
ephemeral-storage 0 (0%) 0 (0%)
hugepages-1Gi 0 (0%) 0 (0%)
hugepages-2Mi 0 (0%) 0 (0%) |
@brandond, I'm searching for a way to change this, but it looks like it is not supported, since the fork is under an organization account... I see however that you created another branch with the code. Regarding the images, I have made progress in changing all required images to support RISC-V. I track the current status here. I have submitted PRs to several related projects, of which only CoreDNS has already accepted the changes and released a RISC-V image. You can find ready-made images under our Docker Hub account. I also maintain another branch in our K3S fork with updated manifests, until the changes are merged upstream. |
|
Yeah, the image thing will be interesting. Getting upstream to add support for it is part of it, but we'll also need to make some changes to https://github.com/rancher/image-mirror to add support for mirroring the risc-v architecture, without changing existing manifest lists. For the CI config, I'd like to discuss with the team perhaps making risc-v only run on nightly and tag, instead of PR, as it ties up the already over-utilized amd64 runners for a significant chunk of time. |
Proposed Changes
Add support for the RISC-V architecture
Types of Changes
Add the architecture to the scripts involved in building and packaging K3s; no code changes.
This is really possible through cross-compilation at the moment,
so I am also attaching a script that can be used from inside an Ubuntu Docker image to build K3s for RISC-V (tested on an M1 MacBook)so an extra script takes care of preparing the cross-compile environment (edit: updated to reflect latest changes).Verification
Starting from a clean checkout of the code, one can cross-compile for RISC-V
in an Ubuntu 22.04 container, by using the following command (edit: updated to reflect latest changes):This will produce the
dist/artifacts/k3s-riscv64binary, which can be installed in a QEMU VM for verification.I provide a pre-compiled binary and instructions here.
Sample output from a RISC-V QEMU VM for knowing what to expect:
Testing
This change is not covered by testing, but I am not sure how this would be possible given limited support of the required utilities for the architecture. This could be initially marked as an "experimental" architecture, until the tools are available for testing.
Linked Issues
This PR is directly related to #7151.
User-Facing Change
Further Comments
This should ideally be integrated with the(edit: The changes are now integrated with the K3s automated build system).Makefileand the automated build system, but I am not sure how these can support cross-compiling K3s - hence the build script. If someone can provide some hints on how to integrate, please do, as it would be really helpful to have an automated K3s release for RISC-V. I will be happy to help.Additionally, this PR is really the first step in the direction of RISC-V support. Utility images should also be made available for the architecture, starting with an image for a(edit: Required images have been ported to the architecture, changes have been submitted as PRs to relevant projects, read on for details.)pausecontainer. I have been unsuccessful in finding the sources and process of buildingrancher/mirrored-pause:3.6. Any pointers on that would also be very useful.