k3s server failed to start in an alpine docker container on a Mac with M1 CPU

[updogliu]

Environmental Info:

Host OS: Mac OS 12.0.1 Monterey

Docker version on Host:

Client:
 Cloud integration: v1.0.22
 Version:           20.10.11
 API version:       1.41
 Go version:        go1.16.10
 Git commit:        dea9396
 Built:             Thu Nov 18 00:36:09 2021
 OS/Arch:           darwin/arm64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.11
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.16.9
  Git commit:       847da18
  Built:            Thu Nov 18 00:34:44 2021
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.4.12
  GitCommit:        7b11cfaabd73bb80907dd23182b9347b4245eb5d
 runc:
  Version:          1.0.2
  GitCommit:        v1.0.2-0-g52b36a2
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

K3s Version:

k3s version 1.21.1+k3s1 (AlpineLinux)
go version go1.16.10

Node(s) CPU architecture, OS, and Version:
In an alpine container

Docker image: Alpine 3.14.2, "Architecture": "amd64",
uname -a (in the alpine container)
Linux 692236d45d59 5.10.76-linuxkit #1 SMP Mon Nov 8 10:21:19 UTC 2021 x86_64 GNU/Linux

Cluster Configuration:
1 server

Describe the bug:
k3s start reports error

E1210 23:15:23.627061       1 server.go:288] "Failed to run kubelet" err="failed to run Kubelet: could not detect clock speed from output: \"processor\\t: 0\\nBogoMIPS\\t: 48.00\\nFeatures\\t: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 asimddp sha512 asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp flagm2 frint\\nCPU implementer\\t: 0x00\\nCPU architecture: 8\\nCPU variant\\t: 0x0\\nCPU part\\t: 0x000\\nCPU revision\\t: 0\\n\\nprocessor\\t: 1\\nBogoMIPS\\t: 48.00\\nFeatures\\t: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 asimddp sha512 asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp flagm2 frint\\nCPU implementer\\t: 0x00\\nCPU architecture: 8\\nCPU variant\\t: 0x0\\nCPU part\\t: 0x000\\nCPU revision\\t: 0\\n\\nprocessor\\t: 2\\nBogoMIPS\\t: 48.00\\nFeatures\\t: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 asimddp sha512 asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp flagm2 frint\\nCPU implementer\\t: 0x00\\nCPU architecture: 8\\nCPU variant\\t: 0x0\\nCPU part\\t: 0x000\\nCPU revision\\t: 0\\n\\nprocessor\\t: 3\\nBogoMIPS\\t: 48.00\\nFeatures\\t: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 asimddp sha512 asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp flagm2 frint\\nCPU implementer\\t: 0x00\\nCPU architecture: 8\\nCPU variant\\t: 0x0\\nCPU part\\t: 0x000\\nCPU revision\\t: 0\\n\\n\""

Steps To Reproduce:

• Installed K3s:
  apk add -U k3s

Expected behavior:
k3s server can start the server without errors.

Actual behavior:
Exited with an error

E1210 23:15:23.627061       1 server.go:288] "Failed to run kubelet" err="failed to run Kubelet: could not detect clock speed from output: ... (see above)

[brandond]

could not detect clock speed from output: ... (see above)

Unfortunately you've not included whatever's above. Can you do so?

Ultimately this is somewhere in upstream or a 3rd party library though, so we'll need to figure out where and likely open an issue there.

[updogliu]

The full error output was in "Describe the bug" (the extreme long line).

I pasted it at here again:

E1210 23:15:23.627061 1 server.go:288] "Failed to run kubelet" err="failed to run Kubelet: could not detect clock speed from output: "processor\t: 0\nBogoMIPS\t: 48.00\nFeatures\t: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 asimddp sha512 asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp flagm2 frint\nCPU implementer\t: 0x00\nCPU architecture: 8\nCPU variant\t: 0x0\nCPU part\t: 0x000\nCPU revision\t: 0\n\nprocessor\t: 1\nBogoMIPS\t: 48.00\nFeatures\t: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 asimddp sha512 asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp flagm2 frint\nCPU implementer\t: 0x00\nCPU architecture: 8\nCPU variant\t: 0x0\nCPU part\t: 0x000\nCPU revision\t: 0\n\nprocessor\t: 2\nBogoMIPS\t: 48.00\nFeatures\t: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 asimddp sha512 asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp flagm2 frint\nCPU implementer\t: 0x00\nCPU architecture: 8\nCPU variant\t: 0x0\nCPU part\t: 0x000\nCPU revision\t: 0\n\nprocessor\t: 3\nBogoMIPS\t: 48.00\nFeatures\t: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 asimddp sha512 asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp flagm2 frint\nCPU implementer\t: 0x00\nCPU architecture: 8\nCPU variant\t: 0x0\nCPU part\t: 0x000\nCPU revision\t: 0\n\n""

[dweomer]

Steps To Reproduce:

• Installed K3s:
  apk add -U k3s

You've installed k3s from the Alpine community channel which is not maintained by the k3s team. This bug report would be more useful over at https://gitlab.alpinelinux.org/alpine/aports.

Please be aware that there are caveats when using this alternate packaging of k3s, specifically the crictl and ctr sub-commands are broken out of the "all-in-one" (multi-call) k3s binary. Additionally, while kubectl is a symlink to k3s k3s kubectl does not work as expected. And finally, because the Alpine community packaging of k3s depends on the containerd package and does not ship the k3s built-in containerd, you will not be able to configure containerd/cri registries via /etc/rancher/k3s/registries.yaml.

---

I have yet to try k3s with docker desktop on my M1 minis but I would be surprised if it did not work. As such, I encourage you to give the docker.io/rancher/k3s image a try. These are the images that the rancher/k3d project leverages by default.

[dweomer]

@updogliu I went ahead and created https://gitlab.alpinelinux.org/alpine/aports/-/issues/13300

[brandond]

In addition to what Jacob noted above, the cpuinfo in your Docker container does appear to be bogus; other than the features all the fields are just 0.

[dweomer]

So, as per https://gitlab.alpinelinux.org/alpine/aports/-/issues/13300#note_197830 ...

I guess my brain refused to see this:

Node(s) CPU architecture, OS, and Version: In an alpine container

Docker image: Alpine 3.14.2, "Architecture": "amd64",
uname -a (in the alpine container)
Linux 692236d45d59 5.10.76-linuxkit #1 SMP Mon Nov 8 10:21:19 UTC 2021 x86_64 GNU/Linux

An amd64 virtual machine on arm64 hardware? Sure, why not.

Does https://docs.docker.com/desktop/mac/apple-silicon/ not run with an arm64 VM by default?

[brandond]

• cpuinfo poorly emulated on M1 Macs docker/for-mac#6047
• https://gitlab.com/qemu-project/qemu/-/issues/750

[updogliu]

Does https://docs.docker.com/desktop/mac/apple-silicon/ not run with an arm64 VM by default?

For some reason, our image (base on Alpine 3.14.2) only has an amd64 version.

Docker Desktop support running amd64 images on Apple silicon, and from 4.3.0 it "requires no Docker Desktop no longer requires Rosetta 2" per the release note.

[brandond]

Can you try running an arm64 image instead? Maybe just our image at rancher/k3s? It sounds like the qemu-based amd64 emulation is incomplete in a number of ways that are incompatible with running Kubernetes.

[updogliu]

Can you try running an arm64 image instead? Maybe just our image at rancher/k3s? It sounds like the qemu-based amd64 emulation is incomplete in a number of ways that are incompatible with running Kubernetes.

Thanks for the suggestions, @brandond. FYI, an arm64 alpine image installed k3s via apk add k3s works fine on a Mac M1 machine.

However, that is a non-option to us, because we need to run k8s pods using some amd64-only images (a typical one is MySQL-5.7). And we did some customizations base on a vanilla alpine OS (apk installs, dns resolvers, etc.), which are not straightforward to migrate to a heavily pruned image like rancher/k3s. That said, I can give a try if I can see the source Dockerfile building the rancher/k3s image. Where can I find it?

[brandond]

That said, I can give a try if I can see the source Dockerfile building the rancher/k3s image. Where can I find it?

Right here in this repo. You'll need to follow the building instructions; the image is produced at

k3s/scripts/package-image

Line 17 in 2f3bfc2

docker build ${PROXY_OPTS} --build-arg VERSION=${VERSION} -t ${IMAGE} -f package/Dockerfile .

Given the problems that the docker for mac apparently has in emulating amd64, I would probably not expect to get amd64 working. Even if you rebuild it yourself that won't change anything about the underlying issue with qemu's broken /proc/cpuinfo not meeting the needs of the kubelet.

[updogliu]

Thanks, @brandond. Is it possible / make sense to implement the handling of /proc/cpuinfo of k3s in a lenient way that accepts both amd64 and arm64 styles, no matter what architecture of the container OS is (because it can be wrong due to qemu's issue)?

[brandond]

No, because the code in question is not in K3s. It is the upstream Kubernetes kubelet code that is erroring out.

[updogliu]

I see. So basically we are stuck with the qemu issue of wrong /proc/cpuinfo.

[brandond]

As long as you are running emulated amd64, yes.

[stale]

This repository uses a bot to automatically label issues which have not had any activity (commit/comment/label) for 180 days. This helps us manage the community issues better. If the issue is still relevant, please add a comment to the issue so the bot can remove the label and we know it is still valid. If it is no longer relevant (or possibly fixed in the latest release), the bot will automatically close the issue in 14 days. Thank you for your contributions.

[nouzun]

Did you guys find a workaround for running amd64 images on Apple silicon? It also fails with rosetta not only qemu.

[corneliusroemer]

There's some hope that maybe now that Rosetta emulation is used instead of QEMU this will be fixed, but for now it sadly doesn't look like it, see:

• Fixing /proc/cpuinfo on Rosetta docker/for-mac#7080

If anyone has a workaround, please share. Even if it involves setting up virtualbox or something like that.

[corneliusroemer]

One more thought based on these comments above:

Thanks, @brandond. Is it possible / make sense to implement the handling of /proc/cpuinfo of k3s in a lenient way that accepts both amd64 and arm64 styles, no matter what architecture of the container OS is (because it can be wrong due to qemu's issue)?

No, because the code in question is not in K3s. It is the upstream Kubernetes kubelet code that is erroring out.

Could an issue be made in https://github.com/kubernetes/kubernetes or a similar repo to not require clock speed?