New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Kubeadm broken on armv6l #253

Closed
mitchhh22 opened this Issue Apr 23, 2017 · 15 comments

Comments

Projects
None yet
10 participants
@mitchhh22
Copy link

mitchhh22 commented Apr 23, 2017

I tried to install kubeadm on my raspberry pi zero W, but get an "Illegal Instruction"
On a raspberry pi 3 (armv7) it works just fine.

@clabu

This comment has been minimized.

Copy link

clabu commented Apr 23, 2017

I am facing the same issue with kubeadm 1.6.1 on a Raspberry Pi Model B+, also armv6.

$ kubelet --help
Illegal instruction

$ uname -a
Linux pi1 4.4.50-hypriotos+ #2 PREEMPT Sun Mar 19 14:44:01 UTC 2017 armv6l GNU/Linux
@clabu

This comment has been minimized.

Copy link

clabu commented Apr 23, 2017

I downgraded to kubeadm 1.5.6 and it works. 1.6.0 gives the same error as 1.6.1.

@mitchhh22

This comment has been minimized.

Copy link

mitchhh22 commented Apr 24, 2017

@clabu yeah downgrading to 1.5.6 works but can't join a 1.6+ cluster.

@luxas

This comment has been minimized.

Copy link
Member

luxas commented Apr 24, 2017

First off, thanks for using Kubernetes on ARM 😄!

This is a known issue; it was discussed in kubernetes/kubernetes#38067 and we dropped armel support (which part of RPi 1 uses when cross-compiling).

Basically armhf (GOARM=7) can't run on the Pi 1, so we used armel with GOARM=6 in -v1.5 to support RPi 1. However, we went all armhf in v1.6, hence it's not working on the Pi 1.

Deprecate armel and use armhf images instead and use GOARM=7 instead of GOARM=6
Motivation:

  • The only GOARM=6 board Go will support in go1.8 is the Raspberry Pi 1 which is just too slow to run newer Kubernetes versions.
  • Small performance improvements when using GOARM=7
  • The armel (http://hub.docker.com/u/armel) images are not updated as often as the armhf (http://hub.docker.com/u/armhf) images are.

For example, https://hub.docker.com/r/armel/debian/ was updated 8 months ago which is really bad from a security standpoint, vs https://hub.docker.com/r/armhf/debian/ which was updated 3 days ago.

Also, with the armhf switch, we were able to use https://hub.docker.com/r/armhf/alpine, which is great.

Hope it helps, but sorry for not being able to support the RPi 1 anymore.

If you want to help with documenting it/spreading the word, please do or come with suggestions

@wsargent

This comment has been minimized.

Copy link

wsargent commented Feb 4, 2018

I'm having this same problem on a Pi Zero

Linux p1 4.9.59+ #1047 Sun Oct 29 11:47:10 GMT 2017 armv6l GNU/Linux
@mthkunze

This comment has been minimized.

Copy link

mthkunze commented Feb 12, 2018

Is it possible to discuses a reintegration of armv6l Support. I found many Posts showing the interest for using Kubernetes on Pi Zero and other armv6l Pi Devices. P Zero is good for hosting Micro Services in Kubernetes or Swarm Cluster Environments. Docker Swarm works well for me. So it would be nice if anyone could recycle the discussion. Pi clusterhat is properly a nice demo infrastructure.

@kfox1111

This comment has been minimized.

Copy link

kfox1111 commented Feb 19, 2018

Looking at the current docker.io build for the pi zero,
Go version: go1.9.3
and docker version: 18.02.0-ce

It does seem to be using a recent version of go.

I agree that there is not enough ram to use k8s on it in a standalone mode, but having it be a slave on a bigger master, it should have enough resources to do some useful things still.

@juliancheal

This comment has been minimized.

Copy link

juliancheal commented Mar 6, 2018

Does anyone know if it's possible just to build from source to use my pi zeros as k8s nodes?

@wxdao

This comment has been minimized.

Copy link

wxdao commented May 7, 2018

For example, https://hub.docker.com/r/armel/debian/ was updated 8 months ago which is really bad from a security standpoint, vs https://hub.docker.com/r/armhf/debian/ which was updated 3 days ago.

This is not true today since official images on different architectures are updated simultaneously. For example https://hub.docker.com/r/arm32v5/debian/, https://hub.docker.com/r/arm32v7/debian/ and https://hub.docker.com/r/amd64/debian/ were all updated 9 days ago.

Also, with the armhf switch, we were able to use https://hub.docker.com/r/armhf/alpine, which is great.

https://hub.docker.com/r/arm32v6/alpine/ runs well on Pi Zero.

I hope that you will reconsider it. Stopping Pi Zero from running latest k8s is so disappointing.

@luxas

@kfox1111

This comment has been minimized.

Copy link

kfox1111 commented May 7, 2018

+1. some confusion has happened as the hub was rearranged and the older repos are still around. The newer ones do seem to be getting frequent updates.

@shinichi-hashitani

This comment has been minimized.

Copy link

shinichi-hashitani commented May 20, 2018

Hi @juliancheal ,

I am still in the middle of building k8s on ClusterHAT, but I was able to compile and build binaries for Pi Zero.

Basically, I have followed the below with some modifications:
https://povilasv.me/raspberrypi-kubelet/

I worked on wsl:
Linux DESKTOP-6GRDDIN 4.4.0-17134-Microsoft #48-Microsoft Fri Apr 27 18:06:00 PST 2018 x86_64 x86_64 x86_64 GNU/Linux

#1 install gcc-arm-linux-gnueabi instead of gcc-arm-linux-gnueabihf
sudo apt-get install gcc-arm-linux-gnueabi <- change

#2 before building for linux/arm, make two modifications to set_platform_envs() in hack/lib/golang.sh
*** add GOARM
export GOOS=${platform%/}
export GOARCH=${platform##
/}
export GOARM=5 <- add
*** change CC
case "${platform}" in
"linux/arm")
export CGO_ENABLED=1
export CC=arm-linux-gnueabi-gcc <-change
;;

GOARM has to be 5. If you specify 6, you will get a linker error during the build. (Which I couldn't resolve.)

@wxdao

This comment has been minimized.

Copy link

wxdao commented May 20, 2018

@shinichi-hashitani It works for my Pi Zero! Thanks!

Also I resolved your problem on linker error. For Pi Zero, set GOARM=6 and keep gcc-arm-linux-gnueabihf. However for Pi 1 you gotta set GOARM=5 and use gcc-arm-linux-gnueabi instead.

@juliancheal

This comment has been minimized.

Copy link

juliancheal commented May 20, 2018

@shinichi-hashitani this is great! I will give it a try thanks!

@dbwest

This comment has been minimized.

Copy link

dbwest commented Nov 21, 2018

@shinichi-hashitani Did you use make all KUBE_BUILD_PLATFORMS=linux/arm to build it? And if you used kubeadm to set up your cluster, how did you do that? Did you copy over kubelet, the init script povilasv mentioned, kubeadm, and kubectl? Did it work?

@shinichi-hashitani

This comment has been minimized.

Copy link

shinichi-hashitani commented Dec 28, 2018

@dbwest Yes, I used make all to build binaries. The exact commands I used were:

make all WHAT=cmd/kube-proxy KUBE_VERBOSE=5 KUBE_BUILD_PLATFORMS=linux/arm
make all WHAT=cmd/kubelet KUBE_VERBOSE=5 KUBE_BUILD_PLATFORMS=linux/arm
make all WHAT=cmd/kubectl KUBE_VERBOSE=5 KUBE_BUILD_PLATFORMS=linux/arm

I needed binaries for nodes, so only those three binaries were needed.

I didn't use kubeadm. I was following Kelsey Hightower's "Kubernetes the Hard Way". As described here, you just need to put those binaries in appropriate location.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment