New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Proposal] Make Kubernetes run on ARM devices #17981

Closed
luxas opened this Issue Nov 30, 2015 · 20 comments

Comments

@luxas
Member

luxas commented Nov 30, 2015

Edited to reflect current status
Since Kubernetes is written in Go, it's possible to add support for ARM devices like Raspberry Pi.
I have done this, it's published here: https://github.com/luxas/kubernetes-on-arm
And it would be great to merge this to mainline k8s.

This work focuses on making it easier for other architectures too, like arm64 and ppc64le

TODO:

  • Upgrade to go1.5 go1.6: #22149
  • Compile ARM binaries on every CI build and for every release. #19769!
  • Compile ARM64 and ppc64le binaries on every CI build and build images for every release. #22149
  • Use the gcr.io/google_containers/binary-arch:version names for images. Discussions: #19061, #23059, #23009
  • Make etcd cross-platform: #19216
  • Add a gcr.io flannel image that uses quay.io/coreos/flannel:version for amd64 and cross-compile for other architectures. #21617
  • Merge the PR that pushes docker images (including hyperkube) when releasing. #19061
  • Edit the hyperkube Makefile to support ARM. #21617
  • Make skydns, kube2sky and exechealthz cross-platform. #19376, kubernetes/contrib#646
  • Add a k8s-in-docker deployment (like #13901) that supports multiple architectures kubernetes/kube-deploy#12
  • Update docker-multinode instructions: #18894
  • Update docker guide to support other architectures also. kubernetes/website#407
  • Write a multiarch proposal #26863
  • Build dashboard for ARM (gcr.io/google_containers/kubernetes-dashboard-arm)
  • Make pause totally cross-platform (and written in C). #23059, #23009
  • Be able to detect server platform #19905
  • Label nodes with os/arch #23684

Status of multiarch images:
Get docker-multinode working:

  • gcr.io/google_containers/hyperkube, fixed for all #21617, #22149
  • gcr.io/google_containers/etcd, fixed for all #19216, #22149
  • gcr.io/google_containers/flannel, fixed for all #21617, #22149
    • Depends on gcr.io/google_containers/debian-iptables, fixed for all #21617
  • gcr.io/google_containers/pause, fixed for all #23009

Get DNS working:

  • gcr.io/google_containers/kube2sky, fixed for all #19376
  • gcr.io/google_containers/skydns, fixed for all #19376
  • gcr.io/google_containers/exechealthz fixed for all kubernetes/contrib#646

Docker wrapped Kubernetes binaries:

  • gcr.io/google_containers/kube-proxy, fixed for all #21617, #22149
  • gcr.io/google_containers/kube-apiserver, fixed for all #19769, #22149
  • gcr.io/google_containers/kube-controller-manager, fixed for all #19769, #22149
  • gcr.io/google_containers/kube-scheduler, fixed for all #19769, #22149
  • gcr.io/google_containers/kubelet, doesn't really exist yet: #19177

I worked with the dashboard team, and it has had multiarch support from the first release.
I'm maybe gonna add multiarch support for official heapster too, now when it has reached v1.
On kubernetes-on-arm, I already have heapster, influxdb and grafana nicely running as a cluster addon.

Things to have in mind:

  • All binaries have to be compiled for ARM, and therefore a common naming structure is important
    • @brendandburns has already proposed this as binary-ARCH:VERSION, which is logical
    • Also, etcd and flannel should be built. etcd is already in cluster/images/etcd, maybe flannel could be included there also? Pushed to gcr.io/google_containers/flannel
  • One big question is how they should be compiled
  • Also all docker images must be compiled on ARM (except those FROM scratch)
  • Base docker images have to be chosen
  • One deployment method that would fit well for Raspberry Pis is k8s-in-docker
    • I'm using a k8s-in-docker-like setup combined with systemd services. I've also written a kube-config script that handles the whole process.
    • #13901
  • Bugs and patches
    • Sometimes, things fail (randomly) on ARM.
    • Example: #17060
    • Often, it's easy to patch. But there should be a specific place for ARM patches.
  • GOARM values
    • It's difference between armv6 and armv7 when compiling. We will have to set ENV GOARM=6 when we compile the images, so e.g. Raspberry Pi 1 is supported.
      Outdated tasks:
  • Use qemu emulation for hyperkube and flannel cross-building Makefiles. #21617

I thought about making a proposal earlier, and now when #17629 was merged, it might be time

/cc
@dalanlan (k8s-in-docker interest)
@resouer (k8s-in-docker interest)
@guybrush (k8s-in-docker interest)
@davidopp (who was involved in #17629)
@roberthbailey (who was involved in #17629)
@zmerlynn (who was mentioned in #17629)
@DieterReuter (interested in Raspberry Pi things)

@luxas

This comment has been minimized.

Show comment
Hide comment
@luxas

luxas Dec 5, 2015

Member

I've digged into this a little bit more, and here are the results:

  • hyperkube must be compiled dynamically, I tested it and it exited if it was built statically:
Error: failed to create kubelet: cAdvisor is unsupported in this build
failed to create kubelet: cAdvisor is unsupported in this build
  • armbuild/debian, which is the image currently used for ARM, doesn't work on ARMv6
    • the reason why armbuild fails is probably because the packages are built for ARMv7
    • the other option would be resin/rpi-raspbian which is a very good debian image
  • Now I may run any ARM docker image on my amd64 machine. This is very interesting.
    • I'm using moul's binfmt-register. However, it should only be able to run and compile binaries with go 1.5.2 or newer.
    • Edit: Even better method: https://resin.io/blog/building-arm-containers-on-any-x86-machine-even-dockerhub/
    • If we use this method, we may compile the binaries dynamically for ARM on a amd64 machine.
    • Some drawbacks though:
      • We can't use the normal kube-build:cross method as it is right now. It just cross-compiles statically. So just adding linux/arm to server platforms in hack/lib/golang.sh won't work. I'm working on this Fixed in #19769
      • Only the go base image must be built on a real ARM machine. That's because, using qemu we can't compile go1.4.3, which is required for go1.5.2. The chicken-or-egg problem.
    • So if we create a gcr.io/google_containers/golang-arm:v1.5.2 (maybe with some other prefix) and build that on a ARM machine, we could build all binaries on amd64
    • Edit: It's possible to cross-compile dynamically with golang:1.5.2 as base image.

So here are some questions for you to answer

  • Should I start working on a WIP: Kubernetes on ARM PR? Yes, but splitting the work in separate PRs
  • Is it acceptable to include flannel in cluster/images and maybe build them there? The same method as skydns... Yes
  • Is using qemu for building ARM docker images OK?
  • Should ARMv6 be supported? That would be great. You've already set GOARM=5 in kube-build:cross Yes
  • Could we switch armbuild/debian for resin/rpi-raspbian or some other ARMv6 compatible image? Yes

At least I would like to discuss these things
Just say if you don't want this functionality at this time in Kubernetes
Personally, I don't think it's that good to promote usage of k8s on Pis on your blog when you have to hack things for it to work (on the other hand, it's very fun 😄)

Member

luxas commented Dec 5, 2015

I've digged into this a little bit more, and here are the results:

  • hyperkube must be compiled dynamically, I tested it and it exited if it was built statically:
Error: failed to create kubelet: cAdvisor is unsupported in this build
failed to create kubelet: cAdvisor is unsupported in this build
  • armbuild/debian, which is the image currently used for ARM, doesn't work on ARMv6
    • the reason why armbuild fails is probably because the packages are built for ARMv7
    • the other option would be resin/rpi-raspbian which is a very good debian image
  • Now I may run any ARM docker image on my amd64 machine. This is very interesting.
    • I'm using moul's binfmt-register. However, it should only be able to run and compile binaries with go 1.5.2 or newer.
    • Edit: Even better method: https://resin.io/blog/building-arm-containers-on-any-x86-machine-even-dockerhub/
    • If we use this method, we may compile the binaries dynamically for ARM on a amd64 machine.
    • Some drawbacks though:
      • We can't use the normal kube-build:cross method as it is right now. It just cross-compiles statically. So just adding linux/arm to server platforms in hack/lib/golang.sh won't work. I'm working on this Fixed in #19769
      • Only the go base image must be built on a real ARM machine. That's because, using qemu we can't compile go1.4.3, which is required for go1.5.2. The chicken-or-egg problem.
    • So if we create a gcr.io/google_containers/golang-arm:v1.5.2 (maybe with some other prefix) and build that on a ARM machine, we could build all binaries on amd64
    • Edit: It's possible to cross-compile dynamically with golang:1.5.2 as base image.

So here are some questions for you to answer

  • Should I start working on a WIP: Kubernetes on ARM PR? Yes, but splitting the work in separate PRs
  • Is it acceptable to include flannel in cluster/images and maybe build them there? The same method as skydns... Yes
  • Is using qemu for building ARM docker images OK?
  • Should ARMv6 be supported? That would be great. You've already set GOARM=5 in kube-build:cross Yes
  • Could we switch armbuild/debian for resin/rpi-raspbian or some other ARMv6 compatible image? Yes

At least I would like to discuss these things
Just say if you don't want this functionality at this time in Kubernetes
Personally, I don't think it's that good to promote usage of k8s on Pis on your blog when you have to hack things for it to work (on the other hand, it's very fun 😄)

@ihmccreery

This comment has been minimized.

Show comment
Hide comment
@ihmccreery

ihmccreery Dec 11, 2015

Member

@brendandburns I know you were working on some Pi stuff; want to chime in here?

Member

ihmccreery commented Dec 11, 2015

@brendandburns I know you were working on some Pi stuff; want to chime in here?

@luxas

This comment has been minimized.

Show comment
Hide comment
@luxas

luxas Dec 12, 2015

Member

Yes #17629

I thought about making a proposal earlier, and now when #17629 was merged, it might be time

There are lots of folks who want this, and since 1st Oct, my project may help with this.
Or one have to hack it by hand...

@ihmccreery At least, should I start writing on the ARM docs?
E.g. about what's different with ARM and what needs to be customized.

Member

luxas commented Dec 12, 2015

Yes #17629

I thought about making a proposal earlier, and now when #17629 was merged, it might be time

There are lots of folks who want this, and since 1st Oct, my project may help with this.
Or one have to hack it by hand...

@ihmccreery At least, should I start writing on the ARM docs?
E.g. about what's different with ARM and what needs to be customized.

@ihmccreery

This comment has been minimized.

Show comment
Hide comment
@ihmccreery

ihmccreery Dec 16, 2015

Member

Assigning to @brendandburns to advise.

Member

ihmccreery commented Dec 16, 2015

Assigning to @brendandburns to advise.

@brendandburns

This comment has been minimized.

Show comment
Hide comment
@brendandburns

brendandburns Dec 17, 2015

Contributor

@luxas I have a working Raspberry PI cluster and have built a bunch of ARM binaries for various pieces. I'm happy to advise and review the PRs..

We can keep this open as a tracking issue. Please assign the PRs to me.

Contributor

brendandburns commented Dec 17, 2015

@luxas I have a working Raspberry PI cluster and have built a bunch of ARM binaries for various pieces. I'm happy to advise and review the PRs..

We can keep this open as a tracking issue. Please assign the PRs to me.

@luxas

This comment has been minimized.

Show comment
Hide comment
@luxas

luxas Dec 17, 2015

Member

@brendandburns Are you OK with using resin/rpi-raspbian instead for armbuild/debian for ARMv6 compability?
And how about including flannel besides etcd in cluster/images?

Member

luxas commented Dec 17, 2015

@brendandburns Are you OK with using resin/rpi-raspbian instead for armbuild/debian for ARMv6 compability?
And how about including flannel besides etcd in cluster/images?

@luxas

This comment has been minimized.

Show comment
Hide comment
@luxas

luxas Dec 26, 2015

Member

@brendandburns Great news! I've been able to compile all kubernetes binaries on my amd64 laptop.
This makes ARM building a lot easier and faster, and the ARM binaries works on both ARMv6 and ARMv7 (aka RPi 1 and 2)

The code is here and it uses golang:1.5.2 as the base image. I could test if it also works with golang:1.4-cross, which you are using today.

hyperkube and flannel is built dynamically for ARM on amd64 thanks to some libraries described here

I also think it would be possible to build ARM docker images on amd64 (only images with COPY works today) using qemu
Here is a blog post about that: https://resin.io/blog/building-arm-containers-on-any-x86-machine-even-dockerhub/

WDYT?

Member

luxas commented Dec 26, 2015

@brendandburns Great news! I've been able to compile all kubernetes binaries on my amd64 laptop.
This makes ARM building a lot easier and faster, and the ARM binaries works on both ARMv6 and ARMv7 (aka RPi 1 and 2)

The code is here and it uses golang:1.5.2 as the base image. I could test if it also works with golang:1.4-cross, which you are using today.

hyperkube and flannel is built dynamically for ARM on amd64 thanks to some libraries described here

I also think it would be possible to build ARM docker images on amd64 (only images with COPY works today) using qemu
Here is a blog post about that: https://resin.io/blog/building-arm-containers-on-any-x86-machine-even-dockerhub/

WDYT?

@ihmccreery

This comment has been minimized.

Show comment
Hide comment
@ihmccreery
Member

ihmccreery commented Dec 29, 2015

@luxas

This comment has been minimized.

Show comment
Hide comment
@luxas

luxas Apr 5, 2016

Member

Again, a major milestone was merged for cross-platform Kubernetes: #21617!
When the v1.3.0-alpha.2 release is out, one is able to run Kubernetes on ARM by following the docker guide.

Member

luxas commented Apr 5, 2016

Again, a major milestone was merged for cross-platform Kubernetes: #21617!
When the v1.3.0-alpha.2 release is out, one is able to run Kubernetes on ARM by following the docker guide.

k8s-merge-robot added a commit that referenced this issue Apr 11, 2016

Merge pull request #19376 from luxas/dns_cross_platform
Automatic merge from submit-queue

Make kube2sky and skydns docker images cross-platform

ARM tracking issue: #17981
Continues on: #19216

Make it possible to create `kube2sky` and `skydns` docker images for ARM and other architectures too
Build in a container, so `golang` isn't a dependency
I've preserved the original default behaviour:
 - `skydns`: It just compiles with go on host
 - `kube2sky`: Build an image

@brendandburns @dchen1107 @ArtfulCoder @thockin @fgrzadkowski

@luxas luxas assigned luxas and unassigned brendandburns Apr 21, 2016

@coryjamesfisher

This comment has been minimized.

Show comment
Hide comment
@coryjamesfisher

coryjamesfisher Apr 30, 2016

@luxas Thanks so much for the work you and the others have put into this. I was smacking my head against the wall trying to get Kubernetes running on a rpi 3 last night using the guide... Then I saw your comment this morning so I changed the k8 version number to the latest alpha and everything just works :) Thank you thank you thank you :)

coryjamesfisher commented Apr 30, 2016

@luxas Thanks so much for the work you and the others have put into this. I was smacking my head against the wall trying to get Kubernetes running on a rpi 3 last night using the guide... Then I saw your comment this morning so I changed the k8 version number to the latest alpha and everything just works :) Thank you thank you thank you :)

k8s-merge-robot added a commit that referenced this issue May 13, 2016

Merge pull request #23684 from luxas/auto_label_arch
Automatic merge from submit-queue

Automatically add node labels beta.kubernetes.io/{os,arch}

Proposal: #17981
As discussed in #22623:
> @davidopp: #9044 says cloud provider but can also cover platform stuff.

Adds a label `beta.kubernetes.io/platform` to `kubelet` that informs about the os/arch it's running on.
Makes it easy to specify `nodeSelectors` for different arches in multi-arch clusters.

```console
$ kubectl get no --show-labels
NAME        STATUS    AGE       LABELS
127.0.0.1   Ready     1m        beta.kubernetes.io/platform=linux-amd64,kubernetes.io/hostname=127.0.0.1
$ kubectl describe no
Name:			127.0.0.1
Labels:			beta.kubernetes.io/platform=linux-amd64,kubernetes.io/hostname=127.0.0.1
CreationTimestamp:	Thu, 31 Mar 2016 20:39:15 +0300
```
@davidopp @vishh @fgrzadkowski @thockin @wojtek-t @ixdy @bgrant0607 @dchen1107 @preillyme

k8s-merge-robot added a commit that referenced this issue May 28, 2016

Merge pull request #25972 from luxas/remove_arch_constants
Automatic merge from submit-queue

Use pause image depending on the server's platform when testing

Removed all pause image constant strings, now the pause image is chosen by arch. Part of the effort of making e2e arch-agnostic.

The pause image name and version is also now only in two places, and it's documented to bump both
Also removed "amd64" constants in the code. Such constants should be replaced by `runtime.GOARCH` or by looking up the server platform

Fixes: #22876 and #15140
Makes it easier for: #25730
Related: #17981

This is for `v1.3`
@ixdy @thockin @vishh @kubernetes/sig-testing @andyzheng0831 @Pensu
@luxas

This comment has been minimized.

Show comment
Hide comment
@luxas

luxas Jun 25, 2016

Member

@david-mcmahon How is the release note writing process for v1.3 managed?
It would be really great to mention a few words about the new non-business-supported architectures in the v1.3 release notes, ideally both on the blog and in the changelog.
WDYT? Please respond when you're seeing this and if you want me to write a short summary, just ping me

Member

luxas commented Jun 25, 2016

@david-mcmahon How is the release note writing process for v1.3 managed?
It would be really great to mention a few words about the new non-business-supported architectures in the v1.3 release notes, ideally both on the blog and in the changelog.
WDYT? Please respond when you're seeing this and if you want me to write a short summary, just ping me

@roberthbailey

This comment has been minimized.

Show comment
Hide comment
@roberthbailey

roberthbailey Jun 25, 2016

Member

Release notes are handled in a few chunks. There are the auto-generated notes from github labels that are bundled up since the last release. And there are hand-curated release notes, which this falls under. Please write up what you'd like to see and send it to @david-mcmahon

Member

roberthbailey commented Jun 25, 2016

Release notes are handled in a few chunks. There are the auto-generated notes from github labels that are bundled up since the last release. And there are hand-curated release notes, which this falls under. Please write up what you'd like to see and send it to @david-mcmahon

@luxas

This comment has been minimized.

Show comment
Hide comment
@luxas

luxas Jun 29, 2016

Member

@david-mcmahon Did you get my email message I sent some days ago?

Member

luxas commented Jun 29, 2016

@david-mcmahon Did you get my email message I sent some days ago?

@david-mcmahon

This comment has been minimized.

Show comment
Hide comment
@david-mcmahon

david-mcmahon Jun 29, 2016

Member

@luxas Yes and I replied.

Member

david-mcmahon commented Jun 29, 2016

@luxas Yes and I replied.

@luxas

This comment has been minimized.

Show comment
Hide comment
@luxas

luxas Jul 2, 2016

Member

Yes, thanks!
Looking forward to see the full release notes for v1.3 😄

Member

luxas commented Jul 2, 2016

Yes, thanks!
Looking forward to see the full release notes for v1.3 😄

@m1093782566

This comment has been minimized.

Show comment
Hide comment
@m1093782566
Member

m1093782566 commented Aug 20, 2016

/cc

@luxas

This comment has been minimized.

Show comment
Hide comment
@luxas

luxas Oct 16, 2016

Member

For everyone interested, arm binaries has been released for kubernetes since v1.2.

For v1.2.x, use my project kubernetes-on-arm v0.7.0
For v1.3.x, use docker-multinode or kubernetes-on-arm v0.8.0
For v1.4.x, use kubeadm

You may also read https://github.com/kubernetes/kubernetes/blob/master/docs/proposals/multi-platform.md

I consider this work done since a while ago thanks to the linked PRs above.
Still, there are work to do mentioned in the proposal, but the original issue Make Kubernetes on ARM devices has worked "officially" since v1.2.x, which is more than a half year ago, so closing this issue

Member

luxas commented Oct 16, 2016

For everyone interested, arm binaries has been released for kubernetes since v1.2.

For v1.2.x, use my project kubernetes-on-arm v0.7.0
For v1.3.x, use docker-multinode or kubernetes-on-arm v0.8.0
For v1.4.x, use kubeadm

You may also read https://github.com/kubernetes/kubernetes/blob/master/docs/proposals/multi-platform.md

I consider this work done since a while ago thanks to the linked PRs above.
Still, there are work to do mentioned in the proposal, but the original issue Make Kubernetes on ARM devices has worked "officially" since v1.2.x, which is more than a half year ago, so closing this issue

@luxas luxas closed this Oct 16, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment