Multi-architecture plan for Kubernetes

[luxas]

Background: I've implemented most of the multi-architecture Kubernetes has today, and wrote the proposal here: https://github.com/kubernetes/community/tree/master/contributors/design-proposals/multi-platform.md

Now it's time to continue improving the multi-arch experience as well.
Tasks to do:

• Deprecate armel and use armhf images instead and use GOARM=7 instead of GOARM=6
  • Motivation:
    • The only GOARM=6 board Go will support in go1.8 is the Raspberry Pi 1 which is just too slow to run newer Kubernetes versions.
    • Small performance improvements when using GOARM=7
    • The armel (http://hub.docker.com/u/armel) images are not updated as often as the armhf (http://hub.docker.com/u/armhf) images are.
• Use go1.8 as fast as possible for arm and ppc64le (and of course generally as well, but that will require the "real" release)
  • Motivation:
    • Brings us a lot of mandatory fixes for arm and ppc64le
    • Brings us the SSA backend which is ~30% faster
    • We can remove the patched golang for arm and start building ppc64le binaries by default again
    • arm hyperkube will start working again
    • Even if it's beta, it's probably better than a self-patched version of go1.7
  • Proposal:
    • Use go1.8-beta1 for the arm builds already in the v1.5 release cc @saad-ali @pwittrock @jessfraz
• Reenable ppc64le builds again by using go1.8betas until the stable version of go1.8 is released and release v1.6 of kubernetes for ppc64le with go1.8
• Evalute s390x as a new platform
  • If no one loudly complains, I'm gonna take care of the PRs [Part 1] Add support for cross-compiling s390x binaries #37092 and [Part 2] Adding s390x cross-compilation support for gcr.io images in this repo #36050 and merge them in time for v1.6
• Convert the essential images that are named registry/binary:version to manifest lists
  • TODO: Investigate rkt support for manifest lists: Support Docker manifest lists appc/docker2aci#193
  • Wait for gcr.io to roll out a v2 schema 2 registry that support manifest lists. @aronchick told me it's gonna happen mid-December.
  • Start building manifest lists when releasing Kubernetes (kube-apiserver, kube-scheduler, etc.)
  • Basically, all images will be named registry/binary-arch:version as most of them are now, but then the image without the -arch bit will be a manifest list which points to the right -arch image depending on which arch docker runs on.
  • Convert all other essential images to manifest lists, namely:
    • etcd
    • all kube-dns images
    • the dashboard image
    • all heapster images: first step: Make this repo support multiple architectures kubernetes-retired/heapster#1387
• Convert the ingress images to multiple architectures

cc-ing involved people here:
@kubernetes/sig-testing @david-mcmahon @saad-ali @pwittrock @Pensu @ixdy @jessfraz @thockin @vishh @gajju26 @brendandburns

[thockin]

For the myriad images that we build that are 'from busybox' or 'from alpine', we need stable, maintained base images for each arch. There appear to be several for busybox, but NOT for alpine (also, don't forget the apkg packages we need)...

…

On Sun, Dec 4, 2016 at 8:20 AM, Lucas Käldström ***@***.***> wrote: Background: I've implemented most of the multi-architecture Kubernetes has today, and wrote the proposal here: https://github.com/kubernetes/ community/tree/master/contributors/design-proposals/multi-platform.md Now it's time to continue improving the multi-arch experience as well. Tasks to do: - Deprecate armel and use armhf images instead and use GOARM=7 instead of GOARM=6 - Motivation: - The only GOARM=6 board Go will support in go1.8 is the Raspberry Pi 1 which is just too slow to run newer Kubernetes versions. - Performance improvements when usiing GOARM=7 - The armel (http://hub.docker.com/u/armel) images are not updated as often as the armhf (http://hub.docker.com/u/armhf) images are. - Use go1.8 as fast as possible for arm and ppc64le (and of course generally as well, but that will require the "real" release) - Motivation: - Brings us a lot of mandatory fixes for arm and ppc64le - Brings us the SSA backend which is ~30% faster - We can remove the patched golang for arm and start building ppc64le binaries by default again - arm hyperkube will start working again - Even if it's beta, it's probably better than a self-patched version of go1.7 - Proposal: - Use go1.8-beta1 for the arm builds already in the v1.5 release cc @saad-ali <https://github.com/saad-ali> @pwittrock <https://github.com/pwittrock> @jessfraz <https://github.com/jessfraz> - Reenable ppc64le builds again by using go1.8betas until the stable version of go1.8 is released and release v1.6 of kubernetes for ppc64le with go1.8 - Evalute s390x as a new platform - If no one loudly complains, I'm gonna take care of the PRs #37092 <#37092> and #36050 <#36050> and merge them in time for v1.6 - Convert the essential images that are named registry/binary:version to manifest lists - TODO: Investigate rkt support for manifest lists: appc/docker2aci#193 <appc/docker2aci#193> - Wait for gcr.io to roll out a v2 schema 2 registry that support manifest lists. @aronchick <https://github.com/aronchick> told me it's gonna happen mid-December. - Start building manifest lists when releasing Kubernetes (kube-apiserver, kube-scheduler, etc.) - Basically, all images will be named registry/binary-arch:version as most of them are now, but then the image *without* the -arch bit will be a manifest list which points to the right -arch image depending on which arch docker runs on. - Convert all other essential images to manifest lists, namely: - etcd - all kube-dns images - the dashboard image - all heapster images: first step: kubernetes-retired/heapster#1387 <kubernetes-retired/heapster#1387> - Convert the ingress images to multiple architectures cc-ing involved people here: @kubernetes/sig-testing <https://github.com/orgs/kubernetes/teams/sig-testing> @david-mcmahon <https://github.com/david-mcmahon> @saad-ali <https://github.com/saad-ali> @pwittrock <https://github.com/pwittrock> @Pensu <https://github.com/pensu> @ixdy <https://github.com/ixdy> @jessfraz <https://github.com/jessfraz> @thockin <https://github.com/thockin> @vishh <https://github.com/vishh> @gajju26 <https://github.com/gajju26> @brendandburns <https://github.com/brendandburns> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#38067>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFVgVGTdwD5A1xAPzTWa7isNdjawuPVxks5rEug6gaJpZM4LDn3V> .

[xiaolou86]

mark, to learn later

[vielmetti]

@thockin - you might sync with @tianon who is doing Alpine builds for aarch64 - we sorted out a few issues and then alpine edge started to work on that platform. Cf. https://hub.docker.com/r/aarch64/alpine/

What I don't know is the time schedule for the next release of Alpine, whether we'll have Alpine 3.5 with aarch64 support before or after Kubernetes 1.6.

[luxas]

@vielmetti That's great!
To date we've only used busybox everywhere, and we have to do that until all architectures support alpine (s390x and ppc64le yet)

The thing that's more important though is getting the manifest lists working properly in core, and to make some manifest lists for official images available from Docker Hub. cc @tianon

[tianon]

docker-library/official-images#2289 is the relevant discussion for supporting manifest lists in official images

[luxas]

Now things are happening with this when go1.8rc1 was released:

Regarding s390x, @ixdy said this in #36050 (comment):

FYI I'd forgotten to push images for s390x, so looking into that now.
Pushed:

• gcr.io/google_containers/debian-iptables-s390x:v5
• gcr.io/google_containers/pause-s390x:3.0
• gcr.io/google-containers/kube-addon-manager-s390x:v6.1

Issues:

• kubedns: @bowei can you handle this, since it moved to a different repo? also it looks like dnsmasq (https://github.com/kubernetes/dns/blob/master/images/dnsmasq/Makefile) has no s390x support yet.
• cluster/images/etcd uses go1.6.3, so we can't currently build gcr.io/google_containers/etcd-s390x
• google_containers/hyperkube-s390x: I suspect this will automatically be released with 1.6. nothing there yet, though.
• cluster/images/kube-discovery has weird versioning - its tag is just "1.0", but it depends on whatever's been built in the tree, so I'm not sure what commit to push for s390x. @dgoodwin can maybe help here.
• gcr.io/google_containers/serve_hostname - it looks like nobody has pushed any non-amd64 images for this one yet.

I and @bowei will look into the DNS images soon I guess.
@ixdy will be able to push the etcd image when #38926 is merged.
hyperkube will be released with the first 1.6 alpha automatically
Yes, kube-discovery has a bit weird versioning, but it was a one-time add and haven't been updated since the initial add and push, so it's safe to just build and push it now @ixdy

We can look into the test images later.

I have #38926 ready for review. Highlights there:

• Removes the golang 1.7 patch
• Instead of the patched golang; go1.8rc1 is put in the GOROOT that arm and ppc64le now uses
• Reenables ppc64le because it's using go1.8rc1 which has all the changes it needs
• armel => armhf
• GOARM 6 => GOARM 7
• Bumps the QEMU version to v2.7.0

After this, I'm gonna start looking into making manifest lists for the official images so we can avoid -ARCH suffixes and instead just let docker pull the right layers for the right arch.

[estesp]

Just FYI--IBM is doing some work to hopefully get alpine supported on ppc64le and s390x (or more completely in cases where work has already been underway).

[luxas]

@estesp That's great! See the conversation about that here please: #40248 (comment)

[xnox]

Hello. about s390x support I see for example k8s-dns-kube-dns-ppc64le and flannel-ppc64le, but not s390x variants for those. Will gcr.io/google_containers/flannel-s390x and gcr.io/google_containers/k8s-dns-kube-dns-s390x be build and published soon? Anything outstanding to make that happen?

[cwsolee]

Both gcr.io/google_containers/flannel-s390x and gcr.io/google_containers/k8s-dns-kube-dns-s390x are work in progress. We just finish flannel porting and PR almost all done. Next is add enable build script to create image in gcr.io, might be a while though.

[luxas]

cc @bowei

The gcr.io/google_containers/k8s-dns-kube-dns-s390x image is just an extra command for the Googler that's releasing the image.

For flannel, you have to coordinate with them, the gcr.io/google_containers/flannel doesn't exist anymore because I and the flannel team ported it "upstream" to flannel.

Overall, there are k8s binaries for s390x since v1.6.0-alpha.1 and adding debs for it + kubeadm is in the progress as well.

[cwsolee]

IC, Yes, my team did the enablement for K8s binaries a while back and just got pick up since 1.6 alpha 1. We'll look at latest Flannel to see what we need to do. To provide gcr.io/google_containers/k8s-dns-kube-dns-s390x. do u know who we can talk to? Or we'll go thru our usual community route.

[bowei]

@cwsolee -- send a pull request enabling the architecture to the repository

[ethernetdan]

@luxas can this be closed or moved to v1.7?

[luxas]

This can be moved to v1.7, it couldn't be fully implemented in v1.6 due to that gcr.io doesn't support manifest lists yet

Moving milestone...

[vielmetti]

Thanks @luxas . Is there a spot where gcr.io support for manifest lists is being discussed or addresssed?

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[mkumatag]

/remove-lifecycle stale

[mschrupp]

@luxas or @mkumatag
just to make sure what is meant here with manifest lists -
will container images such as kube-proxy soon be multi-arch using manifest lists (for images without explicit architecture)?

Background: I currently have the problem that kubeadm generates a DaemonSet
with gcr.io/google_containers/kube-proxy-amd64:v1.11.0, which doesn't allow kubeadm join from arm devices (e.g. RPi).

It should rather use gcr.io/google_containers/kube-proxy:v1.11.0, but this image also only contains amd64 as of now, no manifest list

As workaround I'm putting a nodeSelector for beta.kubernetes.io/arch: amd64
in the kube-system/kube-proxy DaemonSet and duplicate it for
beta.kubernetes.io/arch: arm with gcr.io/google_containers/kube-proxy-arm:v1.11.0

[vielmetti]

@JesusOfSuburbia - that sounds like a problem that should be raised as an issue of its own!

[mkumatag]

Background: I currently have the problem that kubeadm generates a DaemonSet
with gcr.io/google_containers/kube-proxy-amd64:v1.11.0, which doesn't allow kubeadm join from arm devices (e.g. RPi).

It should rather use gcr.io/google_containers/kube-proxy:v1.11.0, but this image also only contains amd64 as of now, no manifest list

@JesusOfSuburbia do you have a mixed cluster? because in kubeadm we already have code to take care arch and additional code also added recently to restrict the target node #64696

[mschrupp]

@mkumatag oh wow, thanks for the link. yes, I have a mixed cluster! it looks like the restriction takes the same approach I'm doing manually at the moment. Still waiting for the manifest lists though, but at least it's documented for now. Thanks again, also for your work!

[dims]

Update as of today

• Fix manifest lists to always use correct size docker/cli#1156 has merged. This fixes a problem with manifest bytes. Then we need a docker release with that patch for subsequent tests.
• Add Makefile target to push fat manifest for multi-arch images #63453 has merged. we need folks with enough privileges to run the commands in Add Makefile target to push fat manifest for multi-arch images #63453 (comment) to upload images to k8s.gcr.io
• Add manifest support for release images release#516 is still waiting to be merged, that will mint images when we push a alpha or beta which we can then use to test.
• Replaced hardcoded busybox image in e2e tests #61097 will allow using a manifest based busybox image (manifest-tool inspect docker.io/busybox shows this has a bunch of arch'es though not the support for windows that 61097 mentions)

Guess eventual goal is to be able to run conformance tests on alternate architectures - vmware-tanzu/sonobuoy#181

@mkumatag anything else i missed?

[mkumatag]

@dims you have covered it really well.. :)

[timothysc]

/cc @liztio @chuckha - re: sonobuoy multi-arch support.

[dims]

as of v1.12.0-beta.2, all the kubernetes release container images are multi-arch capable. Also conformance tests have switched to multi-arch as well.

if there are any other images missing manifests, we should treat it as a bug and close this issue out.

/close

[k8s-ci-robot]

@dims: Closing this issue.

In response to this:

as of v1.12.0-beta.2, all the kubernetes release container images are multi-arch capable. Also conformance tests have switched to multi-arch as well.

if there are any other images missing manifests, we should treat it as a bug and close this issue out.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[mkumatag]

@dims This is the only pending PR I see - #59664 at the moment which is needed for etcd image.

[dims]

@mkumatag i managed to test the dns PR too, we need that to merge - kubernetes/dns#259