Official ARM builds for every new release #19769

luxas · 2016-01-17T12:42:54Z

This PR features support for building arm(, arm64 and ppc64le) binaries dynamically or statically on amd64 hosts. It builds docker images for kube-apiserver, kube-controller-manager, kube-scheduler, kube-proxy for the new architectures and pushes them.
Bug fixed: hyperkube (legacy name for hyperkube-amd64) is pushed too when releasing for compability with older setups

Earlier:

arm
- Server support only when building on an ARM hosts
- Client support also on amd64 hosts
No arm64 or ppc64le support at all.

Now:

arm(, ppc64le, arm64 when we have go1.5)
- Server support when building on amd64 hosts
  - kubelet and hyperkube especially are built dynamically to include cAdvisor
- One patch already merged: update github.com/kr/pty #19712 for arm64 to build server targets
- Client support present
- Test target support fixed for arm64: Add linux_arm64 support onsi/ginkgo#209 and Update ginkgo and bolt Godeps so k8s builds on ARM64 #20546
- Test target support isn't so important for arm, arm64, ppc64le, so test targets are only built for linux/amd64, darwin/amd64 and windows/amd64.
windows/386 client support added

Tracking issue: #17981
Depends on: #14873
A bit related to: #19703

When this is merged, it has the advantage that builds will fail when trying to merge only-working-on-amd64 code for the server targets, so it's easily noticed.

This is WIP for now because #14873 hasn't merged yet (and I have to rebase after that)

~~However, if it takes too long time for arm64 and ginkgo to work, I'm gonna comment out arm64 for the moment and merge that later.~~

Thoughts? This will make Kubernetes more cross-platform, and that will be awesome 👍

@brendandburns @thockin @zmerlynn @fgrzadkowski @david-mcmahon @mikedanese @gmarek @wojtek-t @qq511700505

k8s-bot · 2016-01-17T12:43:10Z

Can one of the admins verify that this patch is reasonable to test? (reply "ok to test", or if you trust the user, reply "add to whitelist")

If this message is too spammy, please complain to ixdy.

k8s-bot · 2016-01-17T12:44:10Z

Can one of the admins verify that this patch is reasonable to test? (reply "ok to test", or if you trust the user, reply "add to whitelist")

If this message is too spammy, please complain to ixdy.

k8s-github-robot · 2016-01-17T12:44:54Z

Labelling this PR as size/M

zmerlynn · 2016-01-19T18:20:46Z

@k8s-bot ok to test

zmerlynn · 2016-01-19T18:21:31Z

build/build-image/Dockerfile

@@ -1,4 +1,4 @@
-# Copyright 2014 Google Inc. All rights reserved.
+# Copyright 2016 Google Inc. All rights reserved.


Please don't bump the copyright on the file unless it's been basically rewritten (it hasn't).

Aha, thought it could be updated... No problem

k8s-bot · 2016-01-19T18:46:23Z

GCE e2e build/test failed for commit e7f388d904953b155665fc19c5228ac385028a6d.

luxas · 2016-01-20T19:07:05Z

@zmerlynn Do you have any specific reason why you are building test targets for all client platforms? Isn't that a little bit unnecessary and slower?
I'm wondering if it could be removed from hack/build-cross.sh... (That code was modified last time 2014)

Of course test targets should be made for amd64, but for arm, windows and ppc64le for example it seems a little bit strange

I know that it might be another discussion though and that it might break backwards-compability (I don't know if you run integration tests on windows)
So I'm just asking...

ixdy · 2016-01-20T23:13:24Z

FYI also @ihmccreery @david-mcmahon

zmerlynn · 2016-01-20T23:18:16Z

@luxas: Yes, we could probably take the test code out of the client build.

zmerlynn · 2016-01-20T23:33:08Z

hack/lib/golang.sh

  darwin/amd64
  darwin/386
  windows/amd64
+  windows/386


Did you add 32-bit windows for a reason?

Because I saw that a compiler in the earlier kube-build:cross was built for windows/386, but it wasn't listed as a client platform. Supposed it just had been forgotten.

But it doesn't harm to build kubectl for 32-bit win also.

zmerlynn · 2016-01-20T23:37:10Z

This looks okay. I'm a little concerned about bumping golang given what lead up to ~~19487~~ #17524, so I'm not sure how to adjudicate that.

ixdy · 2016-01-20T23:46:45Z

#17524 I think you mean?

In any case, it'd probably be good to bump the golang version separate from other changes. #14873 is for that, I think.

zmerlynn · 2016-01-20T23:58:10Z

Yes, I have no idea where I pulled the other number from.

luxas · 2016-01-21T05:18:53Z

golang should be updated in another PR, probably #14873.
I just have to include it now in this WIP stage so it builds.
Will rebase when #14873 gets in

Will rebase and rework the PR...

luxas · 2016-01-21T15:02:40Z

build/common.sh

@@ -97,6 +97,7 @@ readonly GCS_STAGE="${LOCAL_OUTPUT_ROOT}/gcs-stage"
 # The set of master binaries that run in Docker (on Linux)
 # Entry format is "<name-of-binary>,<base-image>".
 # Binaries are placed in /usr/local/bin inside the image.
+# TODO: Make base images for multiple server platforms


@zmerlynn Do you have some suggestions how to nicely handle this?
We need to be able to specify different base images for different arches

k8s-bot · 2016-01-21T15:37:18Z

GCE e2e build/test failed for commit 2360e14e8225c02c1be9363c21a43ec5edbfc334.

luxas · 2016-01-21T17:10:31Z

Looks like a e2e flake.
It built binaries and packaged them as it should.

But some test didn't start in time or something...

k8s-bot · 2016-02-07T16:30:13Z

GCE e2e build/test failed for commit d58844e952f8c6adc35cc7472c1b26b90fca398d.

luxas · 2016-02-07T16:53:07Z

To rebase to go1.4 was an attempt to do this anyway, and see if it works, but it didn't.
linux/arm64 and linux/ppc64le was introduced in go1.5

I'll try to get working this only for linux/arm, since that's the primary goal for this patch
We'll continue with the others post-v1.2 when we have go1.5.3 (or go1.6?) from the beginning

…gets

k8s-bot · 2016-02-07T19:15:39Z

GCE e2e test build/test passed for commit c969c04.

zmerlynn · 2016-02-09T18:03:16Z

LGTM

k8s-github-robot · 2016-02-09T19:02:31Z

@k8s-bot test this

Tests are more than 48 hours old. Re-running tests.

k8s-bot · 2016-02-09T19:46:41Z

GCE e2e test build/test passed for commit c969c04.

k8s-github-robot · 2016-02-09T19:56:19Z

@k8s-bot test this [submit-queue is verifying that this PR is safe to merge]

k8s-bot · 2016-02-09T20:37:13Z

GCE e2e test build/test passed for commit c969c04.

k8s-github-robot · 2016-02-09T20:39:33Z

@k8s-bot test this [submit-queue is verifying that this PR is safe to merge]

k8s-bot · 2016-02-09T21:08:51Z

GCE e2e test build/test passed for commit c969c04.

k8s-github-robot · 2016-02-09T21:09:12Z

Automatic merge from submit-queue

Auto commit by PR queue bot

luxas · 2016-02-09T21:10:40Z

Thanks @zmerlynn!
BTW, could we add a release note for this? e.g.

Official ARM builds for every release (#19769, @luxas)

I think it would be great to announce that we now support ARM :)
(If I remember correctly there's a label for release notes)

david-mcmahon · 2016-02-11T00:55:58Z

@luxas To add this as a release note, add the release-note label and update the PR description to whatever you want the release note to say.

mikedanese · 2016-02-11T01:32:39Z

@david-mcmahon adding a label requires repo write access which not everyone has

david-mcmahon · 2016-02-11T01:37:53Z

@mikedanese thanks for the clarification. Adding release-note label. @luxas please adjust release note to taste.

googlebot added the cla: yes label Jan 17, 2016

k8s-github-robot assigned ixdy Jan 17, 2016

k8s-github-robot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Jan 17, 2016

luxas mentioned this pull request Jan 17, 2016

[Proposal] Make Kubernetes run on ARM devices #17981

Closed

17 tasks

zmerlynn reviewed Jan 19, 2016
View reviewed changes

ixdy assigned zmerlynn and unassigned ixdy Jan 20, 2016

k8s-github-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jan 20, 2016

zmerlynn reviewed Jan 20, 2016
View reviewed changes

zmerlynn mentioned this pull request Jan 21, 2016

Up to golang 1.5 #14873

Closed

luxas force-pushed the server_cross_compilation branch from e7f388d to 2360e14 Compare January 21, 2016 14:55

k8s-github-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jan 21, 2016

luxas reviewed Jan 21, 2016
View reviewed changes

k8s-github-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jan 21, 2016

Add support for building arm, arm64 and ppc64le server and client tar…

c969c04

…gets

luxas force-pushed the server_cross_compilation branch from d58844e to c969c04 Compare February 7, 2016 18:36

luxas changed the title ~~Add support for building arm, arm64 and ppc64le server and client targets~~ Add support for building arm server and client targets in the release process. Prepare for arm64 and ppc64le Feb 7, 2016

luxas mentioned this pull request Feb 9, 2016

push kube-cross to gcr.io #20882

Merged

zmerlynn added the lgtm Indicates that a PR is ready to be merged. label Feb 9, 2016

k8s-github-robot added a commit that referenced this pull request Feb 9, 2016

Merge pull request #19769 from luxas/server_cross_compilation

f36b604

Auto commit by PR queue bot

k8s-github-robot merged commit f36b604 into kubernetes:master Feb 9, 2016

david-mcmahon added the release-note Denotes a PR that will be considered when it comes time to generate release notes. label Feb 11, 2016

luxas changed the title ~~Add support for building arm server and client targets in the release process. Prepare for arm64 and ppc64le~~ Official ARM builds for every new release Feb 12, 2016

luxas mentioned this pull request Feb 16, 2016

Add test targets for ARM #21294

Merged

This was referenced Mar 4, 2016

Include Go version, platform, and other build info in version string #19905

Merged

Upgrading to go 1.6 #22039

Closed

luxas mentioned this pull request Mar 27, 2016

Make kubelet use an arch-specific pause image depending on GOARCH #23059

Merged

luxas mentioned this pull request Apr 6, 2016

Build Kubernetes, etcd and flannel for arm64 and ppc64le #23931

Merged

luxas mentioned this pull request May 15, 2016

gcr.io/google_containers/hyperkube-arm:v1.2.3 missing #24785

Closed

Official ARM builds for every new release #19769

Official ARM builds for every new release #19769

luxas commented Jan 17, 2016

k8s-bot commented Jan 17, 2016

k8s-bot commented Jan 17, 2016

k8s-github-robot commented Jan 17, 2016

zmerlynn commented Jan 19, 2016

zmerlynn Jan 19, 2016

luxas Jan 19, 2016

k8s-bot commented Jan 19, 2016

luxas commented Jan 20, 2016

ixdy commented Jan 20, 2016

zmerlynn commented Jan 20, 2016

zmerlynn Jan 20, 2016

luxas Jan 21, 2016

zmerlynn commented Jan 20, 2016

ixdy commented Jan 20, 2016

zmerlynn commented Jan 20, 2016

luxas commented Jan 21, 2016

luxas Jan 21, 2016

k8s-bot commented Jan 21, 2016

luxas commented Jan 21, 2016

k8s-bot commented Feb 7, 2016

luxas commented Feb 7, 2016

k8s-bot commented Feb 7, 2016

zmerlynn commented Feb 9, 2016

k8s-github-robot commented Feb 9, 2016

k8s-bot commented Feb 9, 2016

k8s-github-robot commented Feb 9, 2016

k8s-bot commented Feb 9, 2016

k8s-github-robot commented Feb 9, 2016

k8s-bot commented Feb 9, 2016

k8s-github-robot commented Feb 9, 2016

luxas commented Feb 9, 2016

david-mcmahon commented Feb 11, 2016

mikedanese commented Feb 11, 2016

david-mcmahon commented Feb 11, 2016

		@@ -1,4 +1,4 @@
		# Copyright 2014 Google Inc. All rights reserved.
		# Copyright 2016 Google Inc. All rights reserved.

Official ARM builds for every new release #19769

Official ARM builds for every new release #19769

Conversation

luxas commented Jan 17, 2016

k8s-bot commented Jan 17, 2016

k8s-bot commented Jan 17, 2016

k8s-github-robot commented Jan 17, 2016

zmerlynn commented Jan 19, 2016

zmerlynn Jan 19, 2016

Choose a reason for hiding this comment

luxas Jan 19, 2016

Choose a reason for hiding this comment

k8s-bot commented Jan 19, 2016

luxas commented Jan 20, 2016

ixdy commented Jan 20, 2016

zmerlynn commented Jan 20, 2016

zmerlynn Jan 20, 2016

Choose a reason for hiding this comment

luxas Jan 21, 2016

Choose a reason for hiding this comment

zmerlynn commented Jan 20, 2016

ixdy commented Jan 20, 2016

zmerlynn commented Jan 20, 2016

luxas commented Jan 21, 2016

luxas Jan 21, 2016

Choose a reason for hiding this comment

k8s-bot commented Jan 21, 2016

luxas commented Jan 21, 2016

k8s-bot commented Feb 7, 2016

luxas commented Feb 7, 2016

k8s-bot commented Feb 7, 2016

zmerlynn commented Feb 9, 2016

k8s-github-robot commented Feb 9, 2016

k8s-bot commented Feb 9, 2016

k8s-github-robot commented Feb 9, 2016

k8s-bot commented Feb 9, 2016

k8s-github-robot commented Feb 9, 2016

k8s-bot commented Feb 9, 2016

k8s-github-robot commented Feb 9, 2016

luxas commented Feb 9, 2016

david-mcmahon commented Feb 11, 2016

mikedanese commented Feb 11, 2016

david-mcmahon commented Feb 11, 2016