Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support arm64 #166

Open
vielmetti opened this issue Dec 11, 2018 · 14 comments

Comments

Projects
7 participants
@vielmetti
Copy link

commented Dec 11, 2018

Device under test is a Packet c1.large.arm 96-core arm64 machine running Ubuntu 18.04.

ed@ed-2a-bcc-llvm:~$ go version
go version go1.11.2 linux/arm64
ed@ed-2a-bcc-llvm:~$ go get sigs.k8s.io/kind
ed@ed-2a-bcc-llvm:~$ go/bin/kind create cluster
Creating cluster 'kind-1' ...
 ✓ Ensuring node image (kindest/node:v1.12.2)  
 ✓ [kind-1-control-plane] Creating node container 📦 
 ✗ [kind-1-control-plane] Fixing mounts 🗻 
Error: failed to create cluster: exit status 1
Usage:  
  kind create cluster [flags]
        
Flags:  
      --config string   path to a kind config file
  -h, --help            help for cluster
      --image string    node docker image to use for booting the cluster
      --name string     cluster context name (default "1")
      --retain          retain nodes for debugging when cluster creation fails
      --wait duration   Wait for control plane node to be ready (default 0s)

Global Flags:
      --loglevel string   logrus log level [panic, fatal, error, warning, info, debug] (default "warning")

failed to create cluster: exit status 1
ed@ed-2a-bcc-llvm:~$ docker version
Client:
 Version:           18.09.0
 API version:       1.39
 Go version:        go1.10.4
 Git commit:        4d60db4
 Built:             Wed Nov  7 00:52:41 2018
 OS/Arch:           linux/arm64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          18.09.0
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.10.4
  Git commit:       4d60db4
  Built:            Wed Nov  7 00:17:01 2018
  OS/Arch:          linux/arm64
  Experimental:     false
@vielmetti

This comment has been minimized.

Copy link
Author

commented Dec 11, 2018

Looking at https://github.com/kubernetes-sigs/kind/blob/master/pkg/cluster/context.go#L196-L206 where the code errors out, there's a TODO for logging from @BenTheElder .

@vielmetti

This comment has been minimized.

Copy link
Author

commented Dec 11, 2018

Looks like the underlying reason is that kindest/node is not multiarchitecture.

ed@ed-2a-bcc-llvm:~$ docker run --rm mplatform/mquery kindest/node:v1.12.2
Image: kindest/node:v1.12.2
 * Manifest List: No
 * Supports: amd64/linux
@tao12345666333

This comment has been minimized.

Copy link
Member

commented Dec 11, 2018

@neolit123

This comment has been minimized.

Copy link
Contributor

commented Dec 12, 2018

/priority important-longterm
/kind feature

@BenTheElder BenTheElder added this to the 2019 goals milestone Dec 12, 2018

@BenTheElder BenTheElder changed the title `kind create cluster` fails on arm64 in `fixing mounts` Support arm64 Dec 13, 2018

@BenTheElder BenTheElder referenced this issue Dec 19, 2018

Open

ARM64 CI #188

@BenTheElder BenTheElder modified the milestones: 2019 goals, 1.0 Dec 19, 2018

@BenTheElder BenTheElder added this to To do in 1.0 via automation Dec 19, 2018

@BenTheElder

This comment has been minimized.

Copy link
Member

commented Dec 19, 2018

@dims hacked up a working version of this: https://paste.fedoraproject.org/paste/gdlF9fqXeSADK-aPN-sEbw/raw 🎉
I think we might want to put a little more thought into how to do this well like #188, but this should be doable. Tentatively tracking this for kind 1.0 in Q1 2019.

@petermbenjamin

This comment has been minimized.

Copy link

commented Apr 25, 2019

We're yet to see details around this announcement, but it ought to simplify the process of building ARM images significantly: https://techcrunch.com/2019/04/24/docker-partners-with-arm/

@BenTheElder

This comment has been minimized.

Copy link
Member

commented Apr 25, 2019

yes!

also somehow forgot to update this issue, we have arm64 support, just no published images yet (that will need some more thinking...) if you build images yourself kind should work on arm64 today 😅

@BenTheElder

This comment has been minimized.

Copy link
Member

commented Apr 25, 2019

@vielmetti

This comment has been minimized.

Copy link
Author

commented Apr 25, 2019

My fervent hope is that the new Docker tooling will make multi-architecture manifests much easier to produce. The current setup for most projects is just a bit complex.

@BenTheElder

This comment has been minimized.

Copy link
Member

commented May 3, 2019

So FWIW I did figure out how to work manifest-tool I think, this definitely seems feasible this year, if a bit clunky, the trickiest part now is we need to write some tooling to cross compile the kind image (or coordinate with kicking off a build on packet or ... 🤔).

I think I'd like to get this into GCB based publishing with a cross-compile so we can start automating publishing mulit-arch node images, I punted looking further into that while working on the breaking image changes in #461, but those are in now :-)

@aojea

This comment has been minimized.

Copy link
Contributor

commented May 3, 2019

@BenTheElder we can use CircleCI or Travis to do that, I see that another project under kubernetes-sig has the integrations enabled https://github.com/kubernetes-sigs/kubeadm-dind-cluster

I have experience building pipelines on those, it will be relatively easy to create a pipeline there to automatically publish the images based on PR or per tag

@BenTheElder

This comment has been minimized.

Copy link
Member

commented May 3, 2019

@aojea

This comment has been minimized.

Copy link
Contributor

commented May 6, 2019

hmm, I think I have to read more about prow https://github.com/kubernetes/test-infra/blob/master/prow/jobs.md

@BenTheElder

This comment has been minimized.

Copy link
Member

commented May 15, 2019

Minor update: I've ensured the ip-masq-agent pushes multi-arch (manifest) images upstream before we adopted it, and our own tiny networking daemon is cross compiled and pushing multi-arch images.

These were simpler than Kubernetes node images, but at least we have some more multi-arch samples and we continue to nominally work on arm64.

I think the next step is to support building node images from Kubernetes release tarballs so we can consume Kubernetes's upsteam cross compilation output and save time building & publishing. Then we start building manifest list images from those.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.