Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Track the gaps when porting to ARM (arm7l) #4294

Open
lwolf opened this issue Feb 22, 2019 · 11 comments

Comments

Projects
None yet
7 participants
@lwolf
Copy link

commented Feb 22, 2019

Is this a BUG REPORT or FEATURE REQUEST? (choose one):
FEATURE REQUEST

Environment:

  • Cloud provider or hardware configuration:
    Hardware

  • OS (printf "$(uname -srm)\n$(cat /etc/os-release)\n"):

Linux 4.14.78-150 armv7l
NAME="Ubuntu"
VERSION="18.04.1 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.1 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic
  • Version of Ansible (ansible --version):
    ansible 2.7.2
    python version = 2.7.10 (default, Oct 6 2017, 22:29:07) [GCC 4.2.1 Compatible Apple LLVM 9.0.0 (clang-900.0.31)]

Kubespray version (commit) (git rev-parse --short HEAD): Latest master branch

Anything else do we need to know:

At the moment it is impossible to install kubespray on pure arm7l hardware.
Most of the containers does not provide binaries/containers for non-amd64 arch.

The aim of this ticket is to make it possible use ARM hardware as pool of worker nodes alongside the amd64 master/nodes.
I've found a several gaps when trying to install kubespray on arm7l devices. So

  1. Checksums aren't available for arm, only for amd64,arm64
  • add checksums for the main components - hyperkube, kubeadm and cni_binary #4261
  1. Add NodeSelector to some manifests
  • find all the places where only amd64 containers are available (tiller, dashboard, dnsautoscaler)
  1. Overlay network support, provide per architecture daemonsets gated by NodeSelector
  • flannel: could be deployed to all the architectures (arm, arm64,amd64,etc)
  • calico: could be run on amd64,arm64
  1. Etcd - etcd does not provide binaries for arm32. Until there will be one, ARM nodes can't act as a master nodes.
  • build etcd for arm7l and create a container

related issues: #4261 #4065

@nmiculinic

This comment has been minimized.

Copy link
Contributor

commented Feb 25, 2019

Also default pause container is invalid (( plus has '\n' in it for arm7l and it breaks systemd unit ))

@nmiculinic

This comment has been minimized.

Copy link
Contributor

commented Feb 25, 2019

pod_infra_image_repo: k8s.gcr.io/pause
pod_infra_image_tag: "3.1"

works for me

@ant31

This comment has been minimized.

Copy link
Contributor

commented Feb 25, 2019

what are the production usecase to run workload on arm7 ?
It's already to complex to have both amd64 and arm64, I would limit the options to only real production usecases.

@nmiculinic

This comment has been minimized.

Copy link
Contributor

commented Feb 25, 2019

IoT on BBB devices in my usecase for example

@nmiculinic

This comment has been minimized.

Copy link
Contributor

commented Feb 25, 2019

(( Beagle bone black ))

@ant31

This comment has been minimized.

Copy link
Contributor

commented Feb 26, 2019

what can of load are you running on those machines? why kubernetes?

@nmiculinic

This comment has been minimized.

Copy link
Contributor

commented Feb 26, 2019

A quite simple load, reading from serial UARTs and sending to the message queue data.

Why kubernetes?

Because I'm familiar with it and it serves me health checks, deployments, upgrade process nicely. It nicely manages secrets and monitoring. With other solutions such as consul + ansible + docker I'd had to have some verification deployment successfully completed + gradual rollout. Maybe I could also use a spinnaker or something like that, though I'm not familiar with it, and I run k8s for the rest of the infrastructure.

K8s downside on edge devices in CPU usage...it's around 15% CPU time just on kubelet, even after tuning various housekeeping/node status freq parameters. Most of the time is spent during syscalls, runtime, and some JSON decoding.

EDIT: This is AM335x 1GHz ARM® Cortex-A8, and I see 30% branch misprediction rate system-wide ( also similar amount for kubelet )...so yeah, not the best processor in the world nor the most powerful.

@b23prodtm

This comment has been minimized.

Copy link

commented Apr 1, 2019

According to coreOS docs

etcd has known issues on 32-bit systems due to a bug in the Go runtime

Etcd-io doesn't provide any 32bits ARM binaries because of a Go language issue. Otherwise we could have downloaded tarballs from cores/etcd-io (https://github.com/coreos/etcd/releases/download/) and checksums .

@Miouge1

This comment has been minimized.

Copy link
Member

commented Apr 5, 2019

/kind feature
/help

@k8s-ci-robot

This comment has been minimized.

Copy link
Contributor

commented Apr 5, 2019

@Miouge1:
This request has been marked as needing help from a contributor.

Please ensure the request meets the requirements listed here.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-help command.

In response to this:

/kind feature
/help

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@rich-nahra

This comment has been minimized.

Copy link

commented May 5, 2019

I ran into this issue today. I have a esxi home lab with limited amount of RAM so I thought it would be nice to run masters / etcd on ARM and minions on vm's. etcd seems to be the only blocker.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.