kubeadm init hits docker container memory limit (code 137) when starting kube-apiserver container

[jrgilman]

Is this a BUG REPORT or FEATURE REQUEST?

BUG REPORT

Versions

kubeadm version (use kubeadm version): kubeadm version: &version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.2", GitCommit:"17c77c7898218073f14c8d573582e8d2313dc740", GitTreeState:"clean", BuildDate:"2018-10-24T06:51:33Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/arm"}

Environment:

• Kubernetes version (use kubectl version): Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.2", GitCommit:"17c77c7898218073f14c8d573582e8d2313dc740", GitTreeState:"clean", BuildDate:"2018-10-24T06:54:59Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/arm"}
• Cloud provider or hardware configuration: Raspberry Pi 3B+
• OS (e.g. from /etc/os-release): Raspbian GNU/Linux 9 (stretch)
• Kernel (e.g. uname -a): Linux rpi0 4.14.79-v7+ #1159 SMP Sun Nov 4 17:50:20 GMT 2018 armv7l GNU/Linux

What happened?

When running kubeadm init the k8s.gcr.io/kube-apiserver v1.12.2 is hitting the memory kill limit and is exiting with code 137. This happens a few times and kubeadm eventually errors out with the following message:

pi@rpi0:~ $ sudo kubeadm init --token-ttl=0 --kubernetes-version v1.12.2 --apiserver-advertise-address=192.168.1.20
...
[init] waiting for the kubelet to boot up the control plane as Static Pods from directory "/etc/kubernetes/manifests"
[init] this might take a minute or longer if the control plane images have to be pulled

Unfortunately, an error has occurred:
	timed out waiting for the condition

This error is likely caused by:
	- The kubelet is not running
	- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)

If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
	- 'systemctl status kubelet'
	- 'journalctl -xeu kubelet'

Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
Here is one example how you may list all Kubernetes containers running in docker:
	- 'docker ps -a | grep kube | grep -v pause'
	Once you have found the failing container, you can inspect its logs with:
	- 'docker logs CONTAINERID'
couldn't initialize a Kubernetes cluster

What you expected to happen?

kubeadm init to not crash.

How to reproduce it (as minimally and precisely as possible)?

1. Get a Raspberry Pi 3B+
2. Run following commands:

# disable swap for kubernetes
sudo dphys-swapfile swapoff && sudo dphys-swapfile uninstall && sudo update-rc.d dphys-swapfile remove
cat /boot/cmdline.txt <(echo "cgroup_enable=cpuset cgroup_enable=memory") | tr '\n' ' ' | sed -E 's/\s+$//g' | sudo tee /boot/cmdline.txt > /dev/null
sudo reboot

# install docker
export VERSION=18.06 && curl -sSL get.docker.com | sh
sudo usermod pi -aG docker
newgrp docker

# install kubernetes
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add - 
echo "deb http://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list
sudo apt-get update -q
sudo apt-get install -qy kubeadm

sudo kubeadm init --apiserver-advertise-address=192.168.1.20

Anything else we need to know?

The container gets killed when the machine approaches 390MB of 927MB of RAM used.

[neolit123]

i think it's better if you pre-pull the images for the Pi 3B+.
https://www.raspberrypi.org/products/raspberry-pi-3-model-b-plus/

you can do this on a more powerful machine:

kubeadm config images list --kubernetes-version v1.12.2
kubeadm config images pull --kubernetes-version v1.12.2

and then move the images.

but our docs do state that the system requirements for kubeadm are dual core CPU + 2GB RAM:
https://kubernetes.io/docs/setup/independent/install-kubeadm/#before-you-begin

[timothysc]

You're running out of memory on a host which probably has a large reserve for the kernel. This isn't a kubeadm issue, but a memory constraint problem.