Kubeadm with unstable deb repo

[nsteinmetz]

Following #140, my findings:

Context:

• 3 RPI 3, 2 RPI2
• All are on HypriotOS 1.1.1
• Master is a RPI3

$ sudo kubeadm init --use-kubernetes-version=v1.4.5
Running pre-flight checks
<master/tokens> generated token: "4ed849.29f2eee78f75d062"
<master/pki> generated Certificate Authority key and certificate:
Issuer: CN=kubernetes | Subject: CN=kubernetes | CA: true
Not before: 2016-10-30 20:47:52 +0000 UTC Not After: 2026-10-28 20:47:52 +0000 UTC
Public: /etc/kubernetes/pki/ca-pub.pem
Private: /etc/kubernetes/pki/ca-key.pem
Cert: /etc/kubernetes/pki/ca.pem
<master/pki> generated API Server key and certificate:
Issuer: CN=kubernetes | Subject: CN=kube-apiserver | CA: false
Not before: 2016-10-30 20:47:52 +0000 UTC Not After: 2017-10-30 20:48:12 +0000 UTC
Alternate Names: [192.168.4.165 10.96.0.1 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local]
Public: /etc/kubernetes/pki/apiserver-pub.pem
Private: /etc/kubernetes/pki/apiserver-key.pem
Cert: /etc/kubernetes/pki/apiserver.pem
<master/pki> generated Service Account Signing keys:
Public: /etc/kubernetes/pki/sa-pub.pem
Private: /etc/kubernetes/pki/sa-key.pem
<master/pki> created keys and certificates in "/etc/kubernetes/pki"
<util/kubeconfig> created "/etc/kubernetes/admin.conf"
<util/kubeconfig> created "/etc/kubernetes/kubelet.conf"
<master/apiclient> created API client configuration
<master/apiclient> created API client, waiting for the control plane to become ready
<master/apiclient> all control plane components are healthy after 167.263559 seconds
<master/apiclient> waiting for at least one node to register and become ready
<master/apiclient> first node is ready after 1.542673 seconds
<master/apiclient> attempting a test deployment
<master/apiclient> test deployment succeeded
<master/discovery> created essential addon: kube-discovery, waiting for it to become ready
<master/discovery> kube-discovery is ready after 18.030144 seconds
<master/addons> created essential addon: kube-proxy
<master/addons> created essential addon: kube-dns

Kubernetes master initialised successfully!

You can now join any number of machines by running the following on each node:

kubeadm join --token=4ed849.29f2eee78f75d062 192.168.4.165

$ kubectl get ns
NAME          STATUS    AGE
default       Active    5m
kube-system   Active    5m

$ kubectl get cs
NAME                 STATUS      MESSAGE                                                                                        ERROR
controller-manager   Unhealthy   Get http://127.0.0.1:10252/healthz: dial tcp 127.0.0.1:10252: getsockopt: connection refused   
scheduler            Healthy     ok                                                                                             
etcd-0               Unhealthy   HTTP probe failed with statuscode: 503       

On a worker node:

$ sudo kubeadm join --token=4ed849.29f2eee78f75d062 192.168.4.165
Running pre-flight checks
<util/tokens> validating provided token
<node/discovery> created cluster info discovery client, requesting info from "http://192.168.4.165:9898/cluster-info/v1/?token-id=4ed849"
<node/discovery> cluster info object received, verifying signature using given token
<node/discovery> cluster info signature and contents are valid, will use API endpoints [https://192.168.4.165:6443]
<node/bootstrap> trying to connect to endpoint https://192.168.4.165:6443
<node/bootstrap> detected server version v1.4.5
<node/bootstrap> successfully established connection with endpoint https://192.168.4.165:6443
<node/csr> created API client to obtain unique certificate for this node, generating keys and certificate signing request
<node/csr> received signed certificate from the API server:
Issuer: CN=kubernetes | Subject: CN=system:node:pico-worker1 | CA: false
Not before: 2016-10-30 20:56:00 +0000 UTC Not After: 2017-10-30 20:56:00 +0000 UTC
<node/csr> generating kubelet configuration
<util/kubeconfig> created "/etc/kubernetes/kubelet.conf"

Node join complete:
* Certificate signing request sent to master and response
  received.
* Kubelet informed of new secure connection details.

Run 'kubectl get nodes' on the master to see this machine join.

And when I create the weave-kube (after I joined a first node btw) :

$ sudo kubectl create -f https://raw.githubusercontent.com/kodbasen/weave-kube-arm/master/weave-daemonset.yaml
daemonset "weave-net" created

And then:

$ kubectl get nodes
NAME           STATUS    AGE
pico-master    Ready     13m
pico-worker1   Ready     2m
HypriotOS/armv7: pirate@pico-master in ~
$ kubectl get cs
NAME                 STATUS    MESSAGE              ERROR
scheduler            Healthy   ok                   
controller-manager   Healthy   ok                   
etcd-0               Healthy   {"health": "true"}   
HypriotOS/armv7: pirate@pico-master in ~
$ kubectl get ns
NAME          STATUS    AGE
default       Active    13m
kube-system   Active    13m
HypriotOS/armv7: pirate@pico-master in ~

So maybe I didn"t wait enough for etcd at the first output above ?

Let me know if you need something else.

[nsteinmetz]

With the 4 RPi jointed into the cluster, load average on master side is between 5 and 8.

[luxas]

Interesting, which component is taking the most load?

[nsteinmetz]

Seems it starts to lower slowly...

From htop, in terms of CPU (with no special order)

• kubelet
• dockerd
• kube-apiserver

I'm now back to a load average of 2.11... just passed to 1.18... still lowering. So seems the init phase was quite ressource heavy. Will let the cluster as is to see till where it goes before doing any new action.

[nsteinmetz]

... but it increases or lowers... workers were quite calm so far, most are still (Load average 0 < LA < 2 for 1 RPI3 and 2 RPI2), but one among them is quite active with LA up to 4.45 (RPI3)

[nsteinmetz]

Some findings ; if you want output of special commands, just gave them to me :)

$ kubectl get pods --all-namespaces
NAMESPACE     NAME                                  READY     STATUS             RESTARTS   AGE
kube-system   dummy-2501624643-ox4bu                1/1       Running            0          34m
kube-system   etcd-pico-master                      1/1       Running            1          34m
kube-system   kube-apiserver-pico-master            1/1       Running            2          34m
kube-system   kube-controller-manager-pico-master   0/1       CrashLoopBackOff   7          34m
kube-system   kube-discovery-2202902116-6d9c4       1/1       Running            0          33m
kube-system   kube-dns-2334855451-kio06             3/3       Running            0          33m
kube-system   kube-proxy-3cw9z                      1/1       Running            0          33m
kube-system   kube-proxy-3efr7                      1/1       Running            0          13m
kube-system   kube-proxy-6yhmv                      1/1       Running            0          13m
kube-system   kube-proxy-l1gy0                      1/1       Running            0          13m
kube-system   kube-proxy-oygdb                      1/1       Running            0          24m
kube-system   kube-scheduler-pico-master            0/1       CrashLoopBackOff   7          33m
kube-system   weave-net-6kro4                       2/2       Running            0          23m
kube-system   weave-net-aab9k                       2/2       Running            0          23m
kube-system   weave-net-b1ib6                       2/2       Running            0          13m
kube-system   weave-net-qraay                       2/2       Running            0          13m
kube-system   weave-net-rswrd                       2/2       Running            1          16m

[nsteinmetz]

Seems there are still some etcd issues:

HypriotOS/armv7: pirate@pico-master in ~
$ sudo kubectl create namespace sock-shop
Error from server: client: etcd member http://127.0.0.1:2379 has no leader

but:

$ kubectl get cs
NAME                 STATUS    MESSAGE              ERROR
controller-manager   Healthy   ok                   
scheduler            Healthy   ok                   
etcd-0               Healthy   {"health": "true"}   

and:

$ kubectl get ns
NAME          STATUS    AGE
default       Active    44m
kube-system   Active    44m
sock-shop     Active    5m

[nsteinmetz]

After 1 day with no activities on it:

• Master is right now with 3 < LA < 5
• Workers looks calm with LA < 1
• On master; my top 3 CPU process are kubelet, dockerd and kubescheduler
  and:

HypriotOS/armv7: pirate@pico-master in ~
$ kubectl cluster-info
The connection to the server localhost:8080 was refused - did you specify the right host or port?
HypriotOS/armv7: pirate@pico-master in ~
$ kubectl get cs
The connection to the server localhost:8080 was refused - did you specify the right host or port?

[nsteinmetz]

Hi,

Till a later release, I'll use my 5 RPi to build a simple docker-swarm cluster and my 2 cubietrucks with HD as a glusterFS endpoint.

Aside of it, I'll test K8S on 64 bits infra to be more famillair with first and then come back to ARM version ;-)