Get role working on CentOS 7 (and Red Hat, by extension)

[geerlingguy]

Currently, I have almost everything working... except for it gets stuck on the kubeadm init task now (it wasn't getting stuck earlier today... so some fix I applied for something else must be blocking init from completing):

TASK [role_under_test : Initialize the Kubernetes master with kubeadm init.] ***
No output has been received in the last 10m0s, this potentially indicates a stalled build or something wrong with the build itself.
Check the details on how to adjust your build configuration on: https://docs.travis-ci.com/user/common-build-problems/#Build-times-out-because-no-output-was-received
The build has been terminated

Failed build: https://travis-ci.org/geerlingguy/ansible-role-kubernetes/jobs/377042422

Probably something blocking/breaking kubelet startup (this was happening for a variety of reasons earlier in this role's as-yet short history).

[geerlingguy]

Yep, some output from journalctl -eu kubelet:

May 09 22:23:23 0b9252601962 kubelet[1182]: E0509 22:23:23.018605    1182 eviction_manager.go:246] eviction manager: failed to get get summary stats: failed to get node info: node "0b9252601962" not found
May 09 22:23:23 0b9252601962 kubelet[1182]: W0509 22:23:23.132950    1182 cni.go:171] Unable to update cni config: No networks found in /etc/cni/net.d
May 09 22:23:25 0b9252601962 kubelet[1182]: E0509 22:23:25.689124    1182 event.go:209] Unable to write event: 'Post https://172.17.0.2:6443/api/v1/namespaces/default/events: dial tcp 172.17.0.2:6443: getsockopt: connection refused' (may retry after sleeping)
May 09 22:23:25 0b9252601962 kubelet[1182]: E0509 22:23:25.689171    1182 event.go:144] Unable to write event '&v1.Event {TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"0b9252601962.152d19dbdd88901a", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64) (nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences []v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, InvolvedObject:v1.ObjectReference{Kind:"Node", Namespace:"", Name:"0b9252601962", UID:"0b9252601962", APIVersion:"", ResourceVersion:"", FieldPath:""}, Reason:"NodeHasSufficientDisk", Message:"Node 0b9252601962 status is now: NodeHasSufficientDisk", Source:v1.EventSource{Component:"kubelet", Host:"0b9252601962"}, FirstTimestamp:v1.Time{Time:time.Time{wall:0xbeb4fac632e2a01a, ext:295113266, loc:(*time.Location)(0x5b9f020)}}, LastTimestamp:v1.Time{Time:time.Time{wall:0xbeb4fac632e2a01a, ext:295113266, loc:(*time.Location)(0x5b9f020)}}, Count:1, Type:"Normal", EventTime:v1.MicroTime{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}' (retry limit exceeded!)
May 09 22:23:25 0b9252601962 kubelet[1182]: E0509 22:23:25.689370    1182 event.go:209] Unable to write event: 'Post https://172.17.0.2:6443/api/v1/namespaces/default/events: dial tcp 172.17.0.2:6443: getsockopt: connection refused' (may retry after sleeping)
May 09 22:23:26 0b9252601962 kubelet[1182]: E0509 22:23:26.429219    1182 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://172.17.0.2:6443/api/v1/pods?fieldSelector=spec.nodeName%3D0b9252601962&limit=500&resourceVersion=0: dial tcp 172.17.0.2:6443: getsockopt: connection refused
May 09 22:23:26 0b9252601962 kubelet[1182]: E0509 22:23:26.448023    1182 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: Failed to list *v1.Service: Get https://172.17.0.2:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.17.0.2:6443: getsockopt: connection refused
May 09 22:23:26 0b9252601962 kubelet[1182]: E0509 22:23:26.450937    1182 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:460: Failed to list *v1.Node: Get https://172.17.0.2:6443/api/v1/nodes?fieldSelector=metadata.name%3D0b9252601962&limit=500&resourceVersion=0: dial tcp 172.17.0.2:6443: getsockopt: connection refused

[geerlingguy]

Well, digging deeper, at least locally, the problem was due to a docker-in-docker aufs issue—you can't store aufs volumes inside the host aufs volume directory, so you have to make sure /var/lib/docker is a volume in the containing Docker container... see warning in http://jpetazzo.github.io/2015/09/03/do-not-use-docker-in-docker-for-ci/

[geerlingguy]

Everything worked great after I added --volume=/var/lib/docker to my docker run command. We'll see if Travis CI is happy as well.

[geerlingguy]

Well... it works locally :P — Travis CI seems to hang.

[geerlingguy]

Works now, it looks like it may have been related to the daemon needing a reload after changing the kubelet options.