etcd Pending status (volume mount timed out)

[garloff]

I have seen the etcd container in a Pending status for extended amounts of time (hours) and the event log indicating that the two volumes could not be mounted. (Which is ridiculous for

  etcd-certs:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/kubernetes/pki/etcd
    HostPathType:  DirectoryOrCreate

(and etcd-data which is /var/lib/etcd)).
It does only happen occasionally; if we're lucky, the etcd container does fail a liveness probe and gets restarted and works just fine then. But we can't rely on it.
I'll look into reproducing this, so I can insert proper logs here.

[garloff]

Here's what I see in the error case:

kubectl describe -n kube-system etcd-$PREFIX-${CLUSTER_NAME}-control-plane-genc01-XXXX
[...]
  Type     Reason       Age   From     Message
  ----     ------       ----  ----     -------
  Warning  FailedMount  65s   kubelet  Unable to attach or mount volumes: unmounted volumes=[etcd-data etcd-certs], unattached volumes=[etcd-data etcd-certs]: timed out waiting for the condition

[garloff]

Restarting the kubelet after kubeadm init avoids the issue, though this feels more like a workaround than a real fix..
If we tweak the heartbeat intervals on the other hand, this is even required to make the parameter take effect, so in some way it's the right thing to do.

[garloff]

We no longer tweak etcd's manifest after it has (potentially) started thanks to merging #282, so there is no real reason for restarting the kubelet any longer. We still have the restart however, as workaround for this issue.
TODO:

• Is it really still required?
• If so, debug the reasoning and search for real fix, likely upstream somewhere ....