Make volume manager more robust across restarts

[saad-ali]

When kubelet is restarted, volume manager loses its internal state. It rebuilds this state and calls attach/mount on volumes that may already be attached/mounted.

This issue is to track two items:

1. Verify that if mount is called on a volume that is already mounted that it does not fail.
2. If a pod was deleted while kubelet was unavailable, the associated volume will not be unmounted (since Kubelet would not know about it after restart). So volume_manager should find a way to recover the pre-attached volumes and populate actual state of the world on startup so that they get cleaned up appropriately.

[yujuhong]

/cc @kubernetes/sig-node @dchen1107

Item (2) will lead to irremovable pod directories (see #27651 for an example), and these directories will accumulate over time. kubelet'd also try to clean up every 2s and fail, wasting cycles and spamming the log.
@saad-ali, is it possible to add a temporary fix for v1.3?

[saad-ali]

Item (2) will lead to irremovable pod directories

Only if kubelet is restarted and a pod with volumes mounted is deleted during that period.

I agree it must be fixed. But I don't think it's a v1.3 blocker. I think it can wait for v1.3.1

@matchstick for triage.

[yujuhong]

Only if kubelet is restarted and a pod with volumes mounted is deleted during that period.

Almost every pod has a default secret volume, so this applies to most pods. The difference between this issue and many other issues is that it's not transient and haunts kubelet forever even after restarts, and would only get worse as time goes by.

I agree it must be fixed. But I don't think it's a v1.3 blocker. I think it can wait for v1.3.1

Could this be documented as known issues in the release note if the decision is to table this for v1.3.1? Thanks!

[yujuhong]

/cc @freehan, this is why your kubelet log is full of error messages :)

[saad-ali]

Spoke with @matchstick. We're going to try to get it in for v1.3. If not, we'll punt it to post-1.3

[dchen1107]

Talked to both @yujuhong and @saad-ali We should fix this for 1.3 release otherwise, Kubelet will be quickly useless due to out-of-disk here. I marked this as P1.

[yifan-gu]

FWIW, this is very easy to reproduce by using local-up-cluster.sh. Basically after local-up, create a pod, nuke the cluster, local-up again, and then we will see messages like kubelet.go:2018] Failed to remove orphaned pod "03a5ea59-371b-11e6-98cf-28d244b00276" dir; err: remove /var/lib/kubelet/pods/03a5ea59-371b-11e6-98cf-28d244b00276/volumes/kubernetes.io~secret/default-token-385u2: device or resource busy.

[saad-ali]

@jingxu97 is working on implementing a clean up routine on startup

[yujuhong]

@jingxu97

A very simple script to reproduce this issue in an e2e cluster:

#!/bin/bash

NODES=( $(gcloud compute instances list $ZONE | awk '{print $1}' | grep e2e-test- | grep minion) )
KUBECTL="./cluster/kubectl.sh"

RC="
apiVersion: v1
kind: ReplicationController
metadata:
  name: foo 
  labels:
    k8s-app: foo 
spec:
  replicas: 90
  selector:
    k8s-app: foo 
  template:
    metadata:
      labels:
        k8s-app: foo 
    spec:
      containers:
      - image: gcr.io/google_containers/pause:2.0
        name: bar
"
# Create N pods.
echo "${RC}" | ${KUBECTL} create -f -

# Sleep a while and assume the pods will be running by then.
sleep 60

# Frocefully delete the pods.
${KUBECTL} delete rc --grace-period=0 foo

# Killing kubelet on each node.
# NOTE: It will be more effective to reproduce this issue if killing with kubelet
# overlaps with the RC deletion command.
for NODE in "${NODES[@]}"; do
    echo "Killing kubelet on node ${NODE}"
    gcloud compute ssh ${NODE} --command "sudo pkill kubelet"
done

# Give kubelet time to start up.
sleep 120

# Grep the error messages from the logs.
for NODE in "${NODES[@]}"; do
    echo "Checking error messages on node ${NODE}"
    gcloud compute ssh ${NODE} --command "grep \"Failed to remove orphaned pod\" /var/log/kubelet.log"
done

[saad-ali]

The design for this should also handle VolumesInUse() gracefully--i.e. if a pod continues to live after kubelet is restarted, it should have no gap in being reported in the Node's VolumesInUse status.

[thockin]

status on this?

[saad-ali]

Punted to post 1.3. Hopefully part of 1.3.x release

[matchstick]

OK we need to make sure we don't lose track of this one, but it is not blocking 1.3 release. Raising to P0 and removing from 1.3 milestone.

[goltermann]

This is an issue marked v1.3 and kind/documentation. Can you get it fixed up this week?