docs: StatefulSet pod is never evicted from shutdown node

[at1984z]

Is this a BUG REPORT or FEATURE REQUEST?:

Uncomment only one, leave it on its own line:

/kind bug

/kind feature

What happened:

1. StatefulSet at scale 1 is created.
2. The only pod is placed and running on one of 2 worker nodes.
3. The worker node with running pod shuts down and never starts up again.
4. The pod never moved to other node.

What you expected to happen:

1. The pod would move to other node after expiration of default tolerations "node.alpha.kubernetes.io/notReady:NoExecute for 300s" and "node.alpha.kubernetes.io/unreachable:NoExecute for 300s"
2. "kubectl delete pod pod-on-shutdown-node" would induce the expected movement while the node is down -- it did not happen either.

How to reproduce it (as minimally and precisely as possible):

1. Create StatefulSet spec with one container and one replica in, say, sset.yml.
2. Have kubernetes installation with 2 worker nodes.
3. kubectl create -f sset.yml
4. kubectl get pod, to check to see where the only pod is scheduled, say, node N.
5. shutdown node N with "shutdown -h".
6. check to see that the pod did not move to other worker node in 10 minutes after node N halt time.

Anything else we need to know?:

1. A Deployment behaves as indicated in the "What you expected to happen" section.

Environment:

• Kubernetes version (use kubectl version): 1.8.1
• Cloud provider or hardware configuration**: Virtual machines with Vagrant 2.0.0 and VirtualBox 5.1.28-117968 on Intel(R) Xeon(R) CPU E5-2690 v3 24 cores with Ubuntu 16.04 LTS
• OS (e.g. from /etc/os-release): Ubuntu 16.04.3 LTS (VM)
• Kernel (e.g. uname -a): 4.4.0-96-generic (VM)
• Install tools: kubeadm 1.8.1-00
• Others:

---

Edit: Goal if this issue is to update the documentation and clarify the expected behavior as per: #54368 (comment)

[at1984z]

/sig area/app-lifecycle area/stateful-apps area/usability

[at1984z]

/sig area/node-lifecycle are/nodecontroller

[dims]

/area app-lifecycle
/area stateful-apps
/area usability
/area node-lifecycle
/area nodecontroller

[dims]

/sig node

[dixudx]

/assign