Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Node VM failure doesn't automatically recreate a pod with attached PV #80040

Open
vmwkkwong opened this issue Jul 11, 2019 · 3 comments

Comments

Projects
None yet
3 participants
@vmwkkwong
Copy link

commented Jul 11, 2019

What happened:
After a deployment is created with PVC, a node hosting the pod shuts down. After 6 mins timeout, the replacement/new pod is created but cannot come up since the volume is attached to a terminating pod on a shutdown node. To work around this, the volume would be detached from the original pod when it's force deleted.

What you expected to happen:
According to the Node VM Failure scenario at https://vmware.github.io/vsphere-storage-for-kubernetes/documentation/high-availability.html, the recovery mechanism is completely automatic. In reality, it requires manual intervention to force delete a pod.

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Environment:

  • Kubernetes version (use kubectl version): v1.12.7-gke.19
  • Cloud provider or hardware configuration: vSphere Cloud Provider
  • OS (e.g: cat /etc/os-release):
  • Kernel (e.g. uname -a):
  • Install tools:
  • Network plugin and version (if this is a network-related bug):
  • Others:

/sig vmware

@vmwkkwong

This comment has been minimized.

Copy link
Author

commented Jul 11, 2019

/sig vmware

@rhockenbury

This comment has been minimized.

Copy link

commented Jul 15, 2019

Out of curiosity, what 6 min timeout are you referring to?

@vmwkkwong

This comment has been minimized.

Copy link
Author

commented Jul 15, 2019

@rhockenbury k8s timeout to force unmount the PV.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.