Failed snapshot provision

[cehoffman]

Is this a BUG REPORT or FEATURE REQUEST?: BUG REPORT

What happened:

Failed to provision a snapshot.

What you expected to happen:

A snapshot was created and the resulting PVC using the snapshot could be bound.

How to reproduce it (as minimally and precisely as possible):

Create a snapshot while the underlying volume is in the process of cloud snapshot (might be related to a prior delete of the snapshot and then recreate.)

Create a PVC to use that snapshot.

Anything else we need to know?:

Environment:

• Kubernetes version (use kubectl version): 1.8.7-coreos.0
• Cloud provider or hardware configuration:
• OS (e.g. from /etc/os-release): CoreOS
• Kernel (e.g. uname -a):
• Install tools:
• Others:

This is the log from the leader stork instance. https://gist.github.com/0bfb04126e655ef4e8ec28058d8be0a3

The volume is review-label-chan-t0okwh-postgresql and at the end of the log. You can see earlier in the log that there was a failure due to the volume the snapshot was based on backing up to cloud storage.

[disrani-px]

Looks like some kind of race between create/delete that got exposed because of the delay/retry in deleting the snapshot. Due to this the controller ended up not trying to create the snapshot for that object.

[cehoffman]

Sounds like a likely explination. In this situation, we are using the snapshots to stand up a review instance of the production work with the new changes. Before doing this is clears out the last review instance by deleting all the resources and then creates new ones so the next set of changes have a fresh base.

[disrani-px]

Filed bug in kubernetes for this: kubernetes-retired/external-storage#646

Will work on a fix.