Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

On AWS Attach followed by a detach of same volume is stuck waiting for volume attachment on old device path #77949

Open
gnufied opened this issue May 15, 2019 · 5 comments

Comments

Projects
None yet
3 participants
@gnufied
Copy link
Member

commented May 15, 2019

We observed in an e2e that if attach is performed on a volume that was detached before from same node, sometimes the mount of the volume (for 2nd pod) could be stuck waiting on invalid devicePath in WaitForAttach call.

The problem is - when attach for second pod failed with "Error attaching EBS volume \"vol-xxx\"" to instance "i-xxx" since volume is currently attached to "i-xxx", this caused an dangling volume error from AWS cloudprovider(implemented by us). The dangling volume error is supposed to auto-correct any volumes which are attached to a node that it shouldn't be. But in this case - the dangling volume error was triggered by the fact that, volume was not actually attached to the node but because describeVolume call for 2nd attachment returned stale data (because of AWS eventual consistency).

So, dangling volume exception caused volume to be added to actual state of world (so as it can be detached later), but this caused volume to be reported to node's Status field. And that caused second pod to assume volume as attached to invalid device path and hence 2nd pod was stuck for 10 minutes waiting for volume to attach on a device path (/dev/xvdbl) which does not exist.

@gnufied

This comment has been minimized.

Copy link
Member Author

commented May 15, 2019

/sig storage
/sig aws

cc @jsafrane @leakingtapan

@gnufied gnufied changed the title On AWS Attach followed by a detach of same volume is stuck waiting for volume attachment on old device path ( On AWS Attach followed by a detach of same volume is stuck waiting for volume attachment on old device path May 15, 2019

@gnufied

This comment has been minimized.

Copy link
Member Author

commented May 15, 2019

Here is k8s logs that confirm that, describeVolume did return stale data for volume:

See detach succeeds at 19.30.25:

I0514 19:30:25.326374       1 operation_generator.go:423] DetachVolume.Detach succeeded for volume "pvc-a007686c-767e-11e9-9339-129c3af80d5c" (UniqueName: "kubernetes.io/aws-ebs/aws://us-east-1b/vol-04f136c2462a930a1") on node "ip-10-0-144-99.ec2.internal" 

and yet:

I0514 19:30:25.510323       1 aws.go:1726] volume ID: vol-04f136c2462a930a1: device mappings from EC2 Instance: map[a:vol-00a246b72606dab0b ca:vol-049251631ad7e7638]
I0514 19:30:25.510335       1 aws.go:1738] volume ID: vol-04f136c2462a930a1: Full device mappings: map[a:vol-00a246b72606dab0b ca:vol-049251631ad7e7638]
I0514 19:30:25.510356       1 aws.go:1779] Assigned mount device bv -> volume vol-04f136c2462a930a1
E0514 19:30:25.732642       1 aws.go:2133] "Error attaching EBS volume \"vol-04f136c2462a930a1\"" to instance "i-018e05c99167cc3f9" since volume is currently attached to "i-018e05c99167cc3f9"

so second DescribeVolume API call is actually returning stale data.

@gnufied

This comment has been minimized.

Copy link
Member Author

commented May 15, 2019

There are couple of ways of fixing this:

  1. Dangling error is used by AWS and Openstack cloudproviders to fix incorrectly attached volumes. We could fix mechanism of dangling volumes, so as when it adds volumes to actual state of the world - https://github.com/kubernetes/kubernetes/blob/master/pkg/volume/util/operationexecutor/operation_generator.go#L339 it does so without reporting the volume to node's Status.

  2. We can make a local fix inside AWS so that when we return DanglingVolume error, before returning that error we also verify volume attachment by querying the instance. We already have instance information in that call. Verifying volume attachment by describeVolume and describeInstance should reduce likelyhood of returning incorrect DanglingVolume errors. There is still a chance though that, both API calls would land on a AWS backend which does not yet have updated information.

@gnufied

This comment has been minimized.

Copy link
Member Author

commented May 15, 2019

expanding on fix#1 - I am thinking when we add a volume to actual state of the world via DanglingVolume error, then we should add the volume as "uncertain". We will update the code to not report "uncertain" volumes in node's status and hence in this case volume state will be fixed correctly.

If I understand correctly - @jingxu97 did not intend to report "uncertain" volumes to node's status anyways.

@jsafrane

This comment has been minimized.

Copy link
Member

commented May 16, 2019

This looks like another symptom of #77389.

we also verify volume attachment by querying the instance

I tried that, it does not always help. Both DescribeVolume and DescribeInstance may return the same stale data, see #77389 (comment)

IMO, #1 is the right way to go. Unless it break something else, all these AWS corner cases were complicated already and now anything can break.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.