-
Notifications
You must be signed in to change notification settings - Fork 38.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Automated cherry pick of #85675 Fix AWS eventual consistency of AttachDisk #89891
Automated cherry pick of #85675 Fix AWS eventual consistency of AttachDisk #89891
Conversation
AWS eventual consistency can go back in time. It can return that a volume is detached and then that it is attached. When this happens during attachment of the same volume to the same node, but with a different device name, retry DescribeVolumes a few times before reporting an error. 10 retries should be enough to get a consistent result.
In case DescribeVolumes returns stale attachment and the volume was previously attached to a different node.
Hi @johanneswuerbach. Thanks for your PR. I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
/kind bug |
/assign @zmerlynn |
/lgtm |
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: johanneswuerbach, jsafrane The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
Cherry pick of #85675 on release-1.17.
#85675: Fix AWS eventual consistency of AttachDisk
For details on the cherry pick process, see the cherry pick requests page.
We are hitting a similar issue @jsafrane described and AWS EBS attach operations randomly failing the first attach after a detach with "timed out waiting for the condition" just to be retried and working afterwards.