New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Bug 1889420: Add dangling volume check for vsphere #445
Bug 1889420: Add dangling volume check for vsphere #445
Conversation
@gnufied: This pull request references Bugzilla bug 1889420, which is valid. The bug has been moved to the POST state. The bug has been updated to refer to the pull request using the external bug tracker. 3 validation(s) were run on this bug
In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
/lgtm |
/retest |
1 similar comment
/retest |
@gnufied: The following test failed, say
Full PR test history. Your PR dashboard. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here. |
attach detach controller fixes https://bugzilla.redhat.com/show_bug.cgi?id=1889420
8a0e3b5
to
3a35aa6
Compare
/retest |
/lgtm |
@gnufied: This pull request references Bugzilla bug 1889420, which is valid. 3 validation(s) were run on this bug
In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
@gnufied: This pull request references Bugzilla bug 1889420, which is valid. 3 validation(s) were run on this bug
In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
/retest |
/approve |
/retest |
@gnufied: This pull request references Bugzilla bug 1889420, which is valid. 3 validation(s) were run on this bug
In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
/assign @soltysh @smarterclayton |
// such nodes won't be part of nodeVolumes map because attach-detach controller does not keep track | ||
// such nodes. But such nodes may still have dangling volumes on them and hence we need to scan all the | ||
// remaining nodes which weren't scanned by code previously. | ||
vs.BuildMissingVolumeNodeMap(ctx) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
How expensive is this? Is this method part of a periodic check, or invoked once per disk attach/detach?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This check is performed periodically. The check should not be more expensive than what we were doing already. We were already scanning all the nodes (although grouped by vcenter) for disks and reconciling them for disks that should be attached to a node but somehow is detached. This mechanism exists for all cloudproviders.
What this change is doing is kind of other way around. As part of same loop (which already runs), it will periodically scan for disks that shouldn't be attached to a node but somehow are (say because of a crashed node) and it ensures that attach-detach-controller knows about such disks.
/retest |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
/approve
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: gnufied, huffmanca, jsafrane, knobunc, soltysh The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
/retest Please review the full test history for this PR and help us cut down flakes. |
12 similar comments
/retest Please review the full test history for this PR and help us cut down flakes. |
/retest Please review the full test history for this PR and help us cut down flakes. |
/retest Please review the full test history for this PR and help us cut down flakes. |
/retest Please review the full test history for this PR and help us cut down flakes. |
/retest Please review the full test history for this PR and help us cut down flakes. |
/retest Please review the full test history for this PR and help us cut down flakes. |
/retest Please review the full test history for this PR and help us cut down flakes. |
/retest Please review the full test history for this PR and help us cut down flakes. |
/retest Please review the full test history for this PR and help us cut down flakes. |
/retest Please review the full test history for this PR and help us cut down flakes. |
/retest Please review the full test history for this PR and help us cut down flakes. |
/retest Please review the full test history for this PR and help us cut down flakes. |
@gnufied: All pull requests linked via external trackers have merged: Bugzilla bug 1889420 has been moved to the MODIFIED state. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
Fixes https://bugzilla.redhat.com/show_bug.cgi?id=1889420
cc @openshift/sig-storage
The dangling volume mechanism in vsphere has one limitation though - since this code relies on verifying-volumes -as-attached feature of ADC, there must be at least one pod running with any vsphere/attachable volumes in the cluster. This means that, if a cluster had only one pod with vsphere volume and that volume was attached to wrong node (i.e dangling to wrong node), then this mechanism won't work. In practice I hope, it is unlikely that we will have just one pod in the cluster and that will be with a dangling volume.