Fix hotplug volume have a io error when use iscsi protocol csi plugin

[watermelon-brother]

What this PR does / why we need it:
Now, virt-controller can delete old attachment pod when the new attachmentpod is running.So,It can't cause the volume iscsi connection break when delete the old attachment pod.
Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #6564

Special notes for your reviewer:

Release note:
None

[kubevirt-bot]

@watermelon-brother: Adding the "do-not-merge/release-note-label-needed" label because no release-note block was detected, please follow our release note process to remove it.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[kubevirt-bot]

Hi @watermelon-brother. Thanks for your PR.

I'm waiting for a kubevirt member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[kubevirt-bot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
To complete the pull request process, please assign mazzystr after the PR has been reviewed.
You can assign the PR to them by writing /assign @mazzystr in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[kubevirt-bot]

@watermelon-brother: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/ok-to-test

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[maya-r]

/ok-to-test

[maya-r]

fyi @awels

[alicefr]

/retest

[watermelon-brother]

/retest

[watermelon-brother]

/retest

[watermelon-brother]

The tide stage of pipeline is not pass,How to solve it? @awels

[alicefr]

@watermelon-brother you need approval and a lgtm from reviewers to get this merged.

@awels do we want this solution? Should we try it with another CSI plugin that do not accept multiple pods accessing a RWO volume?

[awels]

Any reason you used a switch instead of if len(currentPod) == 0 { ... } else { ... }

[kubevirt-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[awels]

The solution I think is fine, it is just I am curious about the switch instead of an if else since there are only 2 options.

[DMajrekar]

I've been testing this change with OpenEBS/Mayastor backed hot plugged volumes, and this PR introduces and undesired side effect.

I've been testing with both block and file backed volumes, and executed the following steps:

1. Create a VM
2. Attach a first hot plug device block-test
3. Partition / Format the device within the VM: sda
4. Attach a second hot plug device block-test-2
5. Partition / Format the device within the VM: sdb
6. unmount the first device, sda from the VM
7. Hot plug remove the device block-test from the VM (virtctl removevolume volume-attach --volume-name block-test
8. Check the mount points within the VM, and notice that sda has been replaced with sdb, leaving sdb in an I/O error state

Looking at the virt-launcher logs for this VM, I'm seeing:

{"component":"virt-launcher","kind":"","level":"info","msg":"Attaching disk block-test, target sda","name":"volume-attach-2438-4bcde4","namespace":"cust-ba6bc6b1-808d-default","pos":"manager.go:929","timestamp":"2022-03-28T18:12:50.396629Z","uid":"ba639b80-983a-4304-b491-fdbc27d449bb"}
{"component":"virt-launcher","kind":"","level":"info","msg":"Attaching disk block-test-2, target sdb","name":"volume-attach-2438-4bcde4","namespace":"cust-ba6bc6b1-808d-default","pos":"manager.go:929","timestamp":"2022-03-28T18:13:16.131228Z","uid":"ba639b80-983a-4304-b491-fdbc27d449bb"}


{"component":"virt-launcher","kind":"","level":"info","msg":"Detaching disk block-test, target sda","name":"volume-attach-2438-4bcde4","namespace":"cust-ba6bc6b1-808d-default","pos":"manager.go:908","timestamp":"2022-03-28T18:13:48.450685Z","uid":"ba639b80-983a-4304-b491-fdbc27d449bb"}
{"component":"virt-launcher","kind":"","level":"info","msg":"Detaching disk block-test-2, target sdb","name":"volume-attach-2438-4bcde4","namespace":"cust-ba6bc6b1-808d-default","pos":"manager.go:908","timestamp":"2022-03-28T18:13:48.567696Z","uid":"ba639b80-983a-4304-b491-fdbc27d449bb"}
{"component":"virt-launcher","kind":"","level":"info","msg":"Attaching disk block-test-2, target sdb","name":"volume-attach-2438-4bcde4","namespace":"cust-ba6bc6b1-808d-default","pos":"manager.go:929","timestamp":"2022-03-28T18:13:48.747398Z","uid":"ba639b80-983a-4304-b491-fdbc27d449bb"}

I however have been experiencing the same problems as seen in #6564 with OpenEBS/Mayastor which relies on NVMe over fabric rather than iSCSI.

Working though the issue, we have identified that the 2nd and 3rd hot plug pods are missing the volumeDevices section for previously mounted volumes. That is causing the Node CSI driver to performa an UNSTAGE of the volume (effectively removing the block device from the underlying compute node).

As a quick hack, I have forced subsequent hot plug pods to maintain the volumeDevice spec with the following patch

diff --git a/pkg/virt-controller/services/template.go b/pkg/virt-controller/services/template.go
index b1e102efa..892598387 100644
--- a/pkg/virt-controller/services/template.go
+++ b/pkg/virt-controller/services/template.go
@@ -1622,7 +1622,7 @@ func (t *templateService) RenderHotplugAttachmentPodTemplate(volumes []*v1.Volum
                }
                skipMount := false
                if hotplugVolumeStatusMap[volume.Name] == v1.VolumeReady || hotplugVolumeStatusMap[volume.Name] == v1.HotplugVolumeMounted {
-                       skipMount = true
+                       skipMount = false
                }
                pod.Spec.Volumes = append(pod.Spec.Volumes, k8sv1.Volume{
                        Name: volume.Name,

This now means the hot plug path is UNPUBLISHED, but is not UNSTAGED from the underlying compute node, meaning IO can still flow. It also does not show the same behaviour as I have described at the start of this, where the guest VM sees both devices detached and then a single device reattached.

These are the Virtual launcher logs with my patch:

{"component":"virt-launcher","kind":"","level":"info","msg":"Attaching disk block-test, target sda","name":"volume-attach-2438-4bcde4","namespace":"cust-ba6bc6b1-808d-default","pos":"manager.go:929","timestamp":"2022-03-28T19:49:43.118806Z","uid":"ba639b80-983a-4304-b491-fdbc27d449bb"}
{"component":"virt-launcher","kind":"","level":"info","msg":"Attaching disk block-test-2, target sdb","name":"volume-attach-2438-4bcde4","namespace":"cust-ba6bc6b1-808d-default","pos":"manager.go:929","timestamp":"2022-03-28T19:49:59.882321Z","uid":"ba639b80-983a-4304-b491-fdbc27d449bb"}

I will write this up in a new issue tomorrow, but at the moment, I feel this PR is not safe to merge.

[kubevirt-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

/lifecycle rotten

[kubevirt-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

/close

[kubevirt-bot]

@kubevirt-bot: Closed this PR.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.