fix(vmclone): delete snapshot and restore after pvc bound #10888

fossedihelm · 2023-12-15T09:38:25Z

The clone process involves vmsnapshot and vmrestore.
Currently, these are both deleted as soon as the restore is finished. If the source vm has a pvc the vmrestore process creates another pvc, pointing to the volumesnapshot previously created by the vmsnapshot.
With Immediate binding mode the pvc(s) created are immediately bound.
When WaitForFirstConsumer binding mode is used the pvc will stay in "Pending" state until a pod is created(the virt-launcher pod).
In the latter case, when the cloned vm is started, it is stuck in "WaitingForVolumeBinding" becuse the related PVC binding is failing, due to the snapshot source has being deleted. In other words, we cannot delete the snapshot resources until all the pvc are Bound.

This patch aim to delay the snapshot and the restore deletion until we are sure that all the pvc are bound.

What this PR does / why we need it:

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #10832
jira-ticket: https://issues.redhat.com/browse/CNV-32667
Special notes for your reviewer:

Checklist

This checklist is not enforcing, but it's a reminder of items that could be relevant to every PR.
Approvers are expected to review this list.

Design: A design document was considered and is present (link) or not required
PR: The PR description is expressive enough and will help future contributors
Code: Write code that humans can understand and Keep it simple
Refactor: You have left the code cleaner than you found it (Boy Scout Rule)
Upgrade: Impact of this change on upgrade flows was considered and addressed if required
Testing: New code requires new unit tests. New features and bug fixes require at least on e2e test
Documentation: A user-guide update was considered and is present (link) or not required. You want a user-guide update if it's a user facing feature / API change.
Community: Announcement to kubevirt-dev was considered

Release note:

[Bugfix] Clone VM with WaitForFirstConsumer binding mode PVC now works.

akalenyu

Looks good to me, just a bit concerned about testing

Should we start talking about a periodic with ceph-wffc so we don't miss this kind of stuff in the future? ofc this would be effort outside of this PR and owned by sig-storage

pkg/virt-controller/watch/clone/clone_test.go

fossedihelm · 2023-12-18T15:06:46Z

Looks good to me, just a bit concerned about testing

Should we start talking about a periodic with ceph-wffc so we don't miss this kind of stuff in the future? ofc this would be effort outside of this PR and owned by sig-storage

Thanks @akalenyu for the review!
I still don't have a strong opinion regard this. I think that probably, we should start thinking about it. In particular if there are many places where WFFC has a different behavior regarding the Immediate binding mode.

fossedihelm · 2024-01-03T13:01:25Z

/retest

acardace · 2024-01-08T10:59:29Z

/retest

enp0s3 · 2024-01-09T13:13:19Z

pkg/virt-controller/watch/clone/clone_test.go

+					expectEvent(TargetVMCreated)
+				})
+
+				It("if not all the PVCs are bound, nothing should happen", func() {


@fossedihelm Hi! just want to make sure, if you run this test w/o the changes in the product code, do you get the issue reproduction?

enp0s3

/approve

Thanks @fossedihelm !

kubevirt-bot · 2024-01-10T13:45:08Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: enp0s3

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [enp0s3]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

xpivarc

Can we get a functional test?

pkg/virt-controller/watch/clone/clone.go

The clone process involves `vmsnapshot` and `vmrestore`. Currently, these are both deleted as soon as the restore is finished. If the source vm has a pvc the `vmrestore` process creates another pvc, pointing to the volumesnapshot previously created by the `vmsnapshot`. With `Immediate` binding mode the pvc(s) created are immediately bound. When `WaitForFirstConsumer` binding mode is used the pvc will stay in "Pending" state until a pod is created(the virt-launcher pod). In the latter case, when the cloned vm is started, it is stuck in "WaitingForVolumeBinding" becuse the related PVC binding is failing, due to the snapshot source has being deleted. In other words, we cannot delete the snapshot resources until all the pvc are Bound. This patch aim to delay the snapshot and the restore deletion until we are sure that all the pvc are bound. Signed-off-by: fossedihelm <ffossemo@redhat.com>

If a pvc associated to a vm clone is using a storage class with wffc binding mode, the vmrestore and the vm snapshot resources should be deleted only after the target pvc are bound. This commit cover this case with an e2e test. Signed-off-by: fossedihelm <ffossemo@redhat.com>

fossedihelm · 2024-01-19T09:38:37Z

@xpivarc I added the functional test and rebased. PTAL Thanks!

kbidarkar · 2024-01-23T06:10:33Z

/retest

xpivarc · 2024-01-23T13:04:40Z

tests/clone_test.go

@@ -526,6 +528,10 @@ var _ = Describe("[Serial]VirtualMachineClone Tests", Serial, func() {
 				vm, err := virtClient.VirtualMachine(vm.Namespace).Create(context.Background(), vm)
 				Expect(err).ToNot(HaveOccurred())

+				if !running && libstorage.IsStorageClassBindingModeWaitForFirstConsumer(storageClass) {


@akalenyu @awels
Should we just extend the HaveSucceded or what is the expected state?

IMO we should not extend the HaveSucceeded, "hiding" the wffc binding mode behavior.
The createVMWithStorageClass function should address the different behavior between the storage classes.

I would prefer the more verbose variant here:
844e69b#diff-847cd06c1bf6ef597ea96894f8b32ecb4d074ad1bc785b39871e11fb6ab71b3dR538

(if statement can be dropped)

xpivarc · 2024-01-23T13:08:06Z

tests/clone_test.go

+						Consistently(func(g Gomega) {
+							vmSnapshot, err := virtClient.VirtualMachineSnapshot(vmClone.Namespace).Get(context.Background(), *vmSnapshotName, v1.GetOptions{})
+							g.Expect(err).ShouldNot(HaveOccurred())
+							g.Expect(vmSnapshot).ShouldNot(BeNil())
+							vmRestore, err := virtClient.VirtualMachineRestore(vmClone.Namespace).Get(context.Background(), *vmRestoreName, v1.GetOptions{})
+							g.Expect(err).ShouldNot(HaveOccurred())
+							g.Expect(vmRestore).ShouldNot(BeNil())
+						}, 30*time.Second).Should(Succeed(), "vmsnapshot and vmrestore should not be deleted until the pvc is bound")


I do wonder if we need this. Isn't it enough to check a phase?

Here I want to be sure the vmrestore and the vmsnapshot are not deleted until the pvc is bound (bu run the vm).
We can check the phase, but I should do it Consistenly IMO. WDYT? Thanks

xpivarc · 2024-01-23T13:09:59Z

tests/clone_test.go

+						Eventually(func(g Gomega) {
+							_, err := virtClient.VirtualMachineSnapshot(vmClone.Namespace).Get(context.Background(), *vmSnapshotName, v1.GetOptions{})
+							g.Expect(errors.IsNotFound(err)).Should(BeTrue())
+							_, err = virtClient.VirtualMachineRestore(vmClone.Namespace).Get(context.Background(), *vmRestoreName, v1.GetOptions{})
+							g.Expect(errors.IsNotFound(err)).Should(BeTrue())
+						}, 1*time.Minute).Should(Succeed(), "vmsnapshot and vmrestore should be deleted once the pvc is bound")


This can be replaced by using BeGone but you do provide the description so I think this is enough.

I have several follow-up PRs, I will adjust it later if you agree. Thanks

xpivarc · 2024-01-24T08:28:55Z

/lgtm

kubevirt-commenter-bot · 2024-01-24T08:29:01Z

Required labels detected, running phase 2 presubmits:
/test pull-kubevirt-e2e-kind-1.27-sriov
/test pull-kubevirt-e2e-k8s-1.27-ipv6-sig-network
/test pull-kubevirt-e2e-k8s-1.27-sig-network
/test pull-kubevirt-e2e-k8s-1.27-sig-storage
/test pull-kubevirt-e2e-k8s-1.27-sig-compute
/test pull-kubevirt-e2e-k8s-1.27-sig-operator
/test pull-kubevirt-e2e-k8s-1.28-sig-network
/test pull-kubevirt-e2e-k8s-1.28-sig-storage
/test pull-kubevirt-e2e-k8s-1.28-sig-compute
/test pull-kubevirt-e2e-k8s-1.28-sig-operator

kbidarkar · 2024-01-24T12:54:14Z

/retest-required

acardace · 2024-01-24T19:35:05Z

/retest-required

fossedihelm · 2024-01-24T19:43:01Z

/cherrypick release-1.1

kubevirt-bot · 2024-01-24T19:43:03Z

@fossedihelm: once the present PR merges, I will cherry-pick it on top of release-1.1 in a new PR and assign it to you.

In response to this:

/cherrypick release-1.1

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

fossedihelm · 2024-01-25T04:00:46Z

/retest-required

fossedihelm · 2024-01-25T06:45:04Z

/retest-required

fossedihelm · 2024-01-25T13:12:06Z

/retest-required

acardace · 2024-01-25T16:30:02Z

/retest

brianmcarey · 2024-01-25T17:54:46Z

/override pull-kubevirt-e2e-k8s-1.29-sig-monitoring

This lane was triggered by prow statusreconciler but is not required for this PR

kubevirt-bot · 2024-01-25T17:54:50Z

@brianmcarey: Overrode contexts on behalf of brianmcarey: pull-kubevirt-e2e-k8s-1.29-sig-monitoring

In response to this:

/override pull-kubevirt-e2e-k8s-1.29-sig-monitoring

This lane was triggered by prow statusreconciler but is not required for this PR

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

kubevirt-commenter-bot · 2024-01-25T23:56:05Z

/retest-required
This bot automatically retries required jobs that failed/flaked on approved PRs.
Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

fossedihelm · 2024-01-26T04:43:56Z

/retest-required

kubevirt-bot · 2024-01-26T05:58:46Z

@fossedihelm: new pull request created: #11085

In response to this:

/cherrypick release-1.1

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

kubevirt-bot added release-note-none Denotes a PR that doesn't merit a release note. dco-signoff: yes Indicates the PR's author has DCO signed all their commits. labels Dec 15, 2023

kubevirt-bot requested review from enp0s3 and mhenriks December 15, 2023 09:38

kubevirt-bot added the size/L label Dec 15, 2023

akalenyu reviewed Dec 17, 2023

View reviewed changes

pkg/virt-controller/watch/clone/clone_test.go Show resolved Hide resolved

kubevirt-bot added release-note Denotes a PR that will be considered when it comes time to generate release notes. and removed release-note-none Denotes a PR that doesn't merit a release note. labels Dec 19, 2023

fossedihelm mentioned this pull request Dec 19, 2023

Use fakeclient instead of reactor in vmclone #10908

Merged

8 tasks

kubevirt-bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jan 3, 2024

fossedihelm force-pushed the vmclone_wffc_fix branch from a6852f6 to c16db41 Compare January 3, 2024 08:50

kubevirt-bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jan 3, 2024

enp0s3 reviewed Jan 9, 2024

View reviewed changes

enp0s3 reviewed Jan 10, 2024

View reviewed changes

kubevirt-bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jan 10, 2024

xpivarc reviewed Jan 18, 2024

View reviewed changes

pkg/virt-controller/watch/clone/clone.go Show resolved Hide resolved

fossedihelm added 2 commits January 19, 2024 10:37

fossedihelm force-pushed the vmclone_wffc_fix branch from c16db41 to 40100b8 Compare January 19, 2024 09:37

fossedihelm mentioned this pull request Jan 22, 2024

fix(vmclone) Delete vmclone resource when target vm is deleted #11058

Merged

8 tasks

xpivarc reviewed Jan 23, 2024

View reviewed changes

kubevirt-bot assigned xpivarc Jan 24, 2024

kubevirt-bot added the lgtm Indicates that a PR is ready to be merged. label Jan 24, 2024

kubevirt-bot merged commit 888cbd3 into kubevirt:main Jan 26, 2024
39 of 40 checks passed

kubevirt-bot mentioned this pull request Jan 26, 2024

[release-1.1] fix(vmclone): delete snapshot and restore after pvc bound #11085

Closed

fossedihelm mentioned this pull request Jan 26, 2024

[release-1.1] fix(vmclone): delete snapshot and restore after pvc bound #11087

Merged

8 tasks

chenrui333 mentioned this pull request Mar 6, 2024

virtctl 1.2.0 Homebrew/homebrew-core#165266

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(vmclone): delete snapshot and restore after pvc bound #10888

fix(vmclone): delete snapshot and restore after pvc bound #10888

fossedihelm commented Dec 15, 2023 •

edited

akalenyu left a comment

fossedihelm commented Dec 18, 2023

fossedihelm commented Jan 3, 2024

acardace commented Jan 8, 2024

enp0s3 Jan 9, 2024

fossedihelm Jan 9, 2024

enp0s3 left a comment

kubevirt-bot commented Jan 10, 2024

xpivarc left a comment

fossedihelm commented Jan 19, 2024

kbidarkar commented Jan 23, 2024

xpivarc Jan 23, 2024

fossedihelm Jan 24, 2024

akalenyu Jan 24, 2024

xpivarc Jan 23, 2024

fossedihelm Jan 23, 2024

xpivarc Jan 23, 2024

fossedihelm Jan 23, 2024

xpivarc commented Jan 24, 2024

kubevirt-commenter-bot commented Jan 24, 2024

kbidarkar commented Jan 24, 2024

acardace commented Jan 24, 2024

fossedihelm commented Jan 24, 2024

kubevirt-bot commented Jan 24, 2024

fossedihelm commented Jan 25, 2024

fossedihelm commented Jan 25, 2024

fossedihelm commented Jan 25, 2024

acardace commented Jan 25, 2024

brianmcarey commented Jan 25, 2024

kubevirt-bot commented Jan 25, 2024

kubevirt-commenter-bot commented Jan 25, 2024

fossedihelm commented Jan 26, 2024

kubevirt-bot commented Jan 26, 2024

fix(vmclone): delete snapshot and restore after pvc bound #10888

fix(vmclone): delete snapshot and restore after pvc bound #10888

Conversation

fossedihelm commented Dec 15, 2023 • edited

akalenyu left a comment

Choose a reason for hiding this comment

fossedihelm commented Dec 18, 2023

fossedihelm commented Jan 3, 2024

acardace commented Jan 8, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

enp0s3 left a comment

Choose a reason for hiding this comment

kubevirt-bot commented Jan 10, 2024

xpivarc left a comment

Choose a reason for hiding this comment

fossedihelm commented Jan 19, 2024

kbidarkar commented Jan 23, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xpivarc commented Jan 24, 2024

kubevirt-commenter-bot commented Jan 24, 2024

kbidarkar commented Jan 24, 2024

acardace commented Jan 24, 2024

fossedihelm commented Jan 24, 2024

kubevirt-bot commented Jan 24, 2024

fossedihelm commented Jan 25, 2024

fossedihelm commented Jan 25, 2024

fossedihelm commented Jan 25, 2024

acardace commented Jan 25, 2024

brianmcarey commented Jan 25, 2024

kubevirt-bot commented Jan 25, 2024

kubevirt-commenter-bot commented Jan 25, 2024

fossedihelm commented Jan 26, 2024

kubevirt-bot commented Jan 26, 2024

fossedihelm commented Dec 15, 2023 •

edited