Delete old version DVs with DIC GC #2749

ido106 · 2023-06-12T16:10:04Z

Signed-off-by: Ido Aharon iaharon@redhat.com

What this PR does / why we need it:
In the old versions, the garbage collection label was added to DV only. In the new version, the GC looks for this label in the PVC, so it does not find it and does not delete the DV's from the old versions

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes bz #2209969

Special notes for your reviewer:

Release note:

Delete old version DV's with DIC garbage collector

kubevirt-bot · 2023-06-12T16:10:13Z

Hi @ido106. Thanks for your PR.

I'm waiting for a kubevirt member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

akalenyu · 2023-08-30T15:44:02Z

/ok-to-test

akalenyu · 2023-08-30T15:45:27Z

If there is a bug opened for this, would be great to include it under Fixes #

ido106 · 2023-08-30T16:07:23Z

If there is a bug opened for this, would be great to include it under Fixes #

There is only a Jira card
https://issues.redhat.com/browse/CNV-29106

akalenyu · 2023-08-30T16:19:08Z

If there is a bug opened for this, would be great to include it under Fixes #

There is only a Jira card https://issues.redhat.com/browse/CNV-29106

Yup, if you look at that issue it translates to a bugzilla

ido106 · 2023-08-31T11:17:57Z

If there is a bug opened for this, would be great to include it under Fixes #

There is only a Jira card https://issues.redhat.com/browse/CNV-29106

Yup, if you look at that issue it translates to a bugzilla

Ok, I added the Bugzilla bug number

alromeros · 2023-08-31T14:25:03Z

pkg/controller/dataimportcron-controller.go

+		return err
+	}
+
+	for _, dv := range dvList.Items {


Out of curiosity, why don't we apply here the logic used above?

if len(pvcList.Items) > maxImports { sort.Slice(pvcList.Items, func(i, j int) bool { return pvcList.Items[i].Annotations[AnnLastUseTime] > pvcList.Items[j].Annotations[AnnLastUseTime] }) for _, pvc := range pvcList.Items[maxImports:] {

At first I did that, but then we decided to delete all those DVs because they are all old.
@arnongilboa
Maybe we'll change it back?

Yep, we would like to delete these (very) old ones and no need to sort, but we can sure add

if len(dvList.Items) > maxImports

alromeros · 2023-08-31T14:25:40Z

pkg/controller/dataimportcron-controller.go

+		}
+
+		if pvc.Labels[common.DataImportCronLabel] != cronName {
+			if err := r.deleteDvPvc(ctx, dv.Name, dv.Namespace); err != nil {


Maybe log the deletion of these dvs/pvcs

ido106 · 2023-09-03T11:00:26Z

/retest

ido106 · 2023-09-04T16:30:47Z

The functional test works on my machine. Does someone know what the problem is?

You should be able to reproduce the failures if you start a fresh dev cluster with KUBEVIRT_STORAGE=hpp

Great, thanks !

ido106 · 2023-09-06T13:42:45Z

Is 7406 cool now ?
@akalenyu

ido106 · 2023-09-07T08:35:04Z

/retest

akalenyu

Looks good just not sure if you meant to keep the separate test

akalenyu · 2023-09-07T08:53:03Z

tests/dataimportcron_test.go

@@ -367,6 +369,120 @@ var _ = Describe("DataImportCron", func() {
 		Entry("[test_id:8266] succeed deleting error DVs when importing new ones", false, true, 2, cdiv1.DataImportCronSourceFormatPvc),
 	)

+	It("[test_id:XXXX] Succeed garbage collecting old version DVs", func() {


We can delete this test now that we are doing the check in 7406

Yup, I was just waiting to see if 7406 is good before deleting the old test

akalenyu · 2023-09-07T08:58:28Z

tests/dataimportcron_test.go

+			Eventually(func() bool {
+				_, err := f.CdiClient.CdiV1beta1().DataVolumes(ns).Get(context.TODO(), oldDvName, metav1.GetOptions{})
+				return err != nil
+			}, dataImportCronTimeout, pollingInterval).Should(Equal(true), "Garbage collection failed cleaning old DV")


Just a nit, if you want, can do something like this:

Eventually(func() error { _, err := f.CdiClient.CdiV1beta1().DataVolumes(ns).Get(context.TODO(), oldDvName, metav1.GetOptions{}) return err }, dataImportCronTimeout, pollingInterval).Should(Satisfy(errors.IsNotFound), "Garbage collection failed cleaning old DV")

Great, I will fix it

akalenyu · 2023-09-07T13:36:43Z

/approve

akalenyu · 2023-09-07T13:36:52Z

/cc @arnongilboa

kubevirt-bot · 2023-09-07T13:37:29Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: akalenyu

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [akalenyu]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

arnongilboa · 2023-09-10T09:14:38Z

pkg/controller/dataimportcron-controller.go

+			}
+
+			if pvc.Labels[common.DataImportCronLabel] != cronName {
+				r.log.Info("Deleting dv/pvc", "name", pvc.Name, "pvc.uid", pvc.UID)


I think this log line should be a bit different from the one above, e.g. "Deleting old version dv/pvc"?

arnongilboa · 2023-09-10T09:17:02Z

pkg/controller/dataimportcron-controller_test.go

+			cron = newDataImportCron(cronName)
+			importsToKeep := int32(1)
+			cron.Spec.ImportsToKeep = &importsToKeep
+			cron.Annotations[AnnSourceDesiredDigest] = testDigest


why do you need to set digest?

You're right, we don't need it. Probably left over from the old test I wrote.
Fixed

arnongilboa · 2023-09-10T09:19:28Z

tests/dataimportcron_test.go

@@ -48,6 +48,7 @@ var _ = Describe("DataImportCron", func() {
 		ns                  string
 		scName              string
 		originalProfileSpec *cdiv1.StorageProfileSpec
+		oldDvName           = "src-garbage-0"


oldDvName should move into the function, as nobody else is using it

arnongilboa · 2023-09-10T09:27:24Z

tests/dataimportcron_test.go

+		for i := 1; i <= garbageSources; i++ {
 			srcName := fmt.Sprintf("src-garbage-%d", i)


Not sure I like this naming hack, oldDvName should have a better name than "src-garbage-0", e.g. "old-version-dv", and the loop can be reverted to 0-based.

Sure, no problem. Fixed

Signed-off-by: Ido Aharon <iaharon@redhat.com>

ido106 · 2023-09-10T16:33:23Z

Do you have any more comments?
@arnongilboa

arnongilboa · 2023-09-11T12:52:34Z

/lgtm

Manual backport of kubevirt#2749 Signed-off-by: Ido Aharon <iaharon@redhat.com>

Manual backport of #2749 Signed-off-by: Ido Aharon <iaharon@redhat.com> Co-authored-by: Ido Aharon <iaharon@redhat.com>

kubevirt-bot added the size/L label Jun 12, 2023

kubevirt-bot requested review from awels and mhenriks June 12, 2023 16:10

ido106 force-pushed the garbage_collect_old_dvs_dic branch from 4edfb75 to a12b74a Compare June 12, 2023 16:11

kubevirt-bot added dco-signoff: yes Indicates the PR's author has DCO signed all their commits. and removed dco-signoff: no Indicates the PR's author has not DCO signed all their commits. labels Jun 12, 2023

ido106 force-pushed the garbage_collect_old_dvs_dic branch from a12b74a to 804ceba Compare August 21, 2023 08:37

kubevirt-bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Aug 21, 2023

ido106 force-pushed the garbage_collect_old_dvs_dic branch 3 times, most recently from 22c28e3 to 8faf5e4 Compare August 24, 2023 14:58

ido106 changed the title ~~[WIP] Delete old version DV's with DIC GC~~ Delete old version DV's with DIC GC Aug 24, 2023

kubevirt-bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Aug 24, 2023

kubevirt-bot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Aug 30, 2023

alromeros reviewed Aug 31, 2023

View reviewed changes

ido106 force-pushed the garbage_collect_old_dvs_dic branch 2 times, most recently from 9796b25 to 91a73d7 Compare September 3, 2023 10:49

ido106 force-pushed the garbage_collect_old_dvs_dic branch 3 times, most recently from 2135092 to 00cfb22 Compare September 6, 2023 13:41

ido106 changed the title ~~Delete old version DV's with DIC GC~~ Delete old version DVs with DIC GC Sep 7, 2023

akalenyu reviewed Sep 7, 2023

View reviewed changes

ido106 force-pushed the garbage_collect_old_dvs_dic branch 2 times, most recently from f635c1a to fc7884f Compare September 7, 2023 09:48

kubevirt-bot requested a review from arnongilboa September 7, 2023 13:36

kubevirt-bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Sep 7, 2023

arnongilboa reviewed Sep 10, 2023

View reviewed changes

ido106 force-pushed the garbage_collect_old_dvs_dic branch from fc7884f to 8d42850 Compare September 10, 2023 16:19

Delete old version DV's with DIC GC

795c34a

Signed-off-by: Ido Aharon <iaharon@redhat.com>

ido106 force-pushed the garbage_collect_old_dvs_dic branch from 8d42850 to 795c34a Compare September 10, 2023 16:31

kubevirt-bot assigned arnongilboa Sep 11, 2023

kubevirt-bot added the lgtm Indicates that a PR is ready to be merged. label Sep 11, 2023

kubevirt-bot merged commit 5b68719 into kubevirt:main Sep 11, 2023
18 checks passed

arnongilboa pushed a commit to arnongilboa/containerized-data-importer that referenced this pull request Sep 28, 2023

[release-v1.57] Delete old version DVs with DIC GC

d13ce31

Manual backport of kubevirt#2749 Signed-off-by: Ido Aharon <iaharon@redhat.com>

arnongilboa mentioned this pull request Sep 28, 2023

[release-v1.57] Delete old version DVs with DIC GC #2911

Merged

kubevirt-bot pushed a commit that referenced this pull request Oct 4, 2023

[release-v1.57] Delete old version DVs with DIC GC (#2911)

5616e91

Manual backport of #2749 Signed-off-by: Ido Aharon <iaharon@redhat.com> Co-authored-by: Ido Aharon <iaharon@redhat.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Delete old version DVs with DIC GC #2749

Delete old version DVs with DIC GC #2749

ido106 commented Jun 12, 2023 •

edited

kubevirt-bot commented Jun 12, 2023

akalenyu commented Aug 30, 2023

akalenyu commented Aug 30, 2023

ido106 commented Aug 30, 2023

akalenyu commented Aug 30, 2023

ido106 commented Aug 31, 2023

alromeros Aug 31, 2023

ido106 Aug 31, 2023

arnongilboa Sep 3, 2023

ido106 Sep 3, 2023

alromeros Aug 31, 2023

ido106 Aug 31, 2023

ido106 commented Sep 3, 2023

ido106 commented Sep 4, 2023

ido106 commented Sep 6, 2023

ido106 commented Sep 7, 2023

akalenyu left a comment

akalenyu Sep 7, 2023

ido106 Sep 7, 2023 •

edited

akalenyu Sep 7, 2023 •

edited

ido106 Sep 7, 2023

akalenyu commented Sep 7, 2023

akalenyu commented Sep 7, 2023

kubevirt-bot commented Sep 7, 2023

arnongilboa Sep 10, 2023

ido106 Sep 10, 2023

arnongilboa Sep 10, 2023

ido106 Sep 10, 2023

arnongilboa Sep 10, 2023

ido106 Sep 10, 2023

arnongilboa Sep 10, 2023

ido106 Sep 10, 2023

ido106 commented Sep 10, 2023

arnongilboa commented Sep 11, 2023

		for i := 1; i <= garbageSources; i++ {
		srcName := fmt.Sprintf("src-garbage-%d", i)

Delete old version DVs with DIC GC #2749

Delete old version DVs with DIC GC #2749

Conversation

ido106 commented Jun 12, 2023 • edited

kubevirt-bot commented Jun 12, 2023

akalenyu commented Aug 30, 2023

akalenyu commented Aug 30, 2023

ido106 commented Aug 30, 2023

akalenyu commented Aug 30, 2023

ido106 commented Aug 31, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ido106 commented Sep 3, 2023

ido106 commented Sep 4, 2023

ido106 commented Sep 6, 2023

ido106 commented Sep 7, 2023

akalenyu left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ido106 Sep 7, 2023 • edited

Choose a reason for hiding this comment

akalenyu Sep 7, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

akalenyu commented Sep 7, 2023

akalenyu commented Sep 7, 2023

kubevirt-bot commented Sep 7, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ido106 commented Sep 10, 2023

arnongilboa commented Sep 11, 2023

ido106 commented Jun 12, 2023 •

edited

ido106 Sep 7, 2023 •

edited

akalenyu Sep 7, 2023 •

edited