CI: Back up and restore a VM. #1333

mrnold · 2024-02-15T19:27:56Z

This pull request adds an actual backup and restore of a running VM. This includes some rework of the existing backup and restore test code, so that different parts of the sequence could be specialized for VM purposes. The result is a little bit awkward, but it makes almost no functional changes to existing tests.

kaovilai · 2024-02-16T01:33:26Z

/hold

re:Do not merge this yet

mrnold · 2024-02-17T22:01:47Z

/retest

mrnold · 2024-02-18T02:21:23Z

/retest

mrnold · 2024-02-18T02:27:36Z

Okay, the VM backup and restore test has worked successfully at least once. This is at least worth having @mateusoliveira43 and @mpryc take a look at it.

mpryc · 2024-02-18T12:40:47Z

/test 4.13-e2e-test-aws

mrnold · 2024-02-18T18:00:16Z

Looks like failures in "Mongo application DATAMOVER" instead of "MySQL application two Vol CSI" this time, maybe one more retest for entertainment.

mrnold · 2024-02-18T18:00:22Z

/retest

mateusoliveira43 · 2024-02-20T17:29:29Z

@weshayutin this resolves https://issues.redhat.com/browse/OADP-2982 ?

weshayutin · 2024-02-21T16:00:02Z

@mateusoliveira43 it only partially resolves OADP-2982. I'll review how to associate a pr w/ a jira again w/ @mrnold . It's a just a jira task and for CI.

mateusoliveira43 · 2024-02-21T19:30:26Z

tests/e2e/lib/backup.go

@@ -24,6 +25,9 @@ func CreateBackupForNamespaces(ocClient client.Client, veleroNamespace, backupNa
 			SnapshotMoveData:         &snapshotMoveData,
 		},
 	}
+	if snapshotVolumes {
+		backup.Spec.SnapshotVolumes = pointer.Bool(true)


maintainers, I never used this flag. This is needed if the PV is not in the same namespace as one of in IncludedNamespaces?

PVs are cluster-scoped and are not in any namespace. PVCs that are not in an included namespace will be excluded from backup.
This flag tells Velero to snapshot PVCs that are 1) included in the backup, and 2) not backed up using fs-backup. If this is false, then non-fs-backup PVCs will not be snapshotted.

mateusoliveira43 · 2024-02-21T19:38:27Z

tests/e2e/lib/restore.go

@@ -22,6 +23,9 @@ func CreateRestoreFromBackup(ocClient client.Client, veleroNamespace, backupName
 			BackupName: backupName,
 		},
 	}
+	if restorePVs {
+		restore.Spec.RestorePVs = pointer.Bool(true)


maintainers, what this option does with the restore?

If I'm remembering correctly, this the restore equivalent to snapshotVolumes -- @shubham-pampattiwar is this correct?
Assuming I'm not misremembering, this means, similar to above, if it's true then we restore the PVC content from snapshot if it is a volume that was backed up via snapshot.

RestorePVs specifies whether to restore all included PVs from snapshot

The snapshot can be cloud provider based or csi snapshot. Both are applicable.

@shubham-pampattiwar It's confusing that the restore and backup flags aren't named similarly, but they are essentially equilalent flags, but one for backup, the other for restore.

The snapshot can be cloud provider based or csi snapshot. Both are applicable.

Same with SnapshotVolumes. There was a fix a while back to make this flag do the same thing for CSI and native snapshots.

yes since velero 1.11 I think, initially these 2 flags were only applicable for cloud provider based snapshots.

mateusoliveira43 · 2024-02-21T19:42:16Z

tests/e2e/backup_restore_suite_test.go

I would suggest breaking this PR in 2

add backup/restore virt

remove duplications from backup/restore and must-gather tests

Okay, I have started this process with #1345.

mrnold · 2024-02-26T21:55:53Z

/hold

Add new VM tests, and work through some extra VM details like removing the DataVolume (but not the PVC) before backup. This still fails because the restore is PartiallyFailed, need to add test-specific validation. Signed-off-by: Matthew Arnold <marnold@redhat.com>

Signed-off-by: Matthew Arnold <marnold@redhat.com>

Add some retries in places that might hit "the object was modified" errors, and other small tweaks. Signed-off-by: Matthew Arnold <marnold@redhat.com>

Trying to addres "DataVolume not found" errors from backup. Signed-off-by: Matthew Arnold <marnold@redhat.com>

Signed-off-by: Matthew Arnold <marnold@redhat.com>

mrnold · 2024-03-04T22:31:52Z

/unhold

shubham-pampattiwar · 2024-03-05T03:07:17Z

/retest

shubham-pampattiwar · 2024-03-05T05:59:25Z

/retest

mrnold · 2024-03-05T14:14:30Z

/test 4.12-e2e-test-aws

weshayutin

/LGTM

openshift-ci · 2024-03-05T18:04:57Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: mrnold, weshayutin

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~tests/e2e/OWNERS~~ [mrnold,weshayutin]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

kaovilai · 2024-03-05T18:07:52Z

/override ci/prow/4.12-e2e-test-aws

ya.. the change is for a completely diff test

:)
ps. we've seen this flake b4

openshift-ci · 2024-03-05T18:09:44Z

@kaovilai: Overrode contexts on behalf of kaovilai: ci/prow/4.12-e2e-test-aws

In response to this:

/override ci/prow/4.12-e2e-test-aws

ya.. the change is for a completely diff test

:)
ps. we've seen this flake b4

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci · 2024-03-05T18:09:48Z

@mrnold: all tests passed!

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

* Back up and restore a VM. Add new VM tests, and work through some extra VM details like removing the DataVolume (but not the PVC) before backup. This still fails because the restore is PartiallyFailed, need to add test-specific validation. Signed-off-by: Matthew Arnold <marnold@redhat.com> * Make log checking optional. Signed-off-by: Matthew Arnold <marnold@redhat.com> * Include PV in VM backup and restore. Signed-off-by: Matthew Arnold <marnold@redhat.com> * Attempt to improve VM test reliability. Add some retries in places that might hit "the object was modified" errors, and other small tweaks. Signed-off-by: Matthew Arnold <marnold@redhat.com> * Wait for PVC owner reference to go away. Trying to addres "DataVolume not found" errors from backup. Signed-off-by: Matthew Arnold <marnold@redhat.com> * Try a readiness delay on VM backup. Signed-off-by: Matthew Arnold <marnold@redhat.com> * Try booting VM from PVC instead of DataVolume. Signed-off-by: Matthew Arnold <marnold@redhat.com> * Make verification hooks really optional. Signed-off-by: Matthew Arnold <marnold@redhat.com> --------- Signed-off-by: Matthew Arnold <marnold@redhat.com>

openshift-ci bot requested review from eemcmullan and savitharaghunathan February 15, 2024 19:28

openshift-ci bot added do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. approved Indicates a PR has been approved by an approver from all required OWNERS files. labels Feb 16, 2024

mateusoliveira43 added the CI label Feb 16, 2024

mrnold changed the title ~~Do not merge: CI: Back up and restore a VM.~~ CI: Back up and restore a VM. Feb 18, 2024

weshayutin removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Feb 19, 2024

mateusoliveira43 reviewed Feb 21, 2024

View reviewed changes

mrnold mentioned this pull request Feb 26, 2024

CI: Split up E2E test's runBackupAndRestore in preparation for virtualization tests. #1345

Merged

openshift-ci bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Feb 26, 2024

openshift-merge-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Mar 1, 2024

mrnold added 7 commits March 4, 2024 16:17

Make log checking optional.

d8c6759

Signed-off-by: Matthew Arnold <marnold@redhat.com>

Include PV in VM backup and restore.

3e34b1b

Signed-off-by: Matthew Arnold <marnold@redhat.com>

Attempt to improve VM test reliability.

fe935df

Add some retries in places that might hit "the object was modified" errors, and other small tweaks. Signed-off-by: Matthew Arnold <marnold@redhat.com>

Wait for PVC owner reference to go away.

1beba27

Trying to addres "DataVolume not found" errors from backup. Signed-off-by: Matthew Arnold <marnold@redhat.com>

Try a readiness delay on VM backup.

6eaa264

Signed-off-by: Matthew Arnold <marnold@redhat.com>

Try booting VM from PVC instead of DataVolume.

b248f85

Signed-off-by: Matthew Arnold <marnold@redhat.com>

Make verification hooks really optional.

14d18b4

Signed-off-by: Matthew Arnold <marnold@redhat.com>

mrnold force-pushed the ci-virt-backup-restore branch from 6cf3641 to 14d18b4 Compare March 4, 2024 22:31

openshift-merge-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Mar 4, 2024

openshift-ci bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Mar 4, 2024

weshayutin approved these changes Mar 5, 2024

View reviewed changes

openshift-ci bot assigned weshayutin Mar 5, 2024

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Mar 5, 2024

openshift-merge-bot bot merged commit bf64895 into openshift:master Mar 5, 2024
16 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CI: Back up and restore a VM. #1333

CI: Back up and restore a VM. #1333

mrnold commented Feb 15, 2024 •

edited

Loading

kaovilai commented Feb 16, 2024

mrnold commented Feb 17, 2024

mrnold commented Feb 18, 2024

mrnold commented Feb 18, 2024

mpryc commented Feb 18, 2024

mrnold commented Feb 18, 2024

mrnold commented Feb 18, 2024

mateusoliveira43 commented Feb 20, 2024

weshayutin commented Feb 21, 2024

mateusoliveira43 Feb 21, 2024

sseago Feb 22, 2024

mateusoliveira43 Feb 21, 2024

sseago Feb 22, 2024

shubham-pampattiwar Feb 22, 2024

shubham-pampattiwar Feb 22, 2024

shubham-pampattiwar Feb 22, 2024

sseago Feb 22, 2024

sseago Feb 22, 2024

shubham-pampattiwar Feb 22, 2024 •

edited

Loading

mateusoliveira43 Feb 21, 2024

mrnold Feb 26, 2024

mrnold commented Feb 26, 2024

mrnold commented Mar 4, 2024

shubham-pampattiwar commented Mar 5, 2024

shubham-pampattiwar commented Mar 5, 2024

mrnold commented Mar 5, 2024

weshayutin left a comment

openshift-ci bot commented Mar 5, 2024

kaovilai commented Mar 5, 2024

openshift-ci bot commented Mar 5, 2024

openshift-ci bot commented Mar 5, 2024

CI: Back up and restore a VM. #1333

CI: Back up and restore a VM. #1333

Conversation

mrnold commented Feb 15, 2024 • edited Loading

kaovilai commented Feb 16, 2024

mrnold commented Feb 17, 2024

mrnold commented Feb 18, 2024

mrnold commented Feb 18, 2024

mpryc commented Feb 18, 2024

mrnold commented Feb 18, 2024

mrnold commented Feb 18, 2024

mateusoliveira43 commented Feb 20, 2024

weshayutin commented Feb 21, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shubham-pampattiwar Feb 22, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mrnold commented Feb 26, 2024

mrnold commented Mar 4, 2024

shubham-pampattiwar commented Mar 5, 2024

shubham-pampattiwar commented Mar 5, 2024

mrnold commented Mar 5, 2024

weshayutin left a comment

Choose a reason for hiding this comment

openshift-ci bot commented Mar 5, 2024

kaovilai commented Mar 5, 2024

openshift-ci bot commented Mar 5, 2024

openshift-ci bot commented Mar 5, 2024

mrnold commented Feb 15, 2024 •

edited

Loading

shubham-pampattiwar Feb 22, 2024 •

edited

Loading