Add part1 manual test case from v1.3.0 validated tickets. #1217

TachunLin · 2024-04-10T16:14:54Z

Add the part 1 of manual test cases from v1.3.0 validated tickets including:

#4611 [ENHANCEMENT] Check free disk space percent before upgrade
#4604 [BUG] Restore from snapshot not work if target VM is restore-replaced from backup
#4588 [BUG] Node pool anti-affinity rules are not working as expected
#4480 [ENHANCEMENT] config sftp dynamically
#4954 [BUG] VMRestore to new VM doesn't work if there is VMSnapshot
#4625 [ENHANCEMENT] Add webhook for CRD virtualmachinetemplateversion.harvesterhci.io
#4678 [BUG] Unable to successfully create LoadBalancer service from Guest cluster post 1.2.1 upgrade
#4907 [ENHANCEMENT] Take advantage of the attach/detach mechanism in Longhorn v1.5+
#4535 [BUG] Sending ctrl-alt-delete does not have any reaction on a windows VM
#4725 [BUG] Upgrade 1.2.0 -> 1.2.1 is stuck in “Waiting for VM live-migration or shutdown...(1 left)” even though there is NO VM run

Tested on the local hugo server preview content.

albinsun · 2024-04-11T03:11:13Z

docs/content/manual/deployment/4480-config-sftp-dynamically.md

+* Deployment
+
+## Verification Steps
+1. Use ipex-example to provision Harvester


typo, should be ipxe.

Thanks for the check.
Correct the repo in line 12.

albinsun · 2024-04-11T03:14:33Z

docs/content/manual/harvester-rancher/4678-load-balancer-different-roles-pool.md

+    ```
+    write_files:
+    - encoding: b64
+    content: {harvester's kube config, the cluster namespace should be same as the pool you created (base64 enconded)}


enconded -> encoded

Thanks for the check.
Correct the repo in line 21.

albinsun · 2024-04-11T03:15:30Z

docs/content/manual/harvester-rancher/4678-load-balancer-different-roles-pool.md

+1. Copy the base64 encoded kubeconfig to the cloud config write file sections above
+{{< image "images/rancher/4678-base64-kubeconfig.png" >}}
+1. Provision the RKE2 guest cluster
+1. After pools are created, we remove the harvester-cloud-provider in Apps > Installed Apps (kube-system namesapce).


namesapce -> namespace

Thanks for the check.
Correct the repo in line 42.

albinsun · 2024-04-11T03:16:16Z

docs/content/manual/harvester-rancher/4678-load-balancer-different-roles-pool.md

+1. Create the first pool, specif the `control plane` role only
+1. Create the second pool, specify the `etcd` role only
+1. Create the third pool, specify the `worker` role only
+1. Repeat the steps 13 - 17 to create load blancer service


blancer -> balancer

Thanks for the check.
Correct the repo in line 58.

albinsun · 2024-04-11T03:17:21Z

docs/content/manual/harvester-rancher/4678-load-balancer-different-roles-pool.md

+
+## Expected Results
+*  Control-plane, ETCD and worker in same pool: 
+    - Can sucessfully create LoadBalancer service on RKE2 guest cluster


sucessfully -> successfully

Thanks for the check.
Correct the repo in line 67

albinsun · 2024-04-11T03:19:08Z

docs/content/manual/virtual-machines/4588-node-pool-anti-affinity.md

+1. Each VM launched will have an additional label added.
+1. The anti-affinity rule on the VM will leverage this label to ensure VM's are spread across in the cluster
+
+1. Corden one of the node on Harvester host page


Corden -> Cordon

Thanks for the check.
Correct the repo in line 27.

albinsun · 2024-04-11T03:19:39Z

docs/content/manual/virtual-machines/4588-node-pool-anti-affinity.md

+* The anti-affinity rule set on the label of vm in the yaml content 
+{{< image "images/virtual-machines/4588-yaml-anti-affinity.png" >}}
+
+* If Node 2 cordened, none of the VM would be scheduled on that node


Thanks for the check.
Correct the repo in line 38.

albinsun · 2024-04-11T03:28:12Z

docs/content/manual/backup-and-restore/4907-vm-backup-restore-longhorn-1.5-plus.md

+          node1:~ # kubectl get volume -A
+          NAMESPACE         NAME                                       STATE      ROBUSTNESS   SCHEDULED   SIZE          NODE   AGE
+          longhorn-system   pvc-5a861225-920d-4059-b501-f02b2fd0ff27   detached   unknown                  10737418240          19m
+      ```


Decrease an indent.

Current

node1:~ # kubec NAMESPACE ...

Should be

node1:~ # kubec NAMESPACE

Thanks for the check.
Decrease the indent to align the format.

albinsun · 2024-04-11T03:28:14Z

docs/content/manual/backup-and-restore/4907-vm-backup-restore-longhorn-1.5-plus.md

+      ```
+          NAMESPACE         NAME                                       STATE      ROBUSTNESS   SCHEDULED   SIZE          NODE   AGE
+          longhorn-system   pvc-d1226d97-ab90-4d40-92f9-960b668093c2   detached   unknown                  10737418240          5m12s
+      ```


Thanks for the check.
Decrease the indent to align the format.

albinsun · 2024-04-11T03:30:00Z

docs/content/manual/backup-and-restore/4954-cannot-resotre-new-vm-have-snapshot.md

+1. Create a snapshot for the vm1
+1. Restore the backup of vm1 to create a new VM
+1. Check can restore vm correctly
+1. Shutdown vm1,


should not end with comma.

Thanks for the check.
Remove the comma in line 19.

khushboo-rancher

Thanks @TachunLin for documenting all the test cases, can you take a look at the questions I have?

docs/content/manual/backup-and-restore/4604-restore-snapshot-restored-from-backup.md

khushboo-rancher · 2024-04-10T19:55:32Z

docs/content/manual/backup-and-restore/4907-vm-backup-restore-longhorn-1.5-plus.md

+1. Create a VM.
+1. After the VM is ready, stop the VM.
+1. Check VM volumes are detached.
+1. Take a snapshot on the VM. The snapshot can be ready.


Do we have this in automated test already?

I went through our existing backend integration test scenarios of:

test_1_volumes.py

test_4_vm_backup_restore.py

test_4_vm_snapshot.py

I found the following test plan in test_4_vm_snapshot.py may covered in the test_create_vm_snapshot_while_pvc_detached

tests/harvester_e2e_tests/integrations/test_4_vm_snapshot.py

Lines 457 to 487 in 790c494

def test_create_vm_snapshot_while_pvc_detached(self, api_client,

vm_snapshot_2_name, source_vm, wait_timeout):

"""

Test that a VM snapshot can be created when the source

PVC is detached.

Prerequisites:

The original VM (`source-vm`) exists and is stopped (so that

the PVC is detached.)

"""

name, _ = source_vm

stop_vm(name, api_client, wait_timeout)

code, _ = api_client.vm_snapshots.create(name, vm_snapshot_2_name)

assert 201 == code

deadline = datetime.now() + timedelta(seconds=wait_timeout)

while deadline > datetime.now():

code, data = api_client.vm_snapshots.get(vm_snapshot_2_name)

if data.get("status", {}).get("readyToUse"):

break

print(f"waiting for {vm_snapshot_2_name} to be ready")

sleep(3)

else:

raise AssertionError(f"timed out waiting for {vm_snapshot_2_name} to be ready")

code, data = api_client.vm_snapshots.get(vm_snapshot_2_name)

assert 200 == code

assert data.get("status", {}).get("readyToUse") is True

Update the test plan, remove the lines that have been covered in the automation e2e test.
And replace to indicate which e2e test covered this line.

khushboo-rancher · 2024-04-11T18:03:23Z

docs/content/manual/backup-and-restore/4907-vm-backup-restore-longhorn-1.5-plus.md

+1. Restore the snapshot to a new VM. The new VM can be ready.
+1. Restore the snapshot to replace the old VM. The old VM can be ready.
+
+Case 3: backup can work on a stopped VM


Same as above, can you check if we have this in automated test?

For Case 2:
I found the following test plan in test_4_vm_snapshot.py integration test may covered in the test test_restore_from_vm_snapshot_while_pvc_detached_from_source

tests/harvester_e2e_tests/integrations/test_4_vm_snapshot.py

Lines 425 to 453 in 790c494

def test_restore_from_vm_snapshot_while_pvc_detached_from_source(self,

api_client,

restored_vm_2,

host_shell,

vm_shell,

ssh_keypair,

wait_timeout):

"""

Test that a new virtual machine can be created from a

VM snapshot created from a source PersistentVolumeClaim

that is now detached.

Prerequisites:

The original VM (`source-vm`) exists and is stopped (so that

the PVC is detached.)

The original snapshot (`vm-snapshot`) exists.

"""

name, ssh_user = restored_vm_2

def actassert(sh):

out, _ = sh.exec_command("cat test.txt")

assert "123" in out

vm_shell_do(name, api_client,

host_shell, vm_shell,

ssh_user, ssh_keypair,

actassert, wait_timeout)

Update the test plan, remove the lines that have been covered in the automation e2e test.
And replace to indicate which e2e test covered this line.

khushboo-rancher · 2024-04-11T18:05:51Z

docs/content/manual/backup-and-restore/4954-cannot-resotre-new-vm-have-snapshot.md

+## Verification Steps
+1. Prepare a 3 nodes Harvester cluster
+1. Create a vm named vm1
+1. Setup nfs backup target
+1. Create a backup for the vm1
+1. Create a snapshot for the vm1
+1. Restore the backup of vm1 to create a new VM
+1. Check can restore vm correctly
+1. Shutdown vm1
+1. Restore the backup of vm to replace the existing vm
+1. Select Retain volume
+1. Check can restore vm correctly


Could you check if this is covered in automated test already?

By checking our existing vm backup and snapshot integration test scenarios in:

test_4_vm_backup_restore.py

test_4_vm_snapshot.py

I did not found a suitable automation test could both cover when vm already have the snapshot and backup on it.

Among these tests

test_4_vm_backup_restore.py

test_connection

tests_backup_vm

test_update_backup_by_yaml

test_restore_with_new_vm

test_restore_replace_with_delete_vols

test_restore_replace_vm_not_stop

test_restore_with_invalid_name

test_backup_migrated_vmtest_restore_replace_migrated_vm

test_backup_multiple

test_delete_last_backup

test_delete_middle_backup

test_4_vm_snapshot.py

test_vm_snapshot_create

test_restore_into_new_vm_from_vm_snapshot

test_replace_is_rejected_when_deletepolicy_is_retain

test_replace_vm_with_vm_snapshot

test_restore_from_vm_snapshot_while_pvc_detached_from_source

test_create_vm_snapshot_while_pvc_detached

test_vm_snapshots_are_cleaned_up_after_source_vm_deleted

test_volume_snapshots_are_cleaned_up_after_source_volume_deleted

khushboo-rancher · 2024-04-11T18:08:48Z

docs/content/manual/harvester-rancher/4678-load-balancer-different-roles-pool.md

+1. Prepare three nodes Harvester and import with Rancher
+1. Provision a RKE2 guest cluster in Rancher
+1. Create the only one pool, specif the control plane, etcd and workers roles


This is already covered in automation test.

After checking and confirmed with team, our existing backend e2e test of rancher integration in test_9_rancher_integration.py:

There we provide the ability to create 3 nodes guest cluster

tests/harvester_e2e_tests/integrations/test_9_rancher_integration.py

Line 152 in 3d21068

pytest.param(3, marks=pytest.mark.skip(reason="Skip for low I/O env."))])

In actual testing helped by Albin, the 3 nodes downstream cluster are created in the same pool and all set to the ALL role.

Thus we can assume this manual test plan to have different roles in different pools did not covered by e2e test.

khushboo-rancher · 2024-04-11T18:09:35Z

docs/content/manual/harvester-rancher/4678-load-balancer-different-roles-pool.md

+1. After pools are created, we remove the harvester-cloud-provider in Apps > Installed Apps (kube-system namespace).
+1. Add new charts in Apps > Repositories, use https://charts.harvesterhci.io to install and select 0.2.3.
+{{< image "images/rancher/4678-add-harvester-repo.png" >}}
+1. Install Harvester cloud provider 0.2.3 from market


Install Harvester cloud provider 0.2.3 from Apps & marketplace

Thanks for the check.
Update to the correct term in this line.

khushboo-rancher · 2024-04-11T18:15:31Z

docs/content/manual/harvester-rancher/4678-load-balancer-different-roles-pool.md

+1. After pools are created, we remove the harvester-cloud-provider in Apps > Installed Apps (kube-system namespace).
+1. Add new charts in Apps > Repositories, use https://charts.harvesterhci.io to install and select 0.2.3.


To check a particular version of chart, removing the default and installing from Apps is fine but in general, this should not be practiced. As the bundled chart might get rolled back to the manifest shipped with guest cluster. So, I think adding this as generalized test case is not required.

Thanks for the suggestion.

That's true, the tester should confirm the cloud provider default provided in the RKE2 guest cluster at least to have 0.2.3 to include this changes.

These steps here are not necessary since these steps are just for the time when the bundled chart is not ready during the issue verification stage.

We can remove these lines and the related static screenshot.
Then add the check step to ensure the desired cloud provider version

khushboo-rancher · 2024-04-11T18:16:45Z

docs/content/manual/harvester-rancher/4678-load-balancer-different-roles-pool.md

+1. Create the first pool, specify the `etcd` and `control plane` roles
+1. Create the second pool, specify the `work` role only


@albin, do we have a test case with 2 pool created while testing guest cluster?

After checking and helped by Albin, currently in the backend e2e test in test_9_rancher_integration.py
We provide creating 3 nodes in the same roles ALL and in the same pool.

Thus it seems the manual test is somehow differ with the backend e2e Rancher integration test.

khushboo-rancher · 2024-04-11T18:19:09Z

docs/content/manual/upgrade/4725-upgrade-shutdown-vm-in-os.md

@@ -0,0 +1,22 @@
+---
+title: Upgrade with VM shutdown in the operating system 


This is being taken care in upgrade automation test.

In my understating, our existing upgrade backend e2e test will shutdown VM before starting upgrade.
This manual test have a little bit difference with e2e test since in this test we shudown VM by accessing into the operating system while the e2e test shutdown VM from the virtual machine related API call.

1. Login to Windows VM and shutdown machine from menu 1. Login to openSUSE VM , use command to shutdown machine

Thus I am thinking whether we can keep this test plan since there operations are somehow different.

Add manual test plan for v1.3.0 validated tickets.

30a29f6

TachunLin added documentation Improvements or additions to documentation area/manual-test labels Apr 10, 2024

TachunLin assigned albinsun, lanfon72, irishgordo, khushboo-rancher and noahgildersleeve and unassigned albinsun, lanfon72, irishgordo, khushboo-rancher and noahgildersleeve Apr 10, 2024

TachunLin requested review from albinsun, lanfon72, irishgordo, khushboo-rancher and noahgildersleeve April 10, 2024 16:15

albinsun requested changes Apr 11, 2024

View reviewed changes

Update typo and decrease indent of code block

b5f10d4

khushboo-rancher requested changes Apr 11, 2024

View reviewed changes

TachunLin added 3 commits April 13, 2024 00:16

Update for the review request

7a9c1c5

Update one test plan with Gherkin format.

473953a

Update the test plan steps already covered in e2e automation

b150ccd

	def test_create_vm_snapshot_while_pvc_detached(self, api_client,
	vm_snapshot_2_name, source_vm, wait_timeout):
	"""
	Test that a VM snapshot can be created when the source
	PVC is detached.

	Prerequisites:
	The original VM (`source-vm`) exists and is stopped (so that
	the PVC is detached.)
	"""
	name, _ = source_vm

	stop_vm(name, api_client, wait_timeout)

	code, _ = api_client.vm_snapshots.create(name, vm_snapshot_2_name)
	assert 201 == code

	deadline = datetime.now() + timedelta(seconds=wait_timeout)
	while deadline > datetime.now():
	code, data = api_client.vm_snapshots.get(vm_snapshot_2_name)
	if data.get("status", {}).get("readyToUse"):
	break
	print(f"waiting for {vm_snapshot_2_name} to be ready")
	sleep(3)
	else:
	raise AssertionError(f"timed out waiting for {vm_snapshot_2_name} to be ready")

	code, data = api_client.vm_snapshots.get(vm_snapshot_2_name)

	assert 200 == code
	assert data.get("status", {}).get("readyToUse") is True

	def test_restore_from_vm_snapshot_while_pvc_detached_from_source(self,
	api_client,
	restored_vm_2,
	host_shell,
	vm_shell,
	ssh_keypair,
	wait_timeout):
	"""
	Test that a new virtual machine can be created from a
	VM snapshot created from a source PersistentVolumeClaim
	that is now detached.

	Prerequisites:
	The original VM (`source-vm`) exists and is stopped (so that
	the PVC is detached.)

	The original snapshot (`vm-snapshot`) exists.
	"""

	name, ssh_user = restored_vm_2

	def actassert(sh):
	out, _ = sh.exec_command("cat test.txt")
	assert "123" in out

	vm_shell_do(name, api_client,
	host_shell, vm_shell,
	ssh_user, ssh_keypair,
	actassert, wait_timeout)

		1. After pools are created, we remove the harvester-cloud-provider in Apps > Installed Apps (kube-system namespace).
		1. Add new charts in Apps > Repositories, use https://charts.harvesterhci.io to install and select 0.2.3.

		1. Create the first pool, specify the `etcd` and `control plane` roles
		1. Create the second pool, specify the `work` role only

		@@ -0,0 +1,22 @@
		---
		title: Upgrade with VM shutdown in the operating system

Add part1 manual test case from v1.3.0 validated tickets. #1217

Are you sure you want to change the base?

Add part1 manual test case from v1.3.0 validated tickets. #1217

Conversation

TachunLin commented Apr 10, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

khushboo-rancher left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

TachunLin Apr 12, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

test_4_vm_backup_restore.py

test_4_vm_snapshot.py

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

TachunLin Apr 12, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

TachunLin Apr 12, 2024 •

edited

TachunLin Apr 12, 2024 •

edited