[TEST] Analyze master branch daily regression failed test cases

[chriscchien]

What's the test to develop? Please describe

• test_engine_image_daemonset_restart:
  • Failure rate ~ 3/30
  • No ticket
• test_cloning_with_detached_source_volume
  • Failure rate ~ 6/30
  • [BUG] Test case test_cloning_with_detached_source_volume failed on pipeline #4848
• test_autosalvage_with_data_locality_enabled
  • Flaky for a long time
  • [TEST] Flaky test case test_autosalvage_with_data_locality_enabled #4814
• test_inc_restoration_with_multiple_rebuild_and_expansion
  • Failure rate ~ 1/30
  • No ticket
• test_node_eviction_multiple_volume
  • Failure rate ~ 12/30
  • No ticket
• test_setting_concurrent_rebuild_limit
  • Failure rate ~ 10/30
  • fix flaky test_setting_concurrent_rebuild_limit longhorn-tests#1215
• test_snapshot_hash_detect_corruption_in_global_enabled_mode
  • Failure rate ~ 12/30
  • [BUG] test_snapshot_hash_detect_corruption_in_global_enabled_mode failed due to not proper cron job value  #5394
  • [BUG] Cron job triggered replica rebuilding keeps repeating itself after corrupting snapshot data #6129
• test_system_backup_and_restore_volume_with_data
  • Failure rate ~ 30/30
  • [TEST] test case test_system_backup_and_restore_volume_with_data always failed #5442

Additional context

This ticket is master branch calculated result from #5459 which arranged by @yangchiu

[roger-ryao]

https://ci.longhorn.io/job/public/job/v1.5.x/job/v1.5.x-longhorn-tests-sles-amd64/1/

Need set allow-volume-creation-with-degraded-availability false(#1512)

• test_rebuild_with_restoration - [TEST] Update rebuilding with restoration test cases #6061
• test_single_replica_restore_failure - [TEST] Update rebuilding with restoration test cases #6061

Flaky(passing after rerun)

• test_migration_with_rebuilding_replica
• test_engine_image_not_fully_deployed_perform_auto_upgrade_engine

Known issue

• test_kubernetes_status - [TEST] Update case test_backup_kubernetes_status #6062
• test_backup_kubernetes_status[s3] - [TEST] Update case test_backup_kubernetes_status #6062
• test_csi_mount_volume_online_expansion - [BUG] The second time PVC online expansion always fail on SLES #6076
• test_inc_restoration_with_multiple_rebuild_and_expansion - [BUG] Test case test_inc_restoration_with_multiple_rebuild_and_expansion failed on SLES #6089
• test_recurring_jobs_allow_detached_volume[s3] - [BUG] test_recurring_jobs_allow_detached_volume failed #6124
• test_recurring_jobs_when_volume_detached_unexpectedly[s3] - [BUG] test_recurring_jobs_allow_detached_volume failed #6124
• test_snapshot_hash_detect_corruption_in_global_enabled_mode - [BUG] Cron job triggered replica rebuilding keeps repeating itself after corrupting snapshot data #6129
• test_snapshot_hash_detect_corruption_in_global_fast_check_mode - [BUG] Cron job triggered replica rebuilding keeps repeating itself after corrupting snapshot data #6129
• test_dr_volume_with_restore_command_error[s3] - [BUG] test_dr_volume_with_restore_command_error failed #6130

[mantissahz]

@longhorn/qa
just found the command awk that is not installed in registry.suse.com/bci/bci-base:15.4
These functions will use the awk command:

• crash_engine_process_with_sigkill
• crash_replica_processes
• get_pod_data_md5sum

It might cause lots of test cases failed.

[innobead]

@mantissahz we can just install it to the test image or even change the test cases w/o using awk?

[mantissahz]

The crash command will be executed in the instance-manager pod, so we need to install the awk when we build the image and only for testing.
I am testing another command without awk to see if it works as awk does.

[roger-ryao]

v1.5.0-rc1 amd64
Flaky(passing after rerun)

• test_ha.py::test_rebuild_with_restoration
• test_ha.py::test_single_replica_restore_failure
• test_recurring_job.py::test_recurring_jobs_when_volume_detached_unexpectedly
• test_settings.py::test_setting_concurrent_volume_backup_restore_limit
• test_settings.py::test_setting_concurrent_volume_backup_restore_limit_should_not_effect_dr_volumes
• test_infra.test_offline_node

Known issue

• test_ha.test_autosalvage_with_data_locality_enabled
  • [TEST] Flaky test case test_autosalvage_with_data_locality_enabled #4814
• tests.test_ha.test_inc_restoration_with_multiple_rebuild_and_expansion
  • [BUG] Test case test_inc_restoration_with_multiple_rebuild_and_expansion failed on SLES #6089

---

v1.5.0-rc1 arm64
Flaky(passing after rerun)

• tests.test_ha.test_rebuild_with_restoration[nfs]
• tests.test_ha.test_single_replica_restore_failure[nfs]

Known issue

• tests.test_ha.test_autosalvage_with_data_locality_enabled
  • [TEST] Flaky test case test_autosalvage_with_data_locality_enabled #4814
• tests.test_ha.test_inc_restoration_with_multiple_rebuild_and_expansion[nfs]
  • [BUG] Test case test_inc_restoration_with_multiple_rebuild_and_expansion failed on SLES #6089
• tests.test_csi.test_csi_mount_volume_online_expansion
  • [BUG] The second time PVC online expansion always fail on SLES #6076
• tests.test_recurring_job.test_recurring_jobs_allow_detached_volume[nfs]
  • [BUG] test_recurring_jobs_allow_detached_volume failed #6124
• tests.test_recurring_job.test_recurring_jobs_when_volume_detached_unexpectedly[nfs]
  • [BUG] test_recurring_jobs_allow_detached_volume failed #6124

---

v1.5.0-rc1 SLE micro amd64
Flaky(passing after rerun)

• test_backing_image.py::test_snapshot_with_backing_image
• test_basic.py::test_backup_status_for_unavailable_replicas
• test_basic.py::test_space_usage_for_rebuilding_only_volume_worst_scenario
• test_engine_upgrade.py::test_engine_live_upgrade_while_replica_concurrent_rebuild
• test_ha.py::test_engine_image_not_fully_deployed_perform_engine_upgrade
• test_recurring_job.py::test_recurring_job
• test_settings.py::test_setting_concurrent_rebuild_limit
• test_settings.py::test_setting_concurrent_volume_backup_restore_limit_should_not_effect_dr_volumes
• test_system_backup_restore.py::test_system_backup_and_restore
• test_system_backup_restore.py::test_system_backup_with_volume_backup_policy_if_not_present
• test_infra.py::test_offline_node

Known issue

• test_ha.py::test_inc_restoration_with_multiple_rebuild_and_expansion
  • [BUG] Test case test_inc_restoration_with_multiple_rebuild_and_expansion failed on SLES #6089
• tests.test_csi.test_csi_mount_volume_online_expansion
  • [BUG] The second time PVC online expansion always fail on SLES #6076

[mantissahz]

Use pgrep -f to get the process id instead of ps uaxw |... |awk {print $2}
longhorn/longhorn-tests#1408

[roger-ryao]

v1.5.0-rc2 amd64

• tests.test_ha.test_inc_restoration_with_multiple_rebuild_and_expansion[nfs]
  • [BUG] Test case test_inc_restoration_with_multiple_rebuild_and_expansion failed on SLES #6089
• tests.test_csi.test_csi_mount_volume_online_expansion
  • [BUG] The second time PVC online expansion always fail on SLES #6076
• tests.test_ha.test_autosalvage_with_data_locality_enabled
  • [TEST] Flaky test case test_autosalvage_with_data_locality_enabled #4814
• tests.test_recurring_job.test_recurring_jobs_when_volume_detached_unexpectedly[nfs]
  • [BUG] test_recurring_jobs_allow_detached_volume failed #6124
• test_ha.py::test_dr_volume_with_restore_command_error[nfs]
  • [BUG] test_dr_volume_with_restore_command_error failed #6130
• tests.test_snapshot.test_snapshot_hash_detect_corruption_in_global_enabled_mode
  • [BUG] Cron job triggered replica rebuilding keeps repeating itself after corrupting snapshot data #6129
• tests.test_snapshot.test_snapshot_hash_detect_corruption_in_global_fast_check_mode
  • [BUG] Cron job triggered replica rebuilding keeps repeating itself after corrupting snapshot data #6129
• tests.test_ha.test_recovery_from_im_deletion
  • passing after rerun

---

v1.5.0-rc2 arm64

• test_basic.test_default_storage_class_syncup
  • passing after rerun
• test_ha.test_dr_volume_with_restore_command_error[s3]
  • [BUG] test_dr_volume_with_restore_command_error failed #6130
• test_ha.test_engine_image_not_fully_deployed_perform_auto_upgrade_engine
  • [BUG] test_dr_volume_with_restore_command_error failed #6130
• test_snapshot.test_snapshot_hash_detect_corruption_in_global_enabled_mode
  • [BUG] Cron job triggered replica rebuilding keeps repeating itself after corrupting snapshot data #6129
• test_csi.test_csi_mount_volume_online_expansion
  • [BUG] The second time PVC online expansion always fail on SLES #6076
• test_recurring_job.test_recurring_jobs_allow_detached_volume[s3]
  • [BUG] test_recurring_jobs_allow_detached_volume failed #6124
• test_ha.test_inc_restoration_with_multiple_rebuild_and_expansion[s3]
  • [BUG] Test case test_inc_restoration_with_multiple_rebuild_and_expansion failed on SLES #6089

---

v1.5.0-rc2 SLE micro amd64

• tests.test_ha.test_inc_restoration_with_multiple_rebuild_and_expansion[nfs]
  • [BUG] Test case test_inc_restoration_with_multiple_rebuild_and_expansion failed on SLES #6089
• tests.test_ha.test_engine_image_not_fully_deployed_perform_auto_upgrade_engine
  • [BUG] Volume gets stuck in an unknown state forever if created in an engine not fully deployed environment #6131
• tests.test_recurring_job.test_recurring_jobs_allow_detached_volume[nfs]
  • [BUG] test_recurring_jobs_allow_detached_volume failed #6124
• tests.test_recurring_job.test_recurring_jobs_when_volume_detached_unexpectedly[nfs]
  • [BUG] test_recurring_jobs_allow_detached_volume failed #6124
• tests.test_ha.test_autosalvage_with_data_locality_enabled
  • [TEST] Flaky test case test_autosalvage_with_data_locality_enabled #4814
• tests.test_csi.test_csi_mount_volume_online_expansion
  • [BUG] The second time PVC online expansion always fail on SLES #6076

---

v1.5.0-rc2 SLE micro arm64

• tests.test_csi.test_csi_mount_volume_online_expansion
  • [BUG] The second time PVC online expansion always fail on SLES #6076
• tests.test_ha.test_inc_restoration_with_multiple_rebuild_and_expansion[s3]
  • [BUG] Test case test_inc_restoration_with_multiple_rebuild_and_expansion failed on SLES #6089
• tests.test_recurring_job.test_recurring_job[s3]
  • passing after rerun
• tests.test_recurring_job.test_recurring_jobs_when_volume_detached_unexpectedly[s3]
  • [BUG] test_recurring_jobs_allow_detached_volume failed #6124
• tests.test_snapshot.test_snapshot_hash_detect_corruption_in_global_enabled_mode
  • [BUG] Cron job triggered replica rebuilding keeps repeating itself after corrupting snapshot data #6129

[roger-ryao]

v1.5.0-rc3 amd64

• tests.test_ha.test_inc_restoration_with_multiple_rebuild_and_expansion[nfs]
  • [BUG] Test case test_inc_restoration_with_multiple_rebuild_and_expansion failed on SLES #6089
• tests.test_ha.test_recovery_from_im_deletion
  • [BUG] test_recovery_from_im_deletion cannot pass in particular environment #6171
• tests.test_csi.test_csi_mount_volume_online_expansion
  • [BUG] The second time PVC online expansion always fail on SLES #6076

---

v1.5.0-rc3 arm64

• test_csi.py::test_csi_mount_volume_online_expansion
  • [BUG] The second time PVC online expansion always fail on SLES #6076
• test_ha.py::test_inc_restoration_with_multiple_rebuild_and_expansion[s3]
  • [BUG] Test case test_inc_restoration_with_multiple_rebuild_and_expansion failed on SLES #6089
• test_ha.py::test_engine_image_not_fully_deployed_perform_auto_upgrade_engine
  • [BUG] Volume gets stuck in an unknown state forever if created in an engine not fully deployed environment #6131
• test_ha.py::test_autosalvage_with_data_locality_enabled
  • [TEST] Flaky test case test_autosalvage_with_data_locality_enabled #4814
• test_migration.py::test_migration_with_rebuilding_replica
  • [BUG] migration test cases could fail due to unexpected volume controllers and replicas status #6215