[TEST] Introduce Negative Testing

[innobead]

What's the test to develop? Please describe

Test Longhorn operations with Negative factors.

For each Longhorn operation and Negative factor listed below, should be implemented in robot:

Longhorn operation:

• Rebuild replica
• Keep writing
• Delete replica while writing
• Create backup while writing
• Restore backup
• Attach/detach volume
• Migrate volume
• Clone volume
• Expand volume
• Upgrade longhorn

For details about what actually these operations should do, please reference and follow the confluence page to implement

Negative factor:

• Reboot volume node
• Reboot replica node
• Delete volume node
• High latency API server
• Introduce network latency on all nodes
• Disconnect network of all nodes
• Increase CPU load
• High memory consumption
• Make filesystem on node as ReadOnly
• Almost fill up Longhorn disk
• Disk with low IOPS (HDD)
• Make some blocks on the device corrupt
• Introduce hard drive SMART failure
• Unschedule Longhorn disk
• Take snapshots on VM or freeze VM
• Force drain nodes
• Restart kubelet

For details about what actually these factors should do and what the corresponding test case looks like, please reference and follow the confluence page to implement

Once the Longhorn operation and Negative factor are implemented, combine them together to form a negative test case like:
Node reboot + Rebuild replica.

Sometimes a negative test case can have more than one negative factor like: High latency API server + Node reboot + Keep writing. So again reference the confluence page to have a clear picture of it could look like before starting implementation.

Describe the items of the test development (DoD, definition of done) you'd like

• Define framework architecture and code structure
• Have separate Jenkins pipeline for negative testing
• Implement each Negative factor
• Consider the real issues encountered, implement corresponding Longhorn operation and combine with Negative factor to create the automated negative test case and execute it to reproduce issues
• Automate manual test items in release testing: [TEST] Automate manual test cases in robot #7211
• Prepare long-running (local lab) test environment

Additional context

cc @longhorn/qa

[innobead]

@khushboo-rancher feel free to update this task description, so we can plan it accordingly.

cc @longhorn/qa

[innobead]

Moved this from 1.5.0, but follow this on QA board instead.

[innobead]

The current status is @yangchiu is working on the framework part with some example test case implementation. When they are ready and get merged, we will start the remaining case implementation based on priority.

cc @nitendra-suse @chriscchien @roger-ryao @khushboo-rancher

[innobead]

@yangchiu I feel the description of this issue is outdated. Please help keep this up-to-date and refer the related issues to this EPIC.

If need a separate epic, go ahead to create it.

cc @khushboo-rancher

[innobead]

@yangchiu remember to update the description of this ticket, as discussed.

[yangchiu]

@yangchiu remember to update the description of this ticket, as discussed.

I find I've updated it before. Read through the ticket description, I think it's quite clear, and the newly created ticket #7211 is also recorded.

[innobead]

Great. Thanks.