qa/suites/crimson-rados/thrash: enable chance_down

[Matan-B]

As the thrash tests were introduced, some options were disabled
until the tests are stabilized.

Re-enable chance_down option (default is 0.4) to detect bugs on restart.

Since it will probably take few iterations before thrash and recovery tests ('default.yaml')
will pass successfully, add anoter 'simple.yaml' which should remain stable.

Fixes: https://tracker.ceph.com/issues/65130

Contribution Guidelines

• To sign and title your commits, please refer to Submitting Patches to Ceph.

• If you are submitting a fix for a stable branch (e.g. "quincy"), please refer to Submitting Patches to Ceph - Backports for the proper workflow.

• When filling out the below checklist, you may click boxes directly in the GitHub web UI. When entering or editing the entire PR message in the GitHub web UI editor, you may also select a checklist item by adding an x between the brackets: [x]. Spaces and capitalization matter when checking off items this way.

Checklist

• Tracker (select at least one)
  • References tracker ticket
  • Very recent bug; references commit where it was introduced
  • New feature (ticket optional)
  • Doc update (no ticket needed)
  • Code cleanup (no ticket needed)
• Component impact
  • Affects Dashboard, opened tracker ticket
  • Affects Orchestrator, opened tracker ticket
  • No impact that needs to be tracked
• Documentation (select at least one)
  • Updates relevant documentation
  • No doc update is appropriate
• Tests (select at least one)
  • Includes unit test(s)
  • Includes integration test(s)
  • Includes bug reproducer
  • No tests

Show available Jenkins commands

• jenkins retest this please
• jenkins test classic perf
• jenkins test crimson perf
• jenkins test signed
• jenkins test make check
• jenkins test make check arm64
• jenkins test submodules
• jenkins test dashboard
• jenkins test dashboard cephadm
• jenkins test api
• jenkins test docs
• jenkins render docs
• jenkins test ceph-volume all
• jenkins test ceph-volume tox
• jenkins test windows
• jenkins test rook e2e

[Matan-B]

@athanatos, testing with #56376 to verify that we catch the error now.

thrash tests with snaps enabled with this PR:
https://pulpito.ceph.com/matan-2024-03-27_08:49:03-crimson-rados-wip-sjust-crimson-testing-2024-03-20-distro-crimson-smithi/

Edit: OSD logs are not available yet but teuthology log looks promising:

2024-03-27T09:23:28.758 INFO:tasks.thrashosds.thrasher:Killing osd 1, live_osds are [1, 2, 3, 0]

2024-03-27T09:23:33.824 INFO:tasks.thrashosds.thrasher:Reviving osd 1
2024-03-27T09:23:33.824 INFO:tasks.ceph.osd.1:Restarting daemon

2024-03-27T09:23:43.176 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s

[Matan-B]

We have other issues that involve restarting that require handling before merging, tested this PR on main as well:
https://pulpito.ceph.com/matan-2024-03-27_13:02:57-crimson-rados-main-distro-crimson-smithi/

Hopefully that's caused by the scrub tracked issue

[athanatos]

@Matan-B Ah, that makes sense. If you have time, it would be helpful to do a first pass of that run and file bugs for the defects. We can split up the work on fixing them.

[Matan-B]

@Matan-B Ah, that makes sense. If you have time, it would be helpful to do a first pass of that run and file bugs for the defects. We can split up the work on fixing them.

@athanatos,

https://tracker.ceph.com/issues/65203
https://tracker.ceph.com/issues/65201
https://tracker.ceph.com/issues/65200

[Matan-B]

https://tracker.ceph.com/issues/65203
https://tracker.ceph.com/issues/65201
https://tracker.ceph.com/issues/65200

Should be addressed by #56611 and #56610, testing
https://shaman.ceph.com/builds/ceph/wip-matanb-crimson-testing-thrash-options

https://pulpito.ceph.com/matan-2024-04-01_13:06:33-crimson-rados-wip-matanb-crimson-testing-thrash-options-distro-crimson-smithi/

[Matan-B]

@athanatos, the previous trackers seem to be ok, new one popped out:

https://tracker.ceph.com/issues/65247

[Matan-B]

Updating relevant PRs:
#56611
#56610
#56832
#56806

CC: @xxhdx1985126, @athanatos

[Matan-B]

I'll merge this based on:
https://pulpito.ceph.com/matan-2024-04-16_14:35:17-crimson-rados-main-distro-crimson-smithi/

@xuxuehan, we can start to test+merge the 'crimson-replicated-recovery' PRs based on the non-recovery thrash tests (as we did before) until both are stable.

[Matan-B]

jenkins test api

[Matan-B]

jenkins test make check

[athanatos]

jenkins test api