osd: Handle oncommits and wait for future work items from mClock queue

[sseshasa]

When a worker thread with the smallest thread index waits for future work
items from the mClock queue, oncommit callbacks are called. But after the
callback, the thread has to continue waiting instead of returning back to
the ShardedThreadPool::shardedthreadpool_worker() loop. Returning results
in the threads with the smallest index across all shards to busy loop
causing very high CPU utilization.

The fix involves reacquiring the shard_lock and waiting on sdata_cond
until notified or until time period lapses. After this, the smallest
thread index repopulates the oncommit queue from the context_queue
if there were any additions.

Fixes: https://tracker.ceph.com/issues/56530
Signed-off-by: Sridhar Seshasayee sseshasa@redhat.com

Contribution Guidelines

• To sign and title your commits, please refer to Submitting Patches to Ceph.

• If you are submitting a fix for a stable branch (e.g. "pacific"), please refer to Submitting Patches to Ceph - Backports for the proper workflow.

Checklist

• Tracker (select at least one)
  • References tracker ticket
  • Very recent bug; references commit where it was introduced
  • New feature (ticket optional)
  • Doc update (no ticket needed)
  • Code cleanup (no ticket needed)
• Component impact
  • Affects Dashboard, opened tracker ticket
  • Affects Orchestrator, opened tracker ticket
  • No impact that needs to be tracked
• Documentation (select at least one)
  • Updates relevant documentation
  • No doc update is appropriate
• Tests (select at least one)
  • Includes unit test(s)
  • Includes integration test(s)
  • Includes bug reproducer
  • No tests

Show available Jenkins commands

• jenkins retest this please
• jenkins test classic perf
• jenkins test crimson perf
• jenkins test signed
• jenkins test make check
• jenkins test make check arm64
• jenkins test submodules
• jenkins test dashboard
• jenkins test dashboard cephadm
• jenkins test api
• jenkins test docs
• jenkins render docs
• jenkins test ceph-volume all
• jenkins test ceph-volume tox
• jenkins test windows

[neha-ojha]

@sseshasa can you please attach a link to your before/after CPU utilization comparison plots to this PR?

[sseshasa]

Test CPU Utilization During Client Ops + Recoveries
The charts below illustrate the before and after effect of the fix. In both the charts, the test has both client ops and recoveries/backfills going on. A few seconds after recoveries and client ops are started, client ops are stopped to check if recoveries alone consume a lot of CPU.

Test using HDDs - Before Fix
The following chart shows the CPU usage during client ops and recoveries/backfills without the fix. It shows very high CPU usage both during and after client ops are stopped. One osd continues to consume 100% of the CPU when recovery rate is low towards the tail end.

Test using HDDs - After Fix
The initial usage when client ops + recoveries are going on are low in the range 5% - 11%. After the client ops are stopped but with recoveries still going on, there is a steep drop in the CPU utilization across all the OSDs and remains that way. For one of the osds towards the tail end of recovery, the CPU utilization hovers at around 1%.

Test using SSDs - Before Fix
The following chart shows CPU usage before the fix with the test using SSDs. It illustrates the same point made above with HDDs with CPU usage being very high with only recoveries going on after client ops are stopped midway.

Test using SSDs - After Fix
The initial high usage is with client ops + recoveries as expected. After the client ops are stopped and with recoveries going on, the CPU usage drops down to 45% and below through the recovery phase.

[athanatos]

This looks right, though ShardedOpWQ::_process has become rather tough to follow.

[sseshasa]

jenkins test windows

[sseshasa]

Teuthology Test (Rados suite)
https://pulpito.ceph.com/sseshasa-2022-07-27_04:42:35-rados-wip-fix-high-cpu-util-in-process-distro-default-smithi/

Summary
17/319 jobs failed. 1 dead job
No related failures observed.

Unrelated Failures

1. https://tracker.ceph.com/issues/52420
2. https://tracker.ceph.com/issues/56652
3. https://tracker.ceph.com/issues/55853
4. https://tracker.ceph.com/issues/52321
5. https://tracker.ceph.com/issues/54371

A handful of failures were due to test infra issues.