mon/OSDMonitor: Ensure kvmon() is writeable before handling "osd new" cmd

[sseshasa]

Before proceeding to handle "osd new" mon command as part of
OSDMonitor::prepare_command_impl(), a check is made to verify if the
authmon is writeable. Later on, prepare_command_osd_new() invokes
KVMonitor::do_osd_new() to create pending dmcrypt keys and calls
propose_pending(). The propose could fail (with an assertion failure)
if there was a prior mon command that resulted in the kvmon invoking
propose_pending().

In order to avoid such a situation, introduce a check to verify that
kvmon is also writeable in OSDMonitor::prepare_command_impl(). If it
is not writeable, the op is pushed into the wait_for_active context
queue to be retried later.

Fixes: https://tracker.ceph.com/issues/55773
Signed-off-by: Sridhar Seshasayee sseshasa@redhat.com

Contribution Guidelines

• To sign and title your commits, please refer to Submitting Patches to Ceph.

• If you are submitting a fix for a stable branch (e.g. "pacific"), please refer to Submitting Patches to Ceph - Backports for the proper workflow.

Checklist

• Tracker (select at least one)
  • References tracker ticket
  • Very recent bug; references commit where it was introduced
  • New feature (ticket optional)
  • Doc update (no ticket needed)
  • Code cleanup (no ticket needed)
• Component impact
  • Affects Dashboard, opened tracker ticket
  • Affects Orchestrator, opened tracker ticket
  • No impact that needs to be tracked
• Documentation (select at least one)
  • Updates relevant documentation
  • No doc update is appropriate
• Tests (select at least one)
  • Includes unit test(s)
  • Includes integration test(s)
  • Includes bug reproducer
  • No tests

Show available Jenkins commands

• jenkins retest this please
• jenkins test classic perf
• jenkins test crimson perf
• jenkins test signed
• jenkins test make check
• jenkins test make check arm64
• jenkins test submodules
• jenkins test dashboard
• jenkins test dashboard cephadm
• jenkins test api
• jenkins test docs
• jenkins render docs
• jenkins test ceph-volume all
• jenkins test ceph-volume tox
• jenkins test windows

[sseshasa]

jenkins test api

[sseshasa]

Tests Performed:
The cluster and OSD deployment scripts using cephadm was executed with the fix in place on the same cluster where the issue was observed. Both the deployment scripts passed with no failures.

After the review, this needs to go through teuthology.

[neha-ojha]

LGTM

[sseshasa]

Teuthology Test Result:

http://pulpito.front.sepia.ceph.com/yuriw-2022-06-13_16:36:31-rados-wip-yuri7-testing-2022-06-13-0706-distro-default-smithi/
Found no related Failures

[ljflores]

http://pulpito.front.sepia.ceph.com/yuriw-2022-06-13_16:36:31-rados-wip-yuri7-testing-2022-06-13-0706-distro-default-smithi/

Failures, unrelated:
1. https://tracker.ceph.com/issues/55986
2. https://tracker.ceph.com/issues/52321
3. https://tracker.ceph.com/issues/52124
4. https://tracker.ceph.com/issues/52316
5. https://tracker.ceph.com/issues/55322
6. https://tracker.ceph.com/issues/55741
7. https://tracker.ceph.com/issues/56034 --> new Tracker; unrelated to the PRs in this run.

Details:
1. cephadm: Test failure: test_cluster_set_reset_user_config (tasks.cephfs.test_nfs.TestNFS) - Ceph - Orchestrator
2. qa/tasks/rook times out: 'check osd count' reached maximum tries (90) after waiting for 900 seconds - Ceph - Orchestrator
3. Invalid read of size 8 in handle_recovery_delete() - Ceph - RADOS
4. qa/tasks/mon_thrash.py: _do_thrash AssertionError len(s['quorum']) == len(mons) - Ceph - RADOS
5. test-restful.sh: mon metadata unable to be retrieved - Ceph - Mgr
6. cephadm/test_dashboard_e2e.sh: Unable to find element cd-modal .custom-control-label when testing on orchestrator/01-hosts.e2e-spec.ts - Ceph - Mgr - Dashboard
7. qa/standalone/osd/divergent-priors.sh fails in test TEST_divergent_3() - Ceph - RADOS