Adding rollback mechanism to handle bootstrap failures

[rkachach]

This PR introduces a simple mechanism to rollback cluster files and processes in case of a failed bootstrap. Two new mutually exclusive options are provided:

1. --cleanup-on-failure: when enabled this option will enable the rollback so broken cluster files are deleted automatically
2. --no-cleanup-on-failure: when enabled this option will disable the rollback keeping the broken cluster files

These options are provided to ease the backport of this new feature. In older releases the rollback mechanism will be disabled by default to keep the legacy behaviour (nothing is deleted). In this case the user will need to pass the --cleanup-on-failure flag in case he wants to enable the rollback mechanism. In main the rollback feature will be enabled by default.

Fixes: https://tracker.ceph.com/issues/57016

Known issue:

In case of a keyboard interruption (ctrl+c) sometimes the following message appears. It seems to be related with some asyncio which is not being closed appropriately:

Ceph version: ceph version 18.0.0-4138-g1657989a (1657989a957fd4a6a732417127612a248586fe16) reef (dev)
Extracting ceph user uid/gid from container image...
Creating initial keys...
^CKeyboardInterrupt: 
Deleting cluster with fsid: 140e69ac-faec-11ed-bb04-525400d333a3
Exception ignored in: <function BaseSubprocessTransport.__del__ at 0x7f02cc99fca0>
Traceback (most recent call last):
  File "/usr/lib64/python3.9/asyncio/base_subprocess.py", line 126, in __del__
    self.close()
  File "/usr/lib64/python3.9/asyncio/base_subprocess.py", line 104, in close
    proto.pipe.close()
  File "/usr/lib64/python3.9/asyncio/unix_events.py", line 536, in close
    self._close(None)
  File "/usr/lib64/python3.9/asyncio/unix_events.py", line 560, in _close
    self._loop.call_soon(self._call_connection_lost, exc)
  File "/usr/lib64/python3.9/asyncio/base_events.py", line 746, in call_soon
    self._check_closed()
  File "/usr/lib64/python3.9/asyncio/base_events.py", line 510, in _check_closed
    raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed

Contribution Guidelines

• To sign and title your commits, please refer to Submitting Patches to Ceph.

• If you are submitting a fix for a stable branch (e.g. "pacific"), please refer to Submitting Patches to Ceph - Backports for the proper workflow.

Checklist

• Tracker (select at least one)
  • References tracker ticket
  • Very recent bug; references commit where it was introduced
  • New feature (ticket optional)
  • Doc update (no ticket needed)
  • Code cleanup (no ticket needed)
• Component impact
  • Affects Dashboard, opened tracker ticket
  • Affects Orchestrator, opened tracker ticket
  • No impact that needs to be tracked
• Documentation (select at least one)
  • Updates relevant documentation
  • No doc update is appropriate
• Tests (select at least one)
  • Includes unit test(s)
  • Includes integration test(s)
  • Includes bug reproducer
  • No tests

Show available Jenkins commands

• jenkins retest this please
• jenkins test classic perf
• jenkins test crimson perf
• jenkins test signed
• jenkins test make check
• jenkins test make check arm64
• jenkins test submodules
• jenkins test dashboard
• jenkins test dashboard cephadm
• jenkins test api
• jenkins test docs
• jenkins render docs
• jenkins test ceph-volume all
• jenkins test ceph-volume tox
• jenkins test windows

[rkachach]

jenkins test make check

[phlogistonjohn]

I'm still a little wary of having this auto-cleanup opt out rather than opt in by default. Take this as my last official request to, at least initially, make this opt-in for at least one version. I will not harp on this subject and this is the last I will pester you about this.

[rkachach]

I'm still a little wary of having this auto-cleanup opt out rather than opt in by default. Take this as my last official request to, at least initially, make this opt-in for at least one version. I will not harp on this subject and this is the last I will pester you about this.

We haven't made a final decision yet, as you know this still on a proposal phase and all the options are still open so code can be adjusted easily in case we need to. Let's wait for some more feedback from the community and based on that have some more discussions on the upcoming weekly to decide whether we go with opt out or opt in alternative.

[adk3798]

LGTM, we'll just have to be careful when backporting to swap the default value for the --cleanup-on-failure flag as discussed in the weekly.

[rkachach]

last force push is to squash all the commits

[adk3798]

https://pulpito.ceph.com/adking-2023-07-11_16:35:52-orch:cephadm-wip-adk-testing-2023-07-11-0946-distro-default-smithi/

reruns of failed/dead jobs: https://pulpito.ceph.com/adking-2023-07-11_19:17:14-orch:cephadm-wip-adk-testing-2023-07-11-0946-distro-default-smithi/

After reruns, 1 failed, 1 dead job

• failed job was known issue with jaeger-tracing https://tracker.ceph.com/issues/59704
• dead job was infra/teuthology issue, Error reimaging machines: reached maximum tries (101) after waiting for 600 seconds

Overall, a pretty clean run. Nothing to block merging