ceph: improve upgrade procedure

[leseb]

Description of your changes:

When a cluster is updated with a different image version, this triggers
a serialized restart of all the pods. Prior to this commit, no safety
check were performed and rook was hoping for the best outcome.

Now before doing restarting a daemon we check it can be restarted. Once
it's restarted we also check we can pursue with the rest of the
platform. For instance, with monitors we check that they are in quorum,
for OSD we check that PGs are clean and for MDS we make sure they are
active.

Fixes: #2889
Signed-off-by: Sébastien Han seb@redhat.com

Which issue is resolved by this Pull Request:
Resolves #2889

Checklist:

• Documentation has been updated, if necessary.
• Pending release notes updated with breaking and/or notable changes, if necessary.
• Upgrade from previous release is tested and upgrade user guide is updated, if necessary.
• Code generation (make codegen) has been run to update object specifications, if necessary.
• Comments have been added or updated based on the standards set in CONTRIBUTING.md

[skip ci]
Known issue.

[leseb]

This is wip since it relies on ceph/ceph#27146. I'll finish this once it's merged.

[leseb]

Ready for an initial review, I'll probably split the commit into multiple ones later.

[BlaineEXE]

I don't have good feelings about merging this right before the v1.0 release. Something that could be an issue that has been an issue for the mon init in the past is that Rook has to be able to get the cluster into a working state if the operator fails midway through any of these interactions, and that failure might cause mons or other daemons to be in unhealthy states. If Rook's actions allow a daemon to get messed up, it seems right to me that Rook should be able to rectify that.

Having a strict check for healthiness before continuing with an upgrade is something that is absolutely needed IMO, but I think there needs to be a lot of consideration and vetting before I would feel that what we have is comfortably stable and won't risk upgrade pains for users.

I would vote that we (1) wait until v1.0 is released before merging this, and I would vote that (2) this PR should be split into different PRs for health checking of different daemons so the changes are smaller and easier to reason through.

[leseb]

A simpler approach here: #3037
I keep this one around and will resume once 1.0 has merged.

[BlaineEXE]

I went through some logic in the past to see if we could rely on health checks to make sure mons are healthy before proceeding with upgrades, and it's tricky stuff. It could be possible for the mons to get into a state where they aren't healthy and where Rook won't correct an issue it is theoretically able to solve because it is waiting on a prerequisite status that would be resolved by continuing to orchestrate.

The code here looks good, and I think it makes sense, but I want to make sure we aren't missing anything. What can we do to make sure we aren't allowing the mons to be deadlocked in an error state?

I think I would be in favor of merging this code early in the v1.1 development phase so we can hopefully see any wrinkles that need fixed before release, because I'm not sure if we can really catch all the little issues. In practicality, I think we could stand to redesign Rook's monitor bootstrap/managment/upgrades from the ground up and focus on (1) creating an idempotent orchestration and (2) testing what happens when the operator crashes at each step so that we can be more sure that we can't get into a deadlocked state.

[leseb]

@BlaineEXE thanks for your feedback, let me do more tests this week so hopefully, we can move forward.

[leseb]

Triggered the CI 3 times since yesterday and always randomly fails on different jobs. So merging like this.