RFE: options for OSD restart gate

[anthonyeleven]

My understanding is that the operator gates OSD restarts via ceph osd ok-to-stop.

I would like the option to change or augment this condition so that the operator will not proceed to restart the next OSD unless all PGs are active+clean.*

We recently adjusted our K8s requests and limitssettings and experienced a subset of inactive PGs that impacted clients on an EC HDD pool. My confidence in ok-to-stop has always been limited, but I don't have a smoking gun.

Is this a bug report or feature request?

• Feature Request

What should the feature do:

What is use case behind this feature:

Environment:

[travisn]

Ok, this would be even a more strict option for upgrades, such as upgradeChecksRequireActiveCleanPGs.
So you don't have evidence that ok-to-stop wasn't working, but it would help you gain more confidence in the upgrades with this extra level of assurance?

[anthonyeleven]

Exactly. This wasn't even an upgrade, it was a K8s-induced rolling restart. Since ok-to-stop as a sole gate would seem to permit OSD restarts while backfill is pending, maybe there's a race condition or something. I don't have the ability to attempt to reproduce, as the production cluster is the only one with any scale and EC HDDs.

Alternately I'd love to even be able to introduce a configurable extra delay between OSD restarts, if that would be simpler. I.e., "sleep for 300 seconds after you get ok-to-stop" before proceeding.

[mmaoyu]

I think this feature requires check cluster if the PGs status is clean before upgrade any osds.
Implementation steps are:

1. Add a configuration UpgradeOSDRequireHealthyPGs in crd cephcluster spec
2. Check ceph cluster whether the stataus is clean when SkipUpgradeChecks=false and UpgradeOSDRequireHealthyPGs=true in method updateExistingOSDs before pop osd query. If status is not clean or check failed return else continue upgrade logic.

I'd like to take the issue😄

[mmaoyu]

/assign

[github-actions]

Thanks for taking this issue! Let us know if you have any questions!

[anthonyeleven]

Thanks for the feature! It occurs to me that I might have asked for a healthy mon quorum too but I think that's a lower risk.