ceph: fix mon pdb reconcile

[sp98]

updated mon pdb reconcile to delete mon pdb and create a new one when mon count changes in the cluster spec

Tests:

• 3 Mons

Every 2.0s: oc get pdb                                                       localhost.localdomain: Wed Oct 21 13:45:00 2020

NAME                MIN AVAILABLE   MAX UNAVAILABLE   ALLOWED DISRUPTIONS   AGE
rook-ceph-mon-pdb   2               N/A               1                     3m58s

• Updated mon count from 3 to 5

Every 2.0s: oc get pdb                                                       localhost.localdomain: Wed Oct 21 13:48:37 2020

NAME                MIN AVAILABLE   MAX UNAVAILABLE   ALLOWED DISRUPTIONS   AGE
rook-ceph-mon-pdb   3               N/A               2                     2m47s

• updated mon count from 5 to 3

Every 2.0s: oc get pdb                                                       localhost.localdomain: Wed Oct 21 13:53:38 2020

NAME                MIN AVAILABLE   MAX UNAVAILABLE   ALLOWED DISRUPTIONS   AGE
rook-ceph-mon-pdb   2               N/A               1                     3m38s

Signed-off-by: Santosh Pillai sapillai@redhat.com

Description of your changes:

Which issue is resolved by this Pull Request:
Resolves #

Checklist:

• Commit Message Formatting: Commit titles and messages follow guidelines in the developer guide.
• Skip Tests for Docs: Add the flag for skipping the build if this is only a documentation change. See here for the flag.
• Skip Unrelated Tests: Add a flag to run tests for a specific storage provider. See test options.
• Reviewed the developer guide on Submitting a Pull Request
• Documentation has been updated, if necessary.
• Unit tests have been added, if necessary.
• Integration tests have been added, if necessary.
• Pending release notes updated with breaking and/or notable changes, if necessary.
• Upgrade from previous release is tested and upgrade user guide is updated, if necessary.
• Code generation (make codegen) has been run to update object specifications, if necessary.

[travisn]

How about this pattern?

if err != nil {
  if apierrors.IsNotFound(err) {
    // insert lines 71-75 to create the pdb
  }
  return errors.Wrap(err, "failed to get mon pdb")
}

[sp98]

Yes, this looks like the common pattern used across rook. Changed code to use this format.

[travisn]

How about making these code changes inside reconcileStaticPDB()? Seems like we would want the update pattern other places as well, right? (or at least it would be a no-op if the minAvailable never changes)

[sp98]

moved changes to reconcileStaticPDB() method.