Update rollout-operator controller to use zpdb before delete

[tcp13equals2]

This PR seeks to address a race condition which can occur when rolling pod updates are running and a voluntary pod eviction is received.

During the main reconcileStatefulSetsGroup() reconcile loop, there is opportunity for a voluntary eviction to have occurred, but the reconcile loop has not detected this and a pod can be deleted (for updating) which would breach the ZPDB.

This is more likely to occur when running the zone aware PodDisruptionBudget in a partition awareness mode. In the traditional max unavailable = 1 PDB, a voluntary eviction is denied if there is any disruption in any zone. Whilst the rolling update controller is performing updates the likelihood of a a voluntary eviction being allowed is small.

However, when in partition awareness mode, the voluntary eviction will be allowed if both pods in the partition are ready. ie a reconcile over ingester-zone-a pods is occurring - with each pod being deleted, a voluntary eviction comes in for ingester-zone-b-50. This eviction will be allowed if the reconcile loop has not yet reached ingester-zone-a-50. The update loop will not be aware of this eviction and issue a delete on ingester-zone-a-50.

The fix for this is to call the zpdb eviction controller prior to each delete and confirm that the zpdb will not be breached by this delete.

Note - this will result in additional statefulset and pod checks before each rolling pod update (delete).

[tcp13equals2]

@charleskorn - can you have another look at this one please. Note that this change means the eviction controller always runs, and note the jsonnet updates to support this.

[tcp13equals2]

@charleskorn Thanks for the feedback - I've adjusted accordingly.

[charleskorn]

LGTM modulo suggestion below