Make Zenith SSHD more robust

[assumptionsandg]

• Set SSHD replicas to 3 by default
• Use topology spread constraints to ensure those 3 pods are spread over the workers as evenly as possible (i.e. maxSkew: 1)
• Use pod disruption budget with maxUnavailable: 1 to ensure that only one SSHD pod is killed at once

[mkjpryor]

@assumptionsandg There are more things than this that we need to do to make Zenith SSHD properly robust. I have changed the title and added some items in the description to reflect this.

[mkjpryor]

Topology spread should work in an AIO, and will make sure that the pods schedule on different nodes by preference on a HA cluster, so I would prefer to have it back...

The reason for using topology spread and not anti-affinity is specifically because you specify a maxSkew between nodes. So if there is one node and 3 replicas, all 3 should go on the same node because there is no other node to compare skew against.

If it wasn't working, we should work out why.

[mkjpryor]

@JohnGarbutt Why 2 and not 3 for the replicas? I have a slight preference for 3 as it will result in one per node in our default HA setup.

[JohnGarbutt]

So it was failing on the single node deployment, so I was mostly suggesting dialling a few things back to see if that gets it working. Now we have something working, we could always have a look at adding back in the max skew... it looked like it was failing to create the first pod, but was OK if you already had one pod running, id I understand what we found out correctly?

[mkjpryor]

LGTM