Anti-affinity for k8s nodes (control-plane and workers)

[garloff]

As a user, I want to be sure that having containers on different nodes are not going down simultaneously because they are actually VMs (servers) on the same host (hypervisor). I need this to be the default behavior.

The control plane must survive for the cluster to survive. This requires hard anti-affinity (or in a shared control plane enough redundancy guaranteed otherwise).
For workers, we still want to avoid co-scheduling, but it does not necessarily need to be a hard anti-affinity.

This could be made opt-out, which could be useful for very small environments.

Definition of Done:

• A standardization document is written (we only produced a Decision Record with this issue in the end)
• A follow up issue is created.

[garloff]

In previous meetings we had agreed that using

• hard anti-affinity on control plane nodes
  (ignore for shared control plane but require enough redundancy in a different way then)
• soft anti-affinity on work nodes
  is the right default.