E2E tests should run with a highly available setup

[yaron2]

Today, end to end tests run the non-production recommended setup with one instance per pod.

As between the two modes (non-HA and HA) the highly available setup is both the recommended one and also the more complex one in terms of different coordination between resources on an otherwise no resource dependency coordinated hosting platform (Kubernetes), it's recommended we run the e2e tests with ha.enabled=true.

/cc @youngbupark @msfussell

Release Note

RELEASE NOTE: Added E2E tests option to run in HA mode

[youngbupark]

I agree on enable HA mode for the existing e2e tests. For non-HA mode, I would add one more KinD test for non-HA mode. @yaron2 what do you think?

1. AKS cluster test - HA mode enabled.
2. KinD test for three k8s versions with HA mode
   **3. Kind test for one version k8s with non-HA mode. ** -> new

cc/ @tcnghia

[tcnghia]

Regarding running HA tests on KinD, I think some experiments will be needed. It would be nice to test HA there if we can fit our tests there.

+1 to leave some coverage for non-HA to ensure a good experience for users trying things out.

[tcnghia]

To clarify the experimentations: currently, our KinD cluster was created with 2 worker nodes. I don't know what our HA test setup looks like, but if we want to have at least 3 control plane Pods on different nodes, then we will need to at least add one more worker Node.