Add the ability to configure pod anti-affinity in the cStor helm chart

[dborman-hpe]

We typically run with a 3-worker node Kubernetes cluster, and run the cstor services with 3 replicas for resiliency. Usually Kubernetes will spread the 3 pods across the 3 worker nodes. However, if a worker node should go down, Kubernetes will schedule a replacement pod on one of the other worker nodes. This puts things into an un-balanced situation where one worker is running two replicas for the service. We would like to be able to turn on pod anti-affinity for these services so that when one worker node goes down, when Kubernetes schedules a replacement pod the pod will get stuck in Pending state until the missing worker node returns, and will then get scheduled on the third node. This would avoid the need to do manual pod rebalancing after a worker node outage.

Here is an example of what is running in our cluster:

NAME                                                              READY   STATUS    RESTARTS   AGE
openebs-cstor-admission-server-7444898b4c-b2lf4                   1/1     Running   0          3d16h
openebs-cstor-admission-server-7444898b4c-n6jlz                   1/1     Running   0          3d16h
openebs-cstor-admission-server-7444898b4c-z5bgq                   1/1     Running   0          3d16h
openebs-cstor-csi-controller-0                                    6/6     Running   3          3d16h
openebs-cstor-csi-controller-1                                    6/6     Running   3          3d16h
openebs-cstor-csi-controller-2                                    6/6     Running   4          3d16h
openebs-cstor-csi-node-9lqkg                                      2/2     Running   0          3d16h
openebs-cstor-csi-node-s7rcv                                      2/2     Running   0          3d16h
openebs-cstor-csi-node-tbwbc                                      2/2     Running   0          3d16h
openebs-cstor-cspc-operator-7976fd9b5f-2kwt7                      1/1     Running   0          3d16h
openebs-cstor-cspc-operator-7976fd9b5f-6nm6l                      1/1     Running   0          3d16h
openebs-cstor-cspc-operator-7976fd9b5f-rlbrj                      1/1     Running   0          3d16h
openebs-cstor-cvc-operator-64d8d94cdd-2tqgv                       1/1     Running   0          3d16h
openebs-cstor-cvc-operator-64d8d94cdd-9s8xr                       1/1     Running   0          3d16h
openebs-cstor-cvc-operator-64d8d94cdd-gl2mw                       1/1     Running   0          3d16h
openebs-cstor-pool-5bvw-c846c746b-8qphn                           3/3     Running   0          3d16h
openebs-cstor-pool-7n7b-894f94489-lr27g                           3/3     Running   0          3d16h
openebs-cstor-pool-bvdj-794ccf597-m2f8p                           3/3     Running   0          3d16h
openebs-localpv-provisioner-84c99fb7ff-4dc99                      1/1     Running   0          3d16h
openebs-localpv-provisioner-84c99fb7ff-n4zb9                      1/1     Running   1          3d16h
openebs-localpv-provisioner-84c99fb7ff-xxrsp                      1/1     Running   0          3d16h
openebs-ndm-9kjrp                                                 1/1     Running   0          3d16h
openebs-ndm-cbfwj                                                 1/1     Running   0          3d16h
openebs-ndm-nc2wq                                                 1/1     Running   0          3d16h
openebs-ndm-operator-5df4d5bfd4-dxfmk                             1/1     Running   0          3d16h
openebs-ndm-operator-5df4d5bfd4-gbmcd                             1/1     Running   0          3d16h
openebs-ndm-operator-5df4d5bfd4-q8v7x                             1/1     Running   0          3d16h