Enable adding annotations of PVCs

[shtripat]

Enable adding of annotations for PVCs through helm charts.

Fixes: #1868

How to verify the PR

Follow the below steps

Step-1: Build the docker image with custom tag

TAG=minio/operator:pvc-ann make

Step-2: Create a kind cluster and load the image to it

kind load docker-image docker.io/minio/operator:pvc-ann

Step-3: Generate operator deployment YAML using kustomize and deploy using it

kubectl kustomize . > operator-test.yaml

Edit the YAML and set the operator and console image values as docker.io/minio/operator:pvc-ann

kubectl apply -f operator-test.yaml

Step-4: Add custom annotations for MinIO tenant under helm/tenant/values.yaml as field tenant.pools.[].StorageAnnotations like

      storageAnnotations:
        field1: "val1"
        field2: "val2"

Step-5: Deploy the tenant

$ kubectl create ns new-tenant
$ helm install -n new-tenant ./helm/tenant/ -f ./helm/tenant/values.yaml --generate-name

Step-6: Verify the PVCs annotations

$ kubectl get pvc -n new-tenant 
NAME                     STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
data0-myminio-pool-0-0   Bound    pvc-d3e4a8e3-db96-48ed-9bbb-d36d55d9b782   10Gi       RWO            standard       17m
data0-myminio-pool-0-1   Bound    pvc-1dcb020a-c674-4ce0-a469-2bc5b9ac987e   10Gi       RWO            standard       17m
data0-myminio-pool-0-2   Bound    pvc-89ad581d-0764-4d37-ae9b-2e0c7373f5ce   10Gi       RWO            standard       17m
data0-myminio-pool-0-3   Bound    pvc-38e60ec7-7ea5-4c45-8a7b-6fc8fff772e3   10Gi       RWO            standard       17m
data1-myminio-pool-0-0   Bound    pvc-5b2a8480-76d6-4f75-a278-d5206c3a50b7   10Gi       RWO            standard       17m
data1-myminio-pool-0-1   Bound    pvc-773e3746-1cf3-4f5c-9cbd-a13cc6b216ce   10Gi       RWO            standard       17m
data1-myminio-pool-0-2   Bound    pvc-771da73f-8a22-401c-8b3b-21a71326fe47   10Gi       RWO            standard       17m
data1-myminio-pool-0-3   Bound    pvc-b3598c19-3eb3-4168-b66a-40791945c21d   10Gi       RWO            standard       17m
data2-myminio-pool-0-0   Bound    pvc-1dfca35f-b9a1-4182-8b19-634bb6515e2e   10Gi       RWO            standard       17m
data2-myminio-pool-0-1   Bound    pvc-5effd3ae-64f9-49d6-ab27-91af901ba4fe   10Gi       RWO            standard       17m
data2-myminio-pool-0-2   Bound    pvc-198ae507-314a-4a6d-9521-164cf2c094de   10Gi       RWO            standard       17m
data2-myminio-pool-0-3   Bound    pvc-948d830e-da73-40ee-a1f0-945ec168517d   10Gi       RWO            standard       17m
data3-myminio-pool-0-0   Bound    pvc-a26b75ae-22c0-4f6a-9ec7-0a1ebec86de6   10Gi       RWO            standard       17m
data3-myminio-pool-0-1   Bound    pvc-336064f1-c239-4b9c-866e-eda31b0faae7   10Gi       RWO            standard       17m
data3-myminio-pool-0-2   Bound    pvc-e119f87b-0f60-4908-aa2a-d8cd6cd05fc6   10Gi       RWO            standard       17m
data3-myminio-pool-0-3   Bound    pvc-233569e6-362e-45b5-94e5-71a07ba6e091   10Gi       RWO            standard       17m

$ k get pvc/data0-myminio-pool-0-0 -n new-tenant -oyaml | yq '.metadata.annotations' 
field1: val1 <<<===== NEW ANNOTATION
field2: val2 <<<===== NEW ANNOTATION
pv.kubernetes.io/bind-completed: "yes"
pv.kubernetes.io/bound-by-controller: "yes"
volume.beta.kubernetes.io/storage-provisioner: rancher.io/local-path
volume.kubernetes.io/selected-node: kind-worker
volume.kubernetes.io/storage-provisioner: rancher.io/local-path