Support upgrade of pods

[amarts]

When the operator starts, check if config map is already present (in which case, its most probable case of upgrade). Just make sure to trigger kubectl apply on storage pods as they most probably need upgrade too.

Updates: #377 #352 and similar few
Signed-off-by: Amar Tumballi amar@kadalu.io

[amarts]

Tests pending, and hence in draft mode. Will be happy if someone tries this upgrade process locally too. (git clone the branch, build one version, now use the new version built to replace the running (test infra) instance, and you should see even the storage pods to be updated).

[amarts]

Did basic testing, and the upgrade is happening.

There are other issues which came up when I tested. The storage pods were not getting Terminated completely, and stayed in 'Terminating' state for more than 60secs (ie, the graceperiod), so I had to kill them manually (with kubectl delete pod -nkadalu storage-pool-pod-name), and the new pod with newer version came up.

The code to upgrade storage pods and CSI pods to latest version seems to be working fine :-)

Will merge this and continue on issues separately!