unable to deploy vSphere CSI Driver using VDO Operator

[divyenpatel]

Issue/Feature Description:
Deploy vSphere CSI Driver using VDO Operator.
Deployment method -Deployment via Releases - https://github.com/vmware-tanzu/vsphere-kubernetes-drivers-operator/blob/main/docs/getting-started/getting-started-from-release.md

What type of Issue/Feature is this?
/kind documentation

Why this fix/ feature is needed(give scenarios or use cases):

Issue:1

Update latest version vdo-spec file.
Users need to figure out vdo spec path from latest tag manually

./vdoctl deploy --spec https://raw.githubusercontent.com/vmware-tanzu/vsphere-kubernetes-drivers-operator/v0.6.0/artifacts/vanilla/vdo-spec.yaml
Tip: now that you have deployed VDO, you might want to try 'vdoctl configure drivers' to configure vsphere drivers

Issue:2

After executing above command, when user tries to execute vdoctl configure drivers, command fails with

./vdoctl configure drivers
Error: not enough replicas of VDO

looking at the help for the command figured out that vdoctl.yaml is required at $HOME/.vdoctl.yaml
but could not find example $HOME/.vdoctl.yaml file anywhere in this repository.

Issue:3

As per workflow diagram - https://github.com/vmware-tanzu/vsphere-kubernetes-drivers-operator/blob/main/docs/getting-started/getting-started-from-release.md#user-flow

after executing vdoctl deploy, we need to execute vdoctl configure, but i got this error.

./vdoctl configure compatibility-matrix
Compatibility matrix is already configured. You can use `vdoctl update matrix` to update compatibility matrix

How to reproduce, in case of a bug:
Just follow the documentation and attempt to install driver using VDO operator without having knowledge of internal implementation detail.

Can you please update documentation for non-openshift environment so we can consume VDO Operator and in future recommend vdo operator for vanilla customer.

Other Notes / Environment Information: (Please give the env information, log link or any useful information for this issue)

# kubectl version
Client Version: v1.28.0
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.28.0

vSphere 8.0

[divyenpatel]

cc: @asifdxtreme

[divyenpatel]

@asifdxtreme what is the latest vSphere CSI Driver release we support?

I have some more questions regarding VDO support.

Do VDO support installation of driver on topology aware setup? topology aware setup requires specific configuration in the vSphere Config secret - https://docs.vmware.com/en/VMware-vSphere-Container-Storage-Plug-in/3.0/vmware-vsphere-csp-getting-started/GUID-162E7582-723B-4A0F-A937-3ACE82EAFD31.html

Do VDO Support Kubernetes Cluster spread across multiple vCenter servers?
https://docs.vmware.com/en/VMware-vSphere-Container-Storage-Plug-in/3.0/vmware-vsphere-csp-getting-started/GUID-8B3B9004-DE37-4E6B-9AA1-234CDA1BD7F9.html

Do VDO recognize that we have windows workload nodes and deploy windows node daemonsets pods?
https://docs.vmware.com/en/VMware-vSphere-Container-Storage-Plug-in/3.0/vmware-vsphere-csp-getting-started/GUID-85203E57-71DD-42B7-AD98-8CE06AEDF4C3.html

[asifdxtreme]

@divyenpatel After running commands in Issue 1, can you please give the output of below commands
kubectl get pods -n vmware-system-vdo
Also can you please let us know what is k8s version being used here, if >= 1.25 then we need to add PSA label in the above namespace. From last few releases we have only focused on IBM Satellite, we need to check the support of VDO on vanilla k8s.
/cc @lgayatri

[divyenpatel]

#  kubectl get pods -n vmware-system-vdo
NAME                                     READY   STATUS    RESTARTS   AGE
vdo-controller-manager-64d748bb4-nfjqb   0/2     Pending   0          13h

looks like pod failed to schedule

Events:
  Type     Reason            Age                From               Message
  ----     ------            ----               ----               -------
  Warning  FailedScheduling  33m (x8 over 69m)  default-scheduler  0/6 nodes are available: 3 node(s) didn't match Pod's node affinity/selector, 3 node(s) had untolerated taint {node-role.kubernetes.io/control-plane: }. preemption: 0/6 nodes are available: 6 Preemption is not helpful for scheduling..

Yes k8s version is v1.28.0

[asifdxtreme]

Hi @divyenpatel ,
Can you please add the label node-role.kubernetes.io/control-plane: on control plane vm's , this was deperecated in 1.25 but till now Openshift didn't adopted this change in Openshift 4.12 so we didn't changed it, this will taken care in the next release. Till then please add this label in all of you CPVM

[divyenpatel]

@asifdxtreme
Can you share vdoctl.yaml file? Do we need this file for deployment? Please refer to Issue:2.
Also can you help answer questions posted here - #184 (comment)

[asifdxtreme]

Please use the below link based on your OS
https://github.com/vmware-tanzu/vsphere-kubernetes-drivers-operator/releases/download/0.3.0/vdoctl-linux-0.3.0.tar.gz
or
https://github.com/vmware-tanzu/vsphere-kubernetes-drivers-operator/releases/download/0.3.0/vdoctl-mac-0.3.0.tar.gz

[asifdxtreme]

Do VDO support installation of driver on topology aware setup? topology aware setup requires specific configuration in the vSphere Config secret
No
Do VDO Support Kubernetes Cluster spread across multiple vCenter servers?
Yes, while using the vdoctl you can configure multiple DC
Do VDO recognize that we have windows workload nodes and deploy windows node daemonsets pods?
No, we don't have support for this, we can add this in upcoming release

[divyenpatel]

Yes, while using the vdoctl you can configure multiple DC

Multiple Datacenter within VC is possible, but do we support adding multiple vCenter servers in the config secret?

[asifdxtreme]

Multiple Datacenter within VC is possible, but do we support adding multiple vCenter servers in the config secret?

Currently we don't support that, I think in my opinion these configuration are straight forward and if you can give some examples of this then we can add these support in our CRD and vdoctl

[divyenpatel]

@asifdxtreme is there a plan to support k8s 1.28?

I1129 17:18:16.607754       1 vdoconfig_controller.go:1220] controllers/VDOConfig "msg"="vSphere Versions "  "version"=["8.0.3"]
I1129 17:18:16.607900       1 vdoconfig_controller.go:1221] controllers/VDOConfig "msg"="k8s Versions "  "version"="1.28"
E1129 17:18:16.608131       1 vdoconfig_controller.go:1300] controllers/VDOConfig "msg"="Error occurred when fetching the CPI deployment yamls" "error"="could not fetch compatible CPI version for vSphere version and k8s version "  
E1129 17:18:16.608218       1 controller.go:302] controller-runtime/manager/controller/vdoconfig "msg"="Reconciler error" "error"="could not fetch compatible CPI version for vSphere version and k8s version " "name"="vdo-confige04796d6" "namespace"="vmware-system-vdo" "reconciler group"="vdo.vmware.com" "reconciler kind"="VDOConfig" 

[asifdxtreme]

We have added the support for CSI 3.0.0 which supports K8s 1.27, you can deploy the existing release to get this support. CSI 3.1 support will be added by next week which will have K8s 1.8 support