Generate events on PV when volume cannot be created for static volumes

[skogta]

What this PR does / why we need it:

If static volume provisioning fails for a PV, then there is now ay for the user to verify that it has failed other than checking syncer logs. So in this PR, we are generating failure event for static volumes.

Note: There are only 2 types events: https://pkg.go.dev/k8s.io/api/core/v1
So setting the type to Warning in this case.

Also, as part of this change, I am changing the method getPVsInBoundAvailableOrReleasedForVc to not return error if a volume cannot be mapped to a VC. One bad volume should not block the rest of the full sync.

Testing done:
On a multi VC testbed, created a RWX static volume with incorrect VolumeHandle that does not exist.
PV describe output:

root@k8s-control-556-1702644001:~# kubectl edit pv pvc-4810dd5e-b51a-46d5-bbd2-f97c90a647ec
persistentvolume/pvc-4810dd5e-b51a-46d5-bbd2-f97c90a647ec edited
root@k8s-control-556-1702644001:~# kubectl describe pv 14-file-static-file-share-pv-name
Name:            14-file-static-file-share-pv-name
Labels:          static-pv-label-key=static-pv-label-value
Annotations:     pv.kubernetes.io/provisioned-by: csi.vsphere.vmware.com
Finalizers:      [kubernetes.io/pv-protection]
StorageClass:    
Status:          Available
Claim:           
Reclaim Policy:  Retain
Access Modes:    RWO
VolumeMode:      Filesystem
Capacity:        1Gi
Node Affinity:   <none>
Message:         
Source:
    Type:              CSI (a Container Storage Interface (CSI) volume source)
    Driver:            csi.vsphere.vmware.com
    FSType:            
    VolumeHandle:      jkcns-jkncsk-3722
    ReadOnly:          false
    VolumeAttributes:      type=vSphere CNS File Volume
Events:
  Type     Reason                             Age   From                Message
  ----     ------                             ----  ----                -------
  Warning  static volume provisioning failed  17m   VSphere CSI Syncer  Failed to identify VC for volume

On multi VC testbed, created a block volume without nodeaffinity rules

root@k8s-control-556-1702644001:~# kubectl describe pv 10-block-static-file-share-pv-name



Name:            10-block-static-file-share-pv-name
Labels:          static-pv-label-key=static-pv-label-value
Annotations:     pv.kubernetes.io/provisioned-by: csi.vsphere.vmware.com
Finalizers:      [kubernetes.io/pv-protection]
StorageClass:    
Status:          Available
Claim:           
Reclaim Policy:  Retain
Access Modes:    RWO
VolumeMode:      Filesystem
Capacity:        1Gi
Node Affinity:   <none>
Message:         
Source:
    Type:              CSI (a Container Storage Interface (CSI) volume source)
    Driver:            csi.vsphere.vmware.com
    FSType:            
    VolumeHandle:      file:ocfd3946-bc5f-4c73-acca-95daae477f80
    ReadOnly:          false
    VolumeAttributes:      type=vSphere CNS File Volume
Events:
  Type     Reason                             Age   From                Message
  ----     ------                             ----  ----                -------
  Warning  static volume provisioning failed  15s   VSphere CSI Syncer  Failed to identify VC for volume.

Single VC failure

root@k8s-control-409-1703204631:~# kubectl describe pv static-file-share-pv-name
Name:            static-file-share-pv-name
Labels:          static-pv-label-key=static-pv-label-value
Annotations:     pv.kubernetes.io/provisioned-by: csi.vsphere.vmware.com
Finalizers:      [kubernetes.io/pv-protection]
StorageClass:    
Status:          Available
Claim:           
Reclaim Policy:  Retain
Access Modes:    RWX
VolumeMode:      Filesystem
Capacity:        1Gi
Node Affinity:   <none>
Message:         
Source:
    Type:              CSI (a Container Storage Interface (CSI) volume source)
    Driver:            csi.vsphere.vmware.com
    FSType:            
    VolumeHandle:      file:236b3e6b-cfb0-4b73-a271-2591b2f31b4c
    ReadOnly:          false
    VolumeAttributes:      type=vSphere CNS File Volume
Events:
  Type     Reason                             Age   From                Message
  ----     ------                             ----  ----                -------
  Warning  static volume provisioning failed  14s   VSphere CSI Syncer  Failed to identify VC for volume

On a single VC, gave wrong volume handle in static PV:

root@k8s-control-556-1702644001:~# kubectl describe pv 13-block-static-file-share-pv-name
Name:              13-block-static-file-share-pv-name
Labels:            static-pv-label-key=static-pv-label-value
Annotations:       pv.kubernetes.io/provisioned-by: csi.vsphere.vmware.com
Finalizers:        [kubernetes.io/pv-protection]
StorageClass:      
Status:            Available
Claim:             
Reclaim Policy:    Retain
Access Modes:      RWO
VolumeMode:        Filesystem
Capacity:          1Gi
Node Affinity:     
  Required Terms:  
    Term 0:        topology.csi.vmware.com/k8s-zone in [zone-2]
Message:           
Source:
    Type:              CSI (a Container Storage Interface (CSI) volume source)
    Driver:            csi.vsphere.vmware.com
    FSType:            
    VolumeHandle:      jkcns-jkncsk-3722
    ReadOnly:          false
    VolumeAttributes:      type=vSphere CNS File Volume
Events:
  Type     Reason                             Age   From                Message
  ----     ------                             ----  ----                -------
  Warning  static volume provisioning failed  63s   VSphere CSI Syncer  Failed to create volume on VC 10.212.239.36

Special notes for your reviewer:

Release note:

[k8s-ci-robot]

Hi @skogta. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[divyenpatel]

/ok-to-test

[divyenpatel]

Can you address review comments?

[divyenpatel]

/approve
/lgtm

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: divyenpatel, skogta

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [divyenpatel]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment