After updating to GKE 1.32.4 seeing "Error updating volume group's metadata: vgck --updatemetadata csi-vg" errors

[elatov]

I recently noticed that I am see the following errors in the pdcsi-node pods in my GKE cluster:

> k logs -n kube-system pdcsi-node-8dpg7 -c gce-pd-driver | tail
I0610 01:00:02.238518      12 cache.go:407] check vg exists output: , volumeGroupName: csi-vg-bwqzk8cq
I0610 01:00:02.238555      12 cache.go:494] Volume group csi-vg-bwqzk8cq not found, no further action needed
E0610 01:00:04.328640      12 cache.go:658] Error updating volume group's metadata: vgck --updatemetadata csi-vg-bwqzk8cq failed here 2: exit status 5; output:   Volume group "csi-vg-bwqzk8cq" not found
  Cannot process volume group csi-vg-bwqzk8cq
I0610 01:00:04.377123      12 cache.go:407] check vg exists output: , volumeGroupName: csi-vg-bwqzk8cq
I0610 01:00:04.378183      12 cache.go:494] Volume group csi-vg-bwqzk8cq not found, no further action needed
E0610 01:00:04.412672      12 cache.go:658] Error updating volume group's metadata: vgck --updatemetadata csi-vg-bwqzk8cq failed here 2: exit status 5; output:   Volume group "csi-vg-bwqzk8cq" not found
  Cannot process volume group csi-vg-bwqzk8cq
I0610 01:00:04.462798      12 cache.go:407] check vg exists output: , volumeGroupName: csi-vg-bwqzk8cq
I0610 01:00:04.462956      12 cache.go:494] Volume group csi-vg-bwqzk8cq not found, no further action needed

I am guessing it might be related to the Data Cache feature, but I don't have that enabled since I just using the standard-rwo storage class:

> k get pvc config-pvc
NAME                STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   VOLUMEATTRIBUTESCLASS   AGE
config-pvc   Bound    pvc-1b9b2f33-21dc-44c1-9d92-faa609b6c4f1   1Gi        RWO            standard-rwo   <unset>                 2y210d

It looks like the error is coming from https://github.com/kubernetes-sigs/gcp-compute-persistent-disk-csi-driver/blob/master/pkg/gce-pd-csi-driver/cache.go#L657. I wonder if we can add some sort of check to make sure cache is enabled prior to running that command?

[fdewinne]

I get exactly the same error for 2 days now. What is really strange is the volume group name is exactly the same on every nodes and on different clusters

[vladorf]

What is really strange is the volume group name is exactly the same on every nodes and on different clusters

Even across different GCP accounts, looks like 🤔 Our error messages contain exactly the same volumeGroupName

[kkrtbhouse]

We are observing the same issue on clusters that have been upgraded to v1.32.4-gke.1236007. Clusters that are still on v1.32.3-gke.1785003 does not produce this error log.

[JerbiAhmed]

Same Volume group csi-vg-bwqzk8cq not found error after upgrade cluster to 1.32.4-gke.1236007

[campee]

I'm seeing the same error in all my projects after upgrading to 1.32.4-gke.1236007: Cannot process volume group csi-vg-bwqzk8cq

[hrvalca]

Had the same issue on multiple clusters - after upgrading to 1.32.4-gke.1353003 errors stopped.

[elatov]

same here, I will close this, since this is fixed in later versions.