[Failing Test] CSI volume tests failing in ci-kubernetes-e2e-gci-gce-alpha-features

[alejandrox1]

Which jobs are failing:
ci-kubernetes-e2e-gci-gce-alpha-features

Which test(s) are failing:

• [sig-storage] CSI mock volume CSI Volume expansion [Feature:ExpandCSIVolumes] should expand volume by restarting pod if attach=off, nodeExpansion=on
• [sig-storage] CSI mock volume CSI online volume expansion [Feature:ExpandCSIVolumes][Feature:ExpandInUseVolumes] should expand volume without restarting pod if attach=on, nodeExpansion=on
• [sig-storage] CSI mock volume CSI Volume expansion [Feature:ExpandCSIVolumes] should not expand volume if resizingOnDriver=off, resizingOnSC=on
• [sig-storage] CSI Volumes [Driver: csi-hostpath] [Testpattern: Dynamic PV (default fs)] provisioning should provision storage with snapshot data source [Feature:VolumeSnapshotDataSource]
• [sig-storage] CSI Volumes [Driver: csi-hostpath] [Testpattern: Dynamic Snapshot] snapshottable should create snapshot with defaults [Feature:VolumeSnapshotDataSource]
• [sig-storage] CSI mock volume CSI Volume expansion [Feature:ExpandCSIVolumes] should expand volume without restarting pod if nodeExpansion=off
• [sig-storage] CSI mock volume CSI Volume expansion [Feature:ExpandCSIVolumes] should expand volume by restarting pod if attach=on, nodeExpansion=on

Since when has it been failing:
Failing since 8/8 at around 4pm PDT
See 6d49d69...ef88694?
Possible cause #80978

Testgrid link:
https://testgrid.k8s.io/sig-release-master-blocking#gce-cos-master-alpha-features

Reason for failure:
All sig-storage relatedfailures are showing up with a message like the one below:

[sig-storage] CSI Volumes [Driver: csi-hostpath] [Testpattern: Dynamic Snapshot] snapshottable should create snapshot with defaults [Feature:VolumeSnapshotDataSource] 

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/snapshottable.go:96 Unexpected error:
     <*errors.errorString \| 0xc0007ff4b0>: {
         s: "PersistentVolumeClaims [pvc-2kx7c] not all in phase Bound within 5m0s",
     }     PersistentVolumeClaims [pvc-2kx7c] not all in phase Bound within 5m0s occurred 
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/snapshottable.go:137

/milestone v1.16
/priority critical-urgent
/kind failing-test
/sig testing
/sig sig-storage
/cc @kubernetes/sig-storage-test-failures
/cc @Verolop @jimangel @soggiest @alenkacz

[msau42]

Yes #80978 is very related. We will need to pick up a new client-go from master with the fix in our csi sidecars, release new sidecars, and then update the images used in the test.

/assign
/cc @wojtek-t

[msau42]

Fix part 1 is at kubernetes-csi/external-provisioner#323

[wojtek-t]

I completely forgot about the existence of alpha-features test suite (we were aware that CSI storage features will stop passing).

#81211 is a mitigation if we need more time to fix those tests (we know what we should do, just requires some time).

@msau42 - thanks a lot for working on that

[msau42]

Part 2 of fix: #81296

This should solve the release blocking failures. Remaining failures in alpha non-blocking job are:

• hostpath-v0: This is testing CSI 0.3 driver, which we deprecated in 1.13, so should be removed by 1.17/18. We don't intend to backport the fix to 0.3 drivers. We may just want to remove hostpath-v0 testing from Kubernetes altogether.

• external provisioner (non-csi) tests: This requires an update and new release of the nfs external provisioner. I've asked @childsb to look into this.

[msau42]

Part 1b of fix: kubernetes-csi/external-snapshotter#160

[msau42]

alpha-features job is passing now. For the remaining failures in alpha-enabled job, I've opened up #81401 and #81403

/close

[k8s-ci-robot]

@msau42: Closing this issue.

In response to this:

alpha-features job is passing now. For the remaining failures in alpha-enabled job, I've opened up #81401 and #81403

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.