vSphere CSI driver is unable to detect not authenticated sessions

[gnufied]

We have seen error logs of following form keep repeating in the logs, where attach keeps failing with "NotAuthenticated" errors. But this is surprising because - Attach code, first checks if CNS session is valid, before attempting attach call, so call should not have even reached there. I am not yet sure if this related to - #2155

│2023-01-09T20:08:10.633998964Z {                                                                                                                                                                            │
│    "level":"error",                                                                                                                                                                                        │
│    "time":"2023-01-09T20:08:10.633767798Z",                                                                                                                                                                │
│    "caller":"volume/manager.go:617",                                                                                                                                                                       │
│    "msg":"CNS AttachVolume failed from vCenter \"foobar.com\" with err: ServerFaultCode: NotAuthenticated",                                                                               │
│    "TraceId":"8d3104c1-d1db-4ef1-bfa9-330ca15fc10f",                                                                                                                                                       │
│    "stacktrace":""                                                                                                                                                                                         │
│    sigs.k8s.io/vsphere-csi-driver/v2/pkg/common/cns-lib/volume.(*defaultManager).AttachVolume.func1                                                                                                        │
│       /go/src/github.com/kubernetes-sigs/vsphere-csi-driver/pkg/common/cns-lib/volume/manager.go:617                                                                                                       │
│    sigs.k8s.io/vsphere-csi-driver/v2/pkg/common/cns-lib/volume.(*defaultManager).AttachVolume                                                                                                              │
│       /go/src/github.com/kubernetes-sigs/vsphere-csi-driver/pkg/common/cns-lib/volume/manager.go:673                                                                                                       │
│    sigs.k8s.io/vsphere-csi-driver/v2/pkg/csi/service/common.AttachVolumeUtil                                                                                                                               │
│       /go/src/github.com/kubernetes-sigs/vsphere-csi-driver/pkg/csi/service/common/vsphereutil.go:590                                                                                                      │
│    sigs.k8s.io/vsphere-csi-driver/v2/pkg/csi/service/vanilla.(*controller).ControllerPublishVolume.func1                                                                                                   │
│       /go/src/github.com/kubernetes-sigs/vsphere-csi-driver/pkg/csi/service/vanilla/controller.go:1087                                                                                                     │
│    sigs.k8s.io/vsphere-csi-driver/v2/pkg/csi/service/vanilla.(*controller).ControllerPublishVolume                                                                                                         │
│       /go/src/github.com/kubernetes-sigs/vsphere-csi-driver/pkg/csi/service/vanilla/controller.go:1100                                                                                                     │
│    github.com/container-storage-interface/spec/lib/go/csi._Controller_ControllerPublishVolume_Handler.func1                                                                                                │
│       /go/src/github.com/kubernetes-sigs/vsphere-csi-driver/vendor/github.com/container-storage-interface/spec/lib/go/csi/csi.pb.go:5632                                                                   │
│    github.com/rexray/gocsi/middleware/serialvolume.(*interceptor).controllerPublishVolume                                                                                                                  │
│       /go/src/github.com/kubernetes-sigs/vsphere-csi-driver/vendor/github.com/rexray/gocsi/middleware/serialvolume/serial_volume_locker.go:120                                                             │
│    github.com/rexray/gocsi/middleware/serialvolume.(*interceptor).handle                                                                                                                                   │
│       /go/src/github.com/kubernetes-sigs/vsphere-csi-driver/vendor/github.com/rexray/gocsi/middleware/serialvolume/serial_volume_locker.go:86                                                              │
│    github.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1                                                                                                                                                │
│       /go/src/github.com/kubernetes-sigs/vsphere-csi-driver/vendor/github.com/rexray/gocsi/utils/utils_middleware.go:99                                                                                    │
│    github.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer.func1                                                                                                                      │
│       /go/src/github.com/kubernetes-sigs/vsphere-csi-driver/vendor/github.com/rexray/gocsi/middleware/specvalidator/spec_validator.go:178                                                                  │
│    github.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handle                                                                                                                                  │
│       /go/src/github.com/kubernetes-sigs/vsphere-csi-driver/vendor/github.com/rexray/gocsi/middleware/specvalidator/spec_validator.go:218                                                                  │
│    github.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer                                                                                                                            │
│       /go/src/github.com/kubernetes-sigs/vsphere-csi-driver/vendor/github.com/rexray/gocsi/middleware/specvalidator/spec_validator.go:177                                                                  │
│    github.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1                                                                                                                                                │
│       /go/src/github.com/kubernetes-sigs/vsphere-csi-driver/vendor/github.com/rexray/gocsi/utils/utils_middleware.go:99                                                                                    │
│    github.com/rexray/gocsi.(*StoragePlugin).injectContext                                                                                                                                                  │
│       /go/src/github.com/kubernetes-sigs/vsphere-csi-driver/vendor/github.com/rexray/gocsi/middleware.go:231                                                                                               │
│    github.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1                                                                                                                                                │
│       /go/src/github.com/kubernetes-sigs/vsphere-csi-driver/vendor/github.com/rexray/gocsi/utils/utils_middleware.go:99                                                                                    │
│    github.com/rexray/gocsi/utils.ChainUnaryServer.func2                                                                                                                                                    │
│       /go/src/github.com/kubernetes-sigs/vsphere-csi-driver/vendor/github.com/rexray/gocsi/utils/utils_middleware.go:106                                                                                   │
│    github.com/container-storage-interface/spec/lib/go/csi._Controller_ControllerPublishVolume_Handler                                                                                                      │
│       /go/src/github.com/kubernetes-sigs/vsphere-csi-driver/vendor/github.com/container-storage-interface/spec/lib/go/csi/csi.pb.go:5634                                                                   │
│    google.golang.org/grpc.(*Server).processUnaryRPC                                                                                                                                                        │
│       /go/src/github.com/kubernetes-sigs/vsphere-csi-driver/vendor/google.golang.org/grpc/server.go:1024                                                                                                   │
│    google.golang.org/grpc.(*Server).handleStream                                                                                                                                                           │
│       /go/src/github.com/kubernetes-sigs/vsphere-csi-driver/vendor/google.golang.org/grpc/server.go:1313                                                                                                   │
│    google.golang.org/grpc.(*Server).serveStreams.func1.1                                                                                                                                                   │
│       /go/src/github.com/kubernetes-sigs/vsphere-csi-driver/vendor/google.golang.org/grpc/server.go:722                                                                                                    │
│""

Which is kind of surprising because we do following call:

err = m.virtualCenter.ConnectCns(ctx)

Before calling task, err := m.virtualCenter.CnsClient.AttachVolume(ctx, cnsAttachSpecList) and hence if session is unauthenticated then m.virtualCenter.ConnectCns(ctx) should have failed.

This is happening with CSI driver version. - 2.5.1.

[chethanv28]

/assign @divyenpatel

Divyen, Can you help take a look and answer any queries about the above issue.

[divyenpatel]

@gnufied I think this commit (dd79f76) will help fix the issue you are facing -

meanwhile, we are also checking #2155

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[divyenpatel]

/remove-lifecycle rotten

[divyenpatel]

Issue is fixed in v2.7.2 and v3.0.x releases