[cinder-csi-plugin]retrieving availability zone from MetaData service failed with error unexpected end of JSON input

[liupeng0518]

Is this a BUG REPORT or FEATURE REQUEST?:

Uncomment only one, leave it on its own line:

/kind bug
/kind feature

What happened:

[root@k8s1 cinder-csi-plugin]# kubectl logs -f -n kube-system csi-cinder-nodeplugin-jmb47   
error: a container name must be specified for pod csi-cinder-nodeplugin-jmb47, choose one of: [node-driver-registrar cinder-csi-plugin]
[root@k8s1 cinder-csi-plugin]# kubectl logs -f -n kube-system csi-cinder-nodeplugin-jmb47    cinder-csi-plugin 
I0721 05:42:43.750055       1 driver.go:57] Driver: cinder.csi.openstack.org version: 1.2.0
I0721 05:42:43.750165       1 driver.go:91] Enabling controller service capability: LIST_VOLUMES
I0721 05:42:43.750211       1 driver.go:91] Enabling controller service capability: CREATE_DELETE_VOLUME
I0721 05:42:43.750216       1 driver.go:91] Enabling controller service capability: PUBLISH_UNPUBLISH_VOLUME
I0721 05:42:43.750220       1 driver.go:91] Enabling controller service capability: CREATE_DELETE_SNAPSHOT
I0721 05:42:43.750226       1 driver.go:91] Enabling controller service capability: LIST_SNAPSHOTS
I0721 05:42:43.750229       1 driver.go:91] Enabling controller service capability: EXPAND_VOLUME
I0721 05:42:43.750233       1 driver.go:91] Enabling controller service capability: CLONE_VOLUME
I0721 05:42:43.750237       1 driver.go:103] Enabling volume access mode: SINGLE_NODE_WRITER
I0721 05:42:43.750242       1 driver.go:113] Enabling node service capability: STAGE_UNSTAGE_VOLUME
I0721 05:42:43.750246       1 driver.go:113] Enabling node service capability: EXPAND_VOLUME
I0721 05:42:43.750863       1 openstack.go:88] Block storage opts: {0 false}
I0721 05:42:44.346342       1 server.go:108] Listening for connections on address: &net.UnixAddr{Name:"/csi/csi.sock", Net:"unix"}
E0721 05:42:46.371536       1 utils.go:83] GRPC error: rpc error: code = Internal desc = retrieving availability zone from MetaData service failed with error unexpected end of JSON input
E0721 05:42:50.362550       1 utils.go:83] GRPC error: rpc error: code = Internal desc = retrieving availability zone from MetaData service failed with error unexpected end of JSON input
E0721 05:43:07.372313       1 utils.go:83] GRPC error: rpc error: code = Internal desc = retrieving availability zone from MetaData service failed with error unexpected end of JSON input
E0721 05:43:35.377008       1 utils.go:83] GRPC error: rpc error: code = Internal desc = retrieving availability zone from MetaData service failed with error unexpected end of JSON input
E0721 05:44:30.400359       1 utils.go:83] GRPC error: rpc error: code = Internal desc = retrieving availability zone from MetaData service failed with error unexpected end of JSON input

What you expected to happen:

How to reproduce it:

Anything else we need to know?:

Environment:

• openstack-cloud-controller-manager(or other related binary) version:

[root@k8s1 cinder-csi-plugin]# grep -r image:
cinder-csi-nodeplugin.yaml:          image: quay.io/k8scsi/csi-node-driver-registrar:v1.2.0
cinder-csi-nodeplugin.yaml:          image: docker.io/k8scloudprovider/cinder-csi-plugin:v1.18.0
cinder-csi-controllerplugin.yaml:          image: quay.io/k8scsi/csi-attacher:v2.1.1
cinder-csi-controllerplugin.yaml:          image: quay.io/k8scsi/csi-provisioner:v1.4.0
cinder-csi-controllerplugin.yaml:          image: quay.io/k8scsi/csi-snapshotter:v1.2.2
cinder-csi-controllerplugin.yaml:          image: quay.io/k8scsi/csi-resizer:v0.4.0
cinder-csi-controllerplugin.yaml:          image: docker.io/k8scloudprovider/cinder-csi-plugin:v1.18.0

• OpenStack version: R
• Others:
  k8s v1.18.3

Thanks.

[liupeng0518]

debug:

I0722 02:23:04.262855       1 server.go:108] Listening for connections on address: &net.UnixAddr{Name:"/csi/csi.sock", Net:"unix"}
I0722 02:23:04.865934       1 utils.go:79] GRPC call: /csi.v1.Identity/GetPluginInfo
I0722 02:23:04.865956       1 utils.go:80] GRPC request: 
I0722 02:23:04.866051       1 identityserver.go:33] Using default GetPluginInfo
I0722 02:23:04.866059       1 utils.go:85] GRPC response: name:"cinder.csi.openstack.org" vendor_version:"1.2.0" 
I0722 02:23:05.045032       1 utils.go:79] GRPC call: /csi.v1.Node/NodeGetInfo
I0722 02:23:05.045070       1 utils.go:80] GRPC request: 
I0722 02:23:05.392239       1 mount.go:226] Got instance id from /var/lib/cloud/data/instance-id: iid-datasource-none
I0722 02:23:05.392308       1 nodeserver.go:541] getNodeIDMountProvider return node id iid-datasource-none
I0722 02:23:05.462180       1 utils.go:85] GRPC response: node_id:"iid-datasource-none" max_volumes_per_node:256 accessible_topology:<segments:<key:"topology.cinder.csi.openstack.org/zone" value:"" > > 
^C

[ramineni]

@liupeng0518 Is metadata service available on nodes?
Try running curl http://169.254.169.254/openstack/latest/meta_data.json on the node and see if it gets o/p

[liupeng0518]

@liupeng0518 Is metadata service available on nodes?
Try running curl http://169.254.169.254/openstack/latest/meta_data.json on the node and see if it gets o/p

Thanks.