cinder-csi-plugin@v1.23.3 fails with "x509: certificate signed by unknown authority"

[ialidzhikov]

Is this a BUG REPORT or FEATURE REQUEST?:
/kind bug

What happened:
I am trying to update out cinder-csi-plugin version from v1.23.0 to v1.23.3 (for K8s 1.23 clusters). We are running docker.io/k8scloudprovider/cinder-csi-plugin@v1.23.0 without issues more than a year.
When I tried v1.23.3, it fails with:

Flag --nodeid has been deprecated, This flag would be removed in future. Currently, the value is ignored by the driver
I0722 07:40:09.034911       1 driver.go:75] Driver: cinder.csi.openstack.org
I0722 07:40:09.034930       1 driver.go:76] Driver version: 2.0.0@v1.23.3
I0722 07:40:09.034933       1 driver.go:77] CSI Spec version: 1.3.0
I0722 07:40:09.034938       1 driver.go:107] Enabling controller service capability: LIST_VOLUMES
I0722 07:40:09.034941       1 driver.go:107] Enabling controller service capability: CREATE_DELETE_VOLUME
I0722 07:40:09.034943       1 driver.go:107] Enabling controller service capability: PUBLISH_UNPUBLISH_VOLUME
I0722 07:40:09.034945       1 driver.go:107] Enabling controller service capability: CREATE_DELETE_SNAPSHOT
I0722 07:40:09.034947       1 driver.go:107] Enabling controller service capability: LIST_SNAPSHOTS
I0722 07:40:09.034949       1 driver.go:107] Enabling controller service capability: EXPAND_VOLUME
I0722 07:40:09.034951       1 driver.go:107] Enabling controller service capability: CLONE_VOLUME
I0722 07:40:09.034953       1 driver.go:107] Enabling controller service capability: LIST_VOLUMES_PUBLISHED_NODES
I0722 07:40:09.034955       1 driver.go:107] Enabling controller service capability: GET_VOLUME
I0722 07:40:09.034958       1 driver.go:119] Enabling volume access mode: SINGLE_NODE_WRITER
I0722 07:40:09.034960       1 driver.go:129] Enabling node service capability: STAGE_UNSTAGE_VOLUME
I0722 07:40:09.034963       1 driver.go:129] Enabling node service capability: EXPAND_VOLUME
I0722 07:40:09.034965       1 driver.go:129] Enabling node service capability: GET_VOLUME_STATS
I0722 07:40:09.034970       1 openstack.go:137] InitOpenStackProvider configFiles: [/etc/kubernetes/cloudprovider/cloudprovider.conf]
I0722 07:40:09.035170       1 openstack.go:90] Block storage opts: {0 true false}
I0722 07:40:09.035194       1 client.go:252] Using user-agent shoot--foo--bar foo-team cp cinder-csi-plugin/v1.23.3 gophercloud/2.0.0
W0722 07:40:09.041370       1 main.go:100] Failed to GetOpenStackProvider: Post "<auth-url>": x509: certificate signed by unknown authority

I checked the diff between v1.23.0..v1.23.3, but I cannot spot a change that would cause such failure. I don't have any change to the configuration - the same v1.23.0 configuration is used.

The more interesting thing is that I don't face such issue when I build v1.23.3 cinder-csi-plugin image from source.

$ git checkout v1.23.3
$ make image-csi-plugin GOOS=linux

An example image that works without issues - innoweek/cinder-csi-plugin-amd64:v1.23.3.

I assume there is something wrong with docker.io/k8scloudprovider/cinder-csi-plugin:v1.23.3. Can it be the case the binary in this image does not correspond to the v1.23.3 tag? Otherwise I cannot explain why it works for my custom built image and does not work for the upstream one.

What you expected to happen:
docker.io/k8scloudprovider/cinder-csi-plugin:v1.23.3 to work without issues ("x509: certificate signed by unknown authority").

How to reproduce it:
See above.

Anything else we need to know?:

Environment:

• openstack-cloud-controller-manager(or other related binary) version: v1.23.3
• OpenStack version:
• Others:

[ialidzhikov]

/sig storage

[jichenjc]

I also found related issue #1938
and seem also suffer an issue that I don't know why .. maybe I didn't follow the exact build process (sorry about that)

do you might I create a new rc release and you give a try first before we upgrade to 1.23.4 ? @ialidzhikov ?

[ialidzhikov]

do you might I create a new rc release and you give a try first before we upgrade to 1.23.4 ? @ialidzhikov ?

Sounds good. Sure, I will test the new rc on our side and provide feedback whether it fixes the issue or not.

[jichenjc]

@ialidzhikov please try 1.23.4-rc1 and see whether it can fix your problem .. didn't do anything other than just do the release procedure again... at least previous issue on #1938 seems ok now on my local env

[ialidzhikov]

I can confirm that 1.23.4-rc1 works fine and this issue is not reproducible with it.

[xmudrii]

I can also confirm that 1.23.4-rc1 works as expected and that the issue is not reproducible with it.

[jichenjc]

https://github.com/kubernetes/cloud-provider-openstack/releases/tag/v1.23.4 is created and
I ran docker run -it k8scloudprovider/cinder-csi-plugin:v1.23.4 which blkid prove to contains blkid output, so hopefully this is better than 1.23.3

root@jjtest1:~/go/src/github.com/cloud-provider-openstack# docker run -it k8scloudprovider/cinder-csi-plugin:v1.23.4 which blkid
/sbin/blkid

[ialidzhikov]

v1.23.4 image works fine for me. Thank you @jichenjc!

/close

[k8s-ci-robot]

@ialidzhikov: Closing this issue.

In response to this:

v1.23.4 image works fine for me. Thank you @jichenjc!

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.