Adopt CSI for Azure extension beginning with Kubernetes 1.21

[rfranzke]

As of Kubernetes 1.18 the in-tree Azure Disk volume driver is deprecated in favour of CSI. We should adopt CSI for 1.18+ clusters (where the Azure Disk migration will be promoted to beta as well)

See also: kubernetes/kubernetes#85231 and https://kubernetes.io/blog/2019/12/09/kubernetes-1-17-feature-csi-migration-beta/

[rfranzke]

Adopting CSI together with support for 1.18 seems to be a good plan:

• The Azure Disk CSI driver migration feature is still alpha, but according to this document it will be promoted to beta with 1.18.
• The CSI migration process is not yet sufficiently documented (there is only https://kubernetes.io/blog/2019/12/09/kubernetes-1-17-feature-csi-migration-beta/ and the design proposal). SIG storage might work on a dedicated migration instruction document, let's see. Particularly, it seems that it is necessary to drain the worker nodes before enabling the CSIMigration* feature gates on the kubelet --> introducing the CSI migration together with a minor release update seems feasible as we get new nodes anyways here and don't need to implement special migration logic.
• There is an open bug with deleting volumes that have been provisioned with the in-tree volume plugin (i.e., prior CSI migration): Need to check migrated PVs get deleted through CSI if CSI migration was enabled post provisioning kubernetes/kubernetes#79043. SIG storage is working on a fix (and it will probably be cherry-picked to 1.17, but for the mentioned reasons above migrating to CSI with 1.18 seems to make most sense).

[rfranzke]

The CSI migration is beta with 1.17, though, the CSI plugins for Azure are not yet ready to cater with Gardener needs. We will have to contribute there before we can migrate to CSI. Hence, postponing this issue until 1.19.

[rfranzke]

I opened kubernetes-sigs/azuredisk-csi-driver#354 for the Azure Disk CSI plugin. After it's merged I'll use the same approach to enhance the Azure File CSI plugin.

[rfranzke]

Turns out that what I described with #3 (comment) was actually a misconfiguration during my tests - the Azure Disk CSI plugin already works for our scenario as it disables the metadata service in the controller service of the CSI driver. Hence kubernetes-sigs/azuredisk-csi-driver#354 is closed again for now.

However, it seems that the node service of the CSI driver also requires the client id/secret credentials in order to talk to the Azure API and find out information about certain disks. This would mean that we would have to expose the credentials in the shoot cluster which is undesired. I'm investigating further...

[rfranzke]

I checked a little bit more, and it seems that the checkDiskExists() function in the NodeGetVolumeStats function is the only place in the node service where the credentials are needed right now.
The Azure CSI maintainers offered to look into whether it is possible to get rid of this check. Afterwards it should be possible to start the CSI driver without client id/secret.

[rfranzke]

I played with the latest kubernetes-sigs/azuredisk-csi-driver@v0.7.0 and kubernetes-sigs/azurefile-csi-driver@v0.6.0 releases and it seems that the problems we faced have been resolved. This would mean that we can go ahead and introduce CSI (migration) already with 1.18 as initially planned.
I'll do some more testing tomorrow, and if successful I'll add the respective changes to the currently open 1.18 support PR #72. I'll also update this ticket with my results.

[rfranzke]

It turns out that the Azure CSI migration is still alpha with 1.18 (although suggested differently by https://kubernetes.io/blog/2019/12/09/kubernetes-1-17-feature-csi-migration-beta/), hence, let's plan for 1.19. There are still flaws in the Azure File CSI migration, the Azure Disk CSI migration worked so far.
See also kubernetes/enhancements#1490.

[rfranzke]

/reopen
We decided to postpone CSI migration for Azure to Kubernetes v1.21, so let's adapt the version constants in the code from 1.20 to 1.21.

[ialidzhikov]

We decided to postpone CSI migration for Azure to Kubernetes v1.21, so let's adapt the version constants in the code from 1.20 to 1.21.

Let's wait few more days to make sure that CSIMigrationAzureFile promotion to beta will be pushed back to v1.21 - ref kubernetes/kubernetes#96293. Theoretically it can get in for v1.20 via an Exception.