chore: upgrade azuredisk csi driver to v1.28.3 (#190)

* upgrade azuredisk-csi-driver to v1.28.3 - Upstream AKS uses azuredisk-csi-driver v1.26.5 for k8s v1.26 and azuredisk-csi-driver v1.28.3 for k8s v1.27. - Upgrade azuredisk csi driver to v1.28.3 - Upgrade all components according to new v1.28.3 csi driver values: https://github.com/kubernetes-sigs/azuredisk-csi-driver/blob/master/charts/v1.28.3/azuredisk-csi-driver/values.yaml * update v1.28.3 csi driver template: csi-azuredisk-node-windows.yaml - https://github.com/kubernetes-sigs/azuredisk-csi-driver/blob/master/charts/v1.26.6/azuredisk-csi-driver/templates/csi-azuredisk-node-windows.yaml - https://github.com/kubernetes-sigs/azuredisk-csi-driver/blob/master/charts/v1.28.3/azuredisk-csi-driver/templates/csi-azuredisk-node-windows.yaml * update v1.28.3 csi driver template: csi-azuredisk-node.yaml - https://github.com/kubernetes-sigs/azuredisk-csi-driver/blob/master/charts/v1.26.6/azuredisk-csi-driver/templates/csi-azuredisk-node.yaml - https://github.com/kubernetes-sigs/azuredisk-csi-driver/blob/master/charts/v1.28.3/azuredisk-csi-driver/templates/csi-azuredisk-node.yaml * update v1.28.3 csi driver template: csi-azuredisk-controller.yaml - https://github.com/kubernetes-sigs/azuredisk-csi-driver/blob/master/charts/v1.26.6/azuredisk-csi-driver/templates/csi-azuredisk-controller.yaml - https://github.com/kubernetes-sigs/azuredisk-csi-driver/blob/master/charts/v1.28.3/azuredisk-csi-driver/templates/csi-azuredisk-controller.yaml * update v1.28.3 csi driver template: csi-snapshot-controller.yaml - https://github.com/kubernetes-sigs/azuredisk-csi-driver/blob/master/charts/v1.26.6/azuredisk-csi-driver/templates/csi-snapshot-controller.yaml - https://github.com/kubernetes-sigs/azuredisk-csi-driver/blob/master/charts/v1.28.3/azuredisk-csi-driver/templates/csi-snapshot-controller.yaml * update v1.28.3 csi driver template: crd-csi-snapshot.yaml - https://github.com/kubernetes-sigs/azuredisk-csi-driver/blob/master/charts/v1.26.6/azuredisk-csi-driver/templates/crd-csi-snapshot.yaml - https://github.com/kubernetes-sigs/azuredisk-csi-driver/blob/master/charts/v1.28.3/azuredisk-csi-driver/templates/crd-csi-snapshot.yaml * Update generated template * Fix syntax * Fix k8s_versions_test * Change addon template based on version * Fix lint * Fix GCR for unit tests * Update documentation for Windows HostProcess containers * Test PVC by default * Add IsAzureDiskCSIDriverVersionGe * Revert "Add IsAzureDiskCSIDriverVersionGe" This reverts commit 3c44032.
Azure · Oct 5, 2023 · 27a359a · 27a359a
1 parent d5fe389
commit 27a359a
Show file tree

Hide file tree

Showing 9 changed files with 1,178 additions and 359 deletions.
diff --git a/.pipelines/e2e-job-template.yaml b/.pipelines/e2e-job-template.yaml
@@ -15,6 +15,7 @@ parameters:
   stabilityIterations: 0
   windowsVHDBuild: ""
   linuxVHDBuild: ""
+  testPVC: true
 
 jobs:
   - job: ${{ parameters.name }}
@@ -55,6 +56,7 @@ jobs:
       LINUX_NODE_IMAGE_GALLERY: AKSeGallery
       LINUX_NODE_IMAGE_NAME: ubuntu-20.04
       LINUX_NODE_IMAGE_VERSION: ${{ parameters.linuxVHDBuild }}
+      TEST_PVC: ${{ parameters.testPVC }}
 
     steps:
       - template: e2e-step-template.yaml

diff --git a/.pipelines/pr-e2e.yaml b/.pipelines/pr-e2e.yaml
@@ -74,6 +74,7 @@ jobs:
       msiId: $(MSI_USER_ASSIGNED_ID_AKS_ENGINE_E2E)
       windowsVHDBuild: $(WINDOWS_VHD_BUILD)
       linuxVHDBuild: $(LINUX_VHD_BUILD)
+      testPVC: true
 
   - template: e2e-job-template.yaml
     parameters:
@@ -93,6 +94,7 @@ jobs:
       msiId: $(MSI_USER_ASSIGNED_ID_AKS_ENGINE_E2E)
       windowsVHDBuild: $(WINDOWS_VHD_BUILD)
       linuxVHDBuild: $(LINUX_VHD_BUILD)
+      testPVC: true
 
   - template: e2e-job-template.yaml
     parameters:
@@ -112,3 +114,4 @@ jobs:
       msiId: $(MSI_USER_ASSIGNED_ID_AKS_ENGINE_E2E)
       windowsVHDBuild: $(WINDOWS_VHD_BUILD)
       linuxVHDBuild: $(LINUX_VHD_BUILD)
+      testPVC: true
diff --git a/docs/topics/azure-stack.md b/docs/topics/azure-stack.md
@@ -185,6 +185,7 @@ The [in-tree volume provisioner](https://kubernetes.io/blog/2019/12/09/kubernete
 AKS Engine will **not** enable any CSI driver by default on Azure Stack Hub. For workloads that require a CSI driver, it is possible to either:
 
 * For AKS Engine versions 0.75.3 and above: explicitly enable the `azuredisk-csi-driver` [addon](../topics/clusterdefinitions.md#addons) (Linux and/or Windows cluster)
+  * If you are using Windows HostProcess containers in your workloads, do not use the addon. Instead use `Helm` to [install the `azuredisk-csi-driver` chart](#1-install-azure-disk-csi-driver-manually) and set the following value: `--set windows.useHostProcessContainers=true`.
 * For AKS Engine versions 0.70.0 and above: explicitly enable the `azuredisk-csi-driver` [addon](../topics/clusterdefinitions.md#addons) (Linux cluster only)
 * For AKS Engine versions 0.70.0 and above: use `Helm` to [install the `azuredisk-csi-driver` chart](#1-install-azure-disk-csi-driver-manually) (Linux and/or Windows clusters).
 
@@ -203,7 +204,8 @@ If the data persisted in the underlying Azure disks should be preserved, then th
 The following script uses `Helm` to install the Azure Disk CSI Driver:
 
 ```bash
-DRIVER_VERSION=v1.26.5
+DRIVER_VERSION=v1.26.5 # if using k8s v1.26
+DRIVER_VERSION=v1.28.3 # if using k8s v1.27
 helm repo add azuredisk-csi-driver https://raw.githubusercontent.com/kubernetes-sigs/azuredisk-csi-driver/master/charts
 helm install azuredisk-csi-driver azuredisk-csi-driver/azuredisk-csi-driver \
   --namespace kube-system \
@@ -212,6 +214,7 @@ helm install azuredisk-csi-driver azuredisk-csi-driver/azuredisk-csi-driver \
   --set node.supportZone=false \
   --set windows.getNodeInfoFromLabels=true \
   --set linux.getNodeInfoFromLabels=true \
+  # --set windows.useHostProcessContainers=true \ # if using Windows HostProcess containers
   --version ${DRIVER_VERSION}
 ```
 
@@ -278,9 +281,18 @@ In this section, please follow the example commands to deploy a StatefulSet appl
 
 ```bash
 # Install CSI Driver
-DRIVER_VERSION=v1.26.5
+DRIVER_VERSION=v1.26.5 # if using k8s v1.26
+DRIVER_VERSION=v1.28.3 # if using k8s v1.27
 helm repo add azuredisk-csi-driver https://raw.githubusercontent.com/kubernetes-sigs/azuredisk-csi-driver/master/charts
-helm install azuredisk-csi-driver azuredisk-csi-driver/azuredisk-csi-driver --namespace kube-system --set cloud=AzureStackCloud --set controller.runOnMaster=true --set node.supportZone=false --set windows.getNodeInfoFromLabels=true --set linux.getNodeInfoFromLabels=true --version ${DRIVER_VERSION}
+helm install azuredisk-csi-driver azuredisk-csi-driver/azuredisk-csi-driver \
+  --namespace kube-system \
+  --set cloud=AzureStackCloud \
+  --set controller.runOnMaster=true \
+  --set node.supportZone=false \
+  --set windows.getNodeInfoFromLabels=true \
+  --set linux.getNodeInfoFromLabels=true \
+  # --set windows.useHostProcessContainers=true \ # if using Windows HostProcess containers
+  --version ${DRIVER_VERSION}
 
 # Deploy Storage Class
 kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/azuredisk-csi-driver/master/deploy/example/storageclass-azuredisk-csi-azurestack.yaml

diff --git a/parts/k8s/addons/azuredisk-csi-driver-deployment.yaml b/parts/k8s/addons/azuredisk-csi-driver-deployment.yaml
diff --git a/pkg/api/k8s_versions.go b/pkg/api/k8s_versions.go
diff --git a/pkg/api/k8s_versions_test.go b/pkg/api/k8s_versions_test.go
@@ -68,16 +68,16 @@ func TestGetK8sVersionComponents(t *testing.T) {
 		common.AzurePolicyAddonName:                       azurePolicyImageReference,
 		common.GatekeeperContainerName:                    gatekeeperImageReference,
 		common.NodeProblemDetectorAddonName:               nodeProblemDetectorImageReference,
-		common.CSIProvisionerContainerName:                csiProvisionerImageReference,
-		common.CSIAttacherContainerName:                   csiAttacherImageReference,
-		common.CSILivenessProbeContainerName:              csiLivenessProbeImageReference,
-		common.CSILivenessProbeWindowsContainerName:       csiLivenessProbeWindowsImageReference,
-		common.CSINodeDriverRegistrarContainerName:        csiNodeDriverRegistrarImageReference,
-		common.CSINodeDriverRegistrarWindowsContainerName: csiNodeDriverRegistrarWindowsImageReference,
-		common.CSISnapshotterContainerName:                csiSnapshotterImageReference,
-		common.CSISnapshotControllerContainerName:         csiSnapshotControllerImageReference,
-		common.CSIResizerContainerName:                    csiResizerImageReference,
-		common.CSIAzureDiskContainerName:                  csiAzureDiskImageReference,
+		common.CSIProvisionerContainerName:                "oss/kubernetes-csi/csi-provisioner:v3.0.0",
+		common.CSIAttacherContainerName:                   "oss/kubernetes-csi/csi-attacher:v3.3.0",
+		common.CSILivenessProbeContainerName:              "oss/kubernetes-csi/livenessprobe:v2.5.0",
+		common.CSILivenessProbeWindowsContainerName:       "oss/kubernetes-csi/livenessprobe:v2.5.0",
+		common.CSINodeDriverRegistrarContainerName:        "oss/kubernetes-csi/csi-node-driver-registrar:v2.4.0",
+		common.CSINodeDriverRegistrarWindowsContainerName: "oss/kubernetes-csi/csi-node-driver-registrar:v2.4.0",
+		common.CSISnapshotterContainerName:                "oss/kubernetes-csi/csi-snapshotter:v4.2.1",
+		common.CSISnapshotControllerContainerName:         "oss/kubernetes-csi/snapshot-controller:v4.2.1",
+		common.CSIResizerContainerName:                    "oss/kubernetes-csi/csi-resizer:v1.3.0",
+		common.CSIAzureDiskContainerName:                  "oss/kubernetes-csi/azuredisk-csi:v1.10.0",
 		common.CSIAzureFileContainerName:                  csiAzureFileImageReference,
 		common.KubeFlannelContainerName:                   kubeFlannelImageReference,
 		"flannel" + common.FlannelInstallCNIContainerName: flannelInstallCNIImageReference,
@@ -157,14 +157,14 @@ func TestGetK8sVersionComponents(t *testing.T) {
 		common.AzurePolicyAddonName:                       azurePolicyImageReference,
 		common.GatekeeperContainerName:                    gatekeeperImageReference,
 		common.NodeProblemDetectorAddonName:               nodeProblemDetectorImageReference,
-		common.CSIProvisionerContainerName:                csiProvisionerImageReference,
-		common.CSIAttacherContainerName:                   csiAttacherImageReference,
-		common.CSILivenessProbeContainerName:              csiLivenessProbeImageReference,
-		common.CSINodeDriverRegistrarContainerName:        csiNodeDriverRegistrarImageReference,
-		common.CSISnapshotterContainerName:                csiSnapshotterImageReference,
-		common.CSISnapshotControllerContainerName:         csiSnapshotControllerImageReference,
-		common.CSIResizerContainerName:                    csiResizerImageReference,
-		common.CSIAzureDiskContainerName:                  csiAzureDiskImageReference,
+		common.CSIProvisionerContainerName:                "oss/kubernetes-csi/csi-provisioner:v3.0.0",
+		common.CSIAttacherContainerName:                   "oss/kubernetes-csi/csi-attacher:v3.3.0",
+		common.CSILivenessProbeContainerName:              "oss/kubernetes-csi/livenessprobe:v2.5.0",
+		common.CSINodeDriverRegistrarContainerName:        "oss/kubernetes-csi/csi-node-driver-registrar:v2.4.0",
+		common.CSISnapshotterContainerName:                "oss/kubernetes-csi/csi-snapshotter:v4.2.1",
+		common.CSISnapshotControllerContainerName:         "oss/kubernetes-csi/snapshot-controller:v4.2.1",
+		common.CSIResizerContainerName:                    "oss/kubernetes-csi/csi-resizer:v1.3.0",
+		common.CSIAzureDiskContainerName:                  "oss/kubernetes-csi/azuredisk-csi:v1.10.0",
 		common.CSIAzureFileContainerName:                  csiAzureFileImageReference,
 		common.KubeFlannelContainerName:                   kubeFlannelImageReference,
 		"flannel" + common.FlannelInstallCNIContainerName: flannelInstallCNIImageReference,
@@ -224,12 +224,12 @@ func TestGetK8sVersionComponents(t *testing.T) {
 		common.SMBFlexVolumeAddonName:                     smbFlexVolumeImageReference,
 		common.IPMASQAgentAddonName:                       getDefaultImage(common.IPMASQAgentAddonName, kubernetesImageBaseType),
 		common.AzureNetworkPolicyAddonName:                azureNPMContainerImageReference,
-		common.CSIProvisionerContainerName:                csiProvisionerImageReference,
-		common.CSIAttacherContainerName:                   csiAttacherImageReference,
-		common.CSILivenessProbeContainerName:              csiLivenessProbeImageReference,
-		common.CSINodeDriverRegistrarContainerName:        csiNodeDriverRegistrarImageReference,
-		common.CSIResizerContainerName:                    csiResizerImageReference,
-		common.CSIAzureDiskContainerName:                  csiAzureDiskImageReference,
+		common.CSIProvisionerContainerName:                "oss/kubernetes-csi/csi-provisioner:v3.0.0",
+		common.CSIAttacherContainerName:                   "oss/kubernetes-csi/csi-attacher:v3.3.0",
+		common.CSILivenessProbeContainerName:              "oss/kubernetes-csi/livenessprobe:v2.5.0",
+		common.CSINodeDriverRegistrarContainerName:        "oss/kubernetes-csi/csi-node-driver-registrar:v2.4.0",
+		common.CSIResizerContainerName:                    "oss/kubernetes-csi/csi-resizer:v1.3.0",
+		common.CSIAzureDiskContainerName:                  "oss/kubernetes-csi/azuredisk-csi:v1.10.0",
 		common.CSIAzureFileContainerName:                  csiAzureFileImageReference,
 		common.CalicoTyphaComponentName:                   calicoTyphaImageReference,
 		common.CalicoCNIComponentName:                     calicoCNIImageReference,
@@ -307,11 +307,11 @@ func TestGetK8sVersionComponents(t *testing.T) {
 		common.SMBFlexVolumeAddonName:                     smbFlexVolumeImageReference,
 		common.IPMASQAgentAddonName:                       getDefaultImage(common.IPMASQAgentAddonName, kubernetesImageBaseType),
 		common.AzureNetworkPolicyAddonName:                azureNPMContainerImageReference,
-		common.CSIProvisionerContainerName:                csiProvisionerImageReference,
-		common.CSIAttacherContainerName:                   csiAttacherImageReference,
-		common.CSILivenessProbeContainerName:              csiLivenessProbeImageReference,
-		common.CSINodeDriverRegistrarContainerName:        csiNodeDriverRegistrarImageReference,
-		common.CSIAzureDiskContainerName:                  csiAzureDiskImageReference,
+		common.CSIProvisionerContainerName:                "oss/kubernetes-csi/csi-provisioner:v3.0.0",
+		common.CSIAttacherContainerName:                   "oss/kubernetes-csi/csi-attacher:v3.3.0",
+		common.CSILivenessProbeContainerName:              "oss/kubernetes-csi/livenessprobe:v2.5.0",
+		common.CSINodeDriverRegistrarContainerName:        "oss/kubernetes-csi/csi-node-driver-registrar:v2.4.0",
+		common.CSIAzureDiskContainerName:                  "oss/kubernetes-csi/azuredisk-csi:v1.10.0",
 		common.CSIAzureFileContainerName:                  csiAzureFileImageReference,
 		common.CalicoTyphaComponentName:                   calicoTyphaImageReference,
 		common.CalicoCNIComponentName:                     calicoCNIImageReference,
@@ -385,11 +385,11 @@ func TestGetK8sVersionComponents(t *testing.T) {
 		common.SMBFlexVolumeAddonName:                     smbFlexVolumeImageReference,
 		common.IPMASQAgentAddonName:                       getDefaultImage(common.IPMASQAgentAddonName, kubernetesImageBaseType),
 		common.AzureNetworkPolicyAddonName:                azureNPMContainerImageReference,
-		common.CSIProvisionerContainerName:                csiProvisionerImageReference,
-		common.CSIAttacherContainerName:                   csiAttacherImageReference,
-		common.CSILivenessProbeContainerName:              csiLivenessProbeImageReference,
-		common.CSINodeDriverRegistrarContainerName:        csiNodeDriverRegistrarImageReference,
-		common.CSIAzureDiskContainerName:                  csiAzureDiskImageReference,
+		common.CSIProvisionerContainerName:                "oss/kubernetes-csi/csi-provisioner:v3.0.0",
+		common.CSIAttacherContainerName:                   "oss/kubernetes-csi/csi-attacher:v3.3.0",
+		common.CSILivenessProbeContainerName:              "oss/kubernetes-csi/livenessprobe:v2.5.0",
+		common.CSINodeDriverRegistrarContainerName:        "oss/kubernetes-csi/csi-node-driver-registrar:v2.4.0",
+		common.CSIAzureDiskContainerName:                  "oss/kubernetes-csi/azuredisk-csi:v1.10.0",
 		common.CSIAzureFileContainerName:                  csiAzureFileImageReference,
 		common.CalicoTyphaComponentName:                   calicoTyphaImageReference,
 		common.CalicoCNIComponentName:                     calicoCNIImageReference,
@@ -462,11 +462,11 @@ func TestGetK8sVersionComponents(t *testing.T) {
 		common.SMBFlexVolumeAddonName:                     smbFlexVolumeImageReference,
 		common.IPMASQAgentAddonName:                       getDefaultImage(common.IPMASQAgentAddonName, kubernetesImageBaseType),
 		common.AzureNetworkPolicyAddonName:                azureNPMContainerImageReference,
-		common.CSIProvisionerContainerName:                csiProvisionerImageReference,
-		common.CSIAttacherContainerName:                   csiAttacherImageReference,
-		common.CSILivenessProbeContainerName:              csiLivenessProbeImageReference,
-		common.CSINodeDriverRegistrarContainerName:        csiNodeDriverRegistrarImageReference,
-		common.CSIAzureDiskContainerName:                  csiAzureDiskImageReference,
+		common.CSIProvisionerContainerName:                "oss/kubernetes-csi/csi-provisioner:v3.0.0",
+		common.CSIAttacherContainerName:                   "oss/kubernetes-csi/csi-attacher:v3.3.0",
+		common.CSILivenessProbeContainerName:              "oss/kubernetes-csi/livenessprobe:v2.5.0",
+		common.CSINodeDriverRegistrarContainerName:        "oss/kubernetes-csi/csi-node-driver-registrar:v2.4.0",
+		common.CSIAzureDiskContainerName:                  "oss/kubernetes-csi/azuredisk-csi:v1.10.0",
 		common.CSIAzureFileContainerName:                  csiAzureFileImageReference,
 		common.CalicoTyphaComponentName:                   calicoTyphaImageReference,
 		common.CalicoCNIComponentName:                     calicoCNIImageReference,