feat: External cloud provider support for Azure Stack Cloud #4635
Changes from 4 commits
886878f
324d891
6a4d571
5afaa3a
c4d3fa1
96d013a
4a2796a
3bdd22f
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -88,18 +88,50 @@ spec: | |
command: | ||
- cloud-node-manager | ||
- --node-name=$(NODE_NAME) | ||
{{- if IsAzureStackCloud}} | ||
- --use-instance-metadata=false | ||
- --cloud-config=/etc/kubernetes/azure.json | ||
- --kubeconfig=/var/lib/kubelet/kubeconfig | ||
{{end}} | ||
env: | ||
- name: NODE_NAME | ||
valueFrom: | ||
fieldRef: | ||
fieldPath: spec.nodeName | ||
{{- if IsAzureStackCloud}} | ||
- name: AZURE_ENVIRONMENT_FILEPATH | ||
value: /etc/kubernetes/azurestackcloud.json | ||
- name: AZURE_GO_SDK_LOG_LEVEL | ||
value: DEBUG | ||
{{end}} | ||
resources: | ||
requests: | ||
cpu: 50m | ||
memory: 50Mi | ||
limits: | ||
cpu: 2000m | ||
memory: 512Mi | ||
{{- if IsAzureStackCloud}} | ||
volumeMounts: | ||
- name: etc-kubernetes | ||
mountPath: /etc/kubernetes | ||
- name: etc-ssl | ||
mountPath: /etc/ssl | ||
readOnly: true | ||
- name: var-lib-kubelet | ||
mountPath: /var/lib/kubelet | ||
readOnly: true | ||
volumes: | ||
- name: etc-kubernetes | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. when possible, I would just mount the file instead of the entire directory: volumes:
- name: etc-kubernetes
hostPath:
path: /etc/kubernetes/azurestackcloud.json
type: FileOrCreate volumeMounts:
- name: custom-environment
mountPath: /etc/kubernetes/azurestackcloud.json
readOnly: true There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The /etc/kubernetes directory also contains azure.json and cert, so I leave this one to folder level access. I updated the /var/lib/kubelet/kubeconfig to file level access. |
||
hostPath: | ||
path: /etc/kubernetes | ||
- name: etc-ssl | ||
hostPath: | ||
path: /etc/ssl | ||
- name: var-lib-kubelet | ||
hostPath: | ||
path: /var/lib/kubelet | ||
{{end}} | ||
{{- if and HasWindows (IsKubernetesVersionGe "1.18.0")}} | ||
--- | ||
apiVersion: apps/v1 | ||
|
@@ -148,16 +180,42 @@ spec: | |
command: | ||
- /cloud-node-manager.exe | ||
- --node-name=$(NODE_NAME) | ||
{{- if IsAzureStackCloud}} | ||
- --use-instance-metadata=false | ||
- --cloud-config=C:\k\azure.json | ||
- --kubeconfig=C:\k\config | ||
lifecycle: | ||
postStart: | ||
exec: | ||
command: | ||
- C:\k\addazsroot.bat | ||
{{end}} | ||
env: | ||
- name: NODE_NAME | ||
valueFrom: | ||
fieldRef: | ||
fieldPath: spec.nodeName | ||
{{- if IsAzureStackCloud}} | ||
- name: AZURE_ENVIRONMENT_FILEPATH | ||
value: C:\k\azurestackcloud.json | ||
- name: AZURE_GO_SDK_LOG_LEVEL | ||
value: DEBUG | ||
{{end}} | ||
resources: | ||
requests: | ||
cpu: 50m | ||
memory: 50Mi | ||
limits: | ||
cpu: 2000m | ||
memory: 512Mi | ||
{{end}} | ||
{{- if IsAzureStackCloud}} | ||
volumeMounts: | ||
- name: azure-config | ||
mountPath: C:\k | ||
volumes: | ||
- name: azure-config | ||
hostPath: | ||
path: C:\k | ||
type: Directory | ||
{{end}} | ||
{{end}} |
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -13,6 +13,13 @@ spec: | |
- name: cloud-controller-manager | ||
image: {{ContainerImage "cloud-controller-manager"}} | ||
imagePullPolicy: IfNotPresent | ||
{{- if IsAzureStackCloud}} | ||
env: | ||
- name: AZURE_ENVIRONMENT_FILEPATH | ||
value: /etc/kubernetes/azurestackcloud.json | ||
- name: AZURE_GO_SDK_LOG_LEVEL | ||
value: DEBUG | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Do we want to keep this DEBUG verbosity for customers? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I will update, thanks for catching this |
||
{{end}} | ||
command: [{{ContainerConfig "command"}}] | ||
args: [{{GetCloudControllerManagerArgs}}] | ||
resources: | ||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -45,8 +45,8 @@ const ( | |
csiSnapshotControllerImageReference string = "oss/kubernetes-csi/snapshot-controller:v2.0.0" | ||
csiAzureDiskImageReference string = "k8s/csi/azuredisk-csi:v0.7.0" | ||
csiAzureFileImageReference string = "k8s/csi/azurefile-csi:v0.6.0" | ||
azureCloudControllerManagerImageReference string = "oss/kubernetes/azure-cloud-controller-manager:v0.5.1" | ||
azureCloudNodeManagerImageReference string = "oss/kubernetes/azure-cloud-node-manager:v0.5.1" | ||
azureCloudControllerManagerImageReference string = "oss/kubernetes/azure-cloud-controller-manager:v1.1.0" | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. is it OK to update this for all clouds? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The change works for Azure and Azure Stack. For other cloud, I think they can continue to use with "useCloudControllerManager" flag set to false, or adopt similar change we did if they want to use external cloud provider as well. Does this make sense? |
||
azureCloudNodeManagerImageReference string = "oss/kubernetes/azure-cloud-node-manager:v1.1.0" | ||
dashboardImageReference string = "mcr.microsoft.com/oss/kubernetes/dashboard:v2.0.4" // deprecated | ||
dashboardMetricsScraperImageReference string = "mcr.microsoft.com/oss/kubernetes/metrics-scraper:v1.0.4" | ||
kubeFlannelImageReference string = "quay.io/coreos/flannel:v0.8.0-amd64" | ||
|
@@ -571,7 +571,7 @@ func getK8sVersionComponents(version, kubernetesImageBaseType string, overrides | |
common.CloudControllerManagerComponentName: azureCloudControllerManagerImageReference, | ||
common.CloudNodeManagerAddonName: azureCloudNodeManagerImageReference, | ||
common.WindowsArtifactComponentName: "v" + version + "/windowszip/v" + version + "-1int.zip", | ||
common.WindowsArtifactAzureStackComponentName: "v" + version + common.AzureStackSuffix + "/windowszip/v" + version + common.AzureStackSuffix + "-1int.zip", | ||
common.WindowsArtifactAzureStackComponentName: "v" + version + "/windowszip/v" + version + "-1int.zip", | ||
common.DashboardAddonName: dashboardImageReference, | ||
common.DashboardMetricsScraperContainerName: dashboardMetricsScraperImageReference, | ||
common.ExecHealthZComponentName: getDefaultImage(common.ExecHealthZComponentName, kubernetesImageBaseType), | ||
|
@@ -655,7 +655,7 @@ func getK8sVersionComponents(version, kubernetesImageBaseType string, overrides | |
common.CloudControllerManagerComponentName: azureCloudControllerManagerImageReference, | ||
common.CloudNodeManagerAddonName: azureCloudNodeManagerImageReference, | ||
common.WindowsArtifactComponentName: "v" + version + "/windowszip/v" + version + "-1int.zip", | ||
common.WindowsArtifactAzureStackComponentName: "v" + version + common.AzureStackSuffix + "/windowszip/v" + version + common.AzureStackSuffix + "-1int.zip", | ||
common.WindowsArtifactAzureStackComponentName: "v" + version + "/windowszip/v" + version + "-1int.zip", | ||
common.DashboardAddonName: dashboardImageReference, | ||
common.DashboardMetricsScraperContainerName: dashboardMetricsScraperImageReference, | ||
common.ExecHealthZComponentName: getDefaultImage(common.ExecHealthZComponentName, kubernetesImageBaseType), | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
do you know why only ASH needs
cloud-config
andkubeconfig
?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For
kubeconfig
, if not provided the ecp (external cloud provider) will use default value from environment variable ofKubernetes Service
(https://github.com/kubernetes-sigs/cloud-provider-azure/blob/master/vendor/k8s.io/client-go/rest/config.go#L488), which doesn't work for Azure Stack as well as Azure Windows node.For
cloud-config
, we need that to correct get cloud and tenant information.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We should figure out which
KUBERNETES_SERVICE_HOST
andKUBERNETES_SERVICE_PORT
values work on ASH and Windows. The same for cloud and tenant info. The least access to the host file system, the better.