feat: External cloud provider support for Azure Stack Cloud #4635
Conversation
| mountPath: /var/lib/kubelet | ||
| readOnly: true | ||
| volumes: | ||
| - name: etc-kubernetes |
There was a problem hiding this comment.
when possible, I would just mount the file instead of the entire directory:
volumes:
- name: etc-kubernetes
hostPath:
path: /etc/kubernetes/azurestackcloud.json
type: FileOrCreatevolumeMounts:
- name: custom-environment
mountPath: /etc/kubernetes/azurestackcloud.json
readOnly: trueThere was a problem hiding this comment.
The /etc/kubernetes directory also contains azure.json and cert, so I leave this one to folder level access. I updated the /var/lib/kubelet/kubeconfig to file level access.
pkg/api/k8s_versions.go
Outdated
| @@ -45,8 +45,8 @@ const ( | |||
| csiSnapshotControllerImageReference string = "oss/kubernetes-csi/snapshot-controller:v2.0.0" | |||
| csiAzureDiskImageReference string = "k8s/csi/azuredisk-csi:v0.7.0" | |||
| csiAzureFileImageReference string = "k8s/csi/azurefile-csi:v0.6.0" | |||
| azureCloudControllerManagerImageReference string = "oss/kubernetes/azure-cloud-controller-manager:v0.5.1" | |||
| azureCloudNodeManagerImageReference string = "oss/kubernetes/azure-cloud-node-manager:v0.5.1" | |||
| azureCloudControllerManagerImageReference string = "oss/kubernetes/azure-cloud-controller-manager:v1.1.0" | |||
There was a problem hiding this comment.
is it OK to update this for all clouds?
There was a problem hiding this comment.
The change works for Azure and Azure Stack. For other cloud, I think they can continue to use with "useCloudControllerManager" flag set to false, or adopt similar change we did if they want to use external cloud provider as well. Does this make sense?
| @@ -88,18 +88,52 @@ spec: | |||
| command: | |||
| - cloud-node-manager | |||
| - --node-name=$(NODE_NAME) | |||
| {{- if IsAzureStackCloud}} | |||
| - --use-instance-metadata=false | |||
| - --cloud-config=/etc/kubernetes/azure.json | |||
There was a problem hiding this comment.
do you know why only ASH needs cloud-config and kubeconfig?
There was a problem hiding this comment.
For kubeconfig, if not provided the ecp (external cloud provider) will use default value from environment variable of Kubernetes Service (https://github.com/kubernetes-sigs/cloud-provider-azure/blob/master/vendor/k8s.io/client-go/rest/config.go#L488), which doesn't work for Azure Stack as well as Azure Windows node.
For cloud-config, we need that to correct get cloud and tenant information.
There was a problem hiding this comment.
We should figure out which KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT values work on ASH and Windows. The same for cloud and tenant info. The least access to the host file system, the better.
| @@ -148,16 +182,40 @@ spec: | |||
| command: | |||
| - /cloud-node-manager.exe | |||
| - --node-name=$(NODE_NAME) | |||
| - --kubeconfig=C:\k\config | |||
There was a problem hiding this comment.
Did we want this new --kubeconfig configuration to be outside of the Azure Stack block?
There was a problem hiding this comment.
Based on my test on Azure, the windows node will also need this config to launch the cloud node manager pod. The default IP in external-cloud-provider is not working.
There was a problem hiding this comment.
Sample error:
E0902 16:06:16.057842 1344 nodemanager.go:183] Error monitoring node status: Get "https://10.0.0.1:443/api/v1/nodes?fieldSelector=metadata.name%3D2356k8s010&resourceVersion=0": dial tcp 10.0.0.1:443: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
| - name: AZURE_ENVIRONMENT_FILEPATH | ||
| value: /etc/kubernetes/azurestackcloud.json | ||
| - name: AZURE_GO_SDK_LOG_LEVEL | ||
| value: DEBUG |
There was a problem hiding this comment.
Do we want to keep this DEBUG verbosity for customers?
There was a problem hiding this comment.
I will update, thanks for catching this
|
@haofan-ms |
Reason for Change:
Issue Fixed:
Credit Where Due:
Does this change contain code from or inspired by another project?
If "Yes," did you notify that project's maintainers and provide attribution?
Requirements:
Notes: