Allow setting Azure cloud provider from Kubernetes secrets instread of local configure file

[feiskyer]

What type of PR is this?

Uncomment only one /kind <> line, hit enter to put that in a new line, and remove leading whitespaces from that line:

/kind api-change
/kind bug
/kind cleanup
/kind design
/kind documentation
/kind failing-test
/kind feature
/kind flake

What this PR does / why we need it:

Refer Azure/container-upstream#27. This PR allows configure Azure cloud provider from Kubernetes secrets.

To support this, a new option cloudConfigType is added. Supported values are:

• file: The cloud provider configuration is read from cloud-config file.
• secret: the cloud provider configuration must be overridden by a secret.
• merge: the cloud provider configuration can be optionally overridden by a secret when it is set explicitly in the secret, this is default value.

Note that the secret is a serialized version of azure.json file with key cloud-config. And the secret name is azure-cloud-provider.

Since Azure cloud provider would read Kubernetes secrets, the following RBAC should also be configured:

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  labels:
    kubernetes.io/cluster-service: "true"
  name: system:azure-cloud-provider-secret-getter
rules:
- apiGroups: [""]
  resources: ["secrets"]
  resourceNames: ["azure-cloud-provider"]
  verbs:
  - get
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  labels:
    kubernetes.io/cluster-service: "true"
  name: system:azure-cloud-provider-secret-getter
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:azure-cloud-provider-secret-getter
subjects:
- kind: ServiceAccount
  name: azure-cloud-provider
  namespace: kube-system

Which issue(s) this PR fixes:

Fixes Azure/container-upstream#27

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

Azure cloud provider could now be configured by Kubernetes secrets and a new option `cloudConfigType` is introduced, whose candicate values are `file`, `secret` and `merge` (default is `merge`).

action required:

Since Azure cloud provider would read Kubernetes secrets, the following RBAC should be configured:

  apiVersion: rbac.authorization.k8s.io/v1beta1
  kind: ClusterRole
  metadata:
    labels:
      kubernetes.io/cluster-service: "true"
    name: system:azure-cloud-provider-secret-getter
  rules:
  - apiGroups: [""]
    resources: ["secrets"]
    resourceNames: ["azure-cloud-provider"]
    verbs:
    - get
  ---
  apiVersion: rbac.authorization.k8s.io/v1beta1
  kind: ClusterRoleBinding
  metadata:
    labels:
      kubernetes.io/cluster-service: "true"
    name: system:azure-cloud-provider-secret-getter
  roleRef:
    apiGroup: rbac.authorization.k8s.io
    kind: ClusterRole
    name: system:azure-cloud-provider-secret-getter
  subjects:
  - kind: ServiceAccount
    name: azure-cloud-provider
    namespace: kube-system

/sig azure
/kind feature
/priority important-soon

/assign @khenidak @andyzhangx

[andyzhangx]

one question about the code logic, we have two path

init --> NewCloud --> initializeCloudFromConfig(fromSecret: false)
Cloud.Initialize --> initializeCloudFromSecret --> initializeCloudFromConfig(fromSecret: true)

so init would be invoked first and Cloud.Initialize could be invoked second, and finally it will goto initializeCloudFromConfig, right?

[feiskyer]

/retest

[andyzhangx]

one question about the code logic, we have two path

init --> NewCloud --> initializeCloudFromConfig(fromSecret: false)
Cloud.Initialize --> initializeCloudFromSecret --> initializeCloudFromConfig(fromSecret: true)

so init would be invoked first and Cloud.Initialize could be invoked second, and finally it will goto initializeCloudFromConfig, right?

[feiskyer]

so init would be invoked first and Cloud.Initialize could be invoked second, and finally it will goto initializeCloudFromConfig, right?

Both would go to initializeCloudFromConfig, the difference is Kubelet won't invoke Initialize, while controller-manager would do. And for controller-manager, we want NewCloud won't error out, and initialize from secret again. But when secret initializing failed, controller-manager would fail with an error.

[andyzhangx]

so init would be invoked first and Cloud.Initialize could be invoked second, and finally it will goto initializeCloudFromConfig, right?

Both would go to initializeCloudFromConfig, the difference is Kubelet won't invoke Initialize, while controller-manager would do. And for controller-manager, we want NewCloud won't error out, and initialize from secret again. But when secret initializing failed, controller-manager would fail with an error.

There are other drivers like CSI driver would also leverage this azure cloud provider lib, it uses NewCloud: https://github.com/kubernetes-sigs/azuredisk-csi-driver/blob/master/pkg/azuredisk/azure.go#L45
can we add a switch in NewCloud, let it read from secret also? by default, fromSecret could be false in NewCloud

[andrewsykim]

/hold

So sorry for the last minute hold, but I think the kubelet checks are worth raising a flag for. Can we do this change without the kubelet checks?

[feiskyer]

@andrewsykim thanks for reviewing. Agreed, the Kubelet check should be changed in alternative ways. will address these comments before v1.15 release

[andrewsykim]

/hold cancel

Thanks @feiskyer

[andrewsykim]

heads up @claurence, we might have some follow-up PRs come in for this one

[andrewsykim]

/hold

@feiskyer addressing comments given the code freeze extension

[feiskyer]

@andrewsykim Addressed comments. PTAL

/hold cancel

[justaugustus]

/lgtm
/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: andrewsykim, feiskyer, justaugustus

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• staging/src/k8s.io/legacy-cloud-providers/azure/OWNERS [andrewsykim,feiskyer]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[andrewsykim]

lgtm, thanks for iterating @feiskyer!

[jackfrancis]

@feiskyer will these capabilities ever be backported into pre-1.15 k8s or is this only applicable for 1.15 and greater?

[andyzhangx]

@feiskyer will these capabilities ever be backported into pre-1.15 k8s or is this only applicable for 1.15 and greater?

It won't be backported, only applicable for 1.15 and greater

[feiskyer]

@jackfrancis Yep, the feature is only available for v1.15 and greater