New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Azure w/ cloud provider fails to provision at kubelet health check #13233

Closed
bmdepesa opened this Issue May 1, 2018 · 1 comment

Comments

Projects
None yet
2 participants
@bmdepesa

bmdepesa commented May 1, 2018

Rancher versions:
rancher/server or rancher/rancher: v2.0.0

k8s: 1.10.1
docker: 17.03

Steps to Reproduce:
Create Azure cluster w/ cloud provider in UI
After kube-api healthcheck completes, kubelet fails healthcheck.

It looks like cloud config is being deleted between kube-api and kubelet start? kube-api has the correct cloud config. Subscription id and location values are correct in kubelet.

 echo kubelet --enforce-node-allocatable= --hostname-override=brandon-azure1 --fail-swap-on=false --root-dir=/var/lib/kubelet --feature-gates=MountPropagation=false --v=2 --address=0.0.0.0 --cadvisor-port=0 --read-only-port=0 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --cgroups-per-qos=True --network-plugin=cni --allow-privileged=true --volume-plugin-dir=/var/lib/kubelet/volumeplugins --cluster-dns=10.43.0.10 --cloud-provider=azure --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin --resolv-conf=/etc/resolv.conf --anonymous-auth=false --cluster-domain=cluster.local --pod-infra-container-image=rancher/pause-amd64:3.1 --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --cloud-config=/etc/kubernetes/cloud-config.json
+ grep -q cloud-provider=azure
+ '[' kubelet = kubelet ']'
+ source /opt/rke/cloud-provider.sh
++ AZURE_META_URL=http://<>/metadata/instance/compute
++ AZURE_CLOUD_CONFIG_PATH=/etc/kubernetes/cloud-config.json
+ set_azure_config
++ curl -s -H Metadata:true 'http://<>/metadata/instance/compute/resourceGroupName?api-version=2017-08-01&format=text'
+ local az_resources_group=docker-machine
++ curl -s -H Metadata:true 'http://<>/metadata/instance/compute/subscriptionId?api-version=2017-08-01&format=text'
+ local az_subscription_id=<subscription id> // this was correct
++ curl -s -H Metadata:true 'http://<>/metadata/instance/compute/location?api-version=2017-08-01&format=text'
+ local az_location=westus
++ curl -s -H Metadata:true 'http://<>/metadata/instance/compute/name?api-version=2017-08-01&format=text'
+ local az_vm_name=brandon-azure1
++ jq -r .cloud
++ cat /etc/kubernetes/cloud-config.json
+ local azure_cloud=
++ cat /etc/kubernetes/cloud-config.json
++ jq -r .aadClientId
+ local azure_client_id=
++ cat /etc/kubernetes/cloud-config.json
++ jq -r .aadClientSecret
+ local azure_client_secret=
++ cat /etc/kubernetes/cloud-config.json
++ jq -r .tenantId
+ local azure_tenant_id=
+ '[' '' = null ']'
+ '[' '' = '' ']'
+ azure_cloud=AzureCloud
+ az cloud set --name AzureCloud
+ az login --service-principal -u -p --tenant
ERROR: az login: error: argument --username/-u: expected one argument
usage: az login [-h] [--verbose] [--debug] [--output {json,jsonc,table,tsv}]
                [--query JMESPATH] [--username USERNAME] [--password PASSWORD]
                [--service-principal] [--tenant TENANT]
                [--allow-no-subscriptions] [-i]
                [--identity-port IDENTITY_PORT]
++ cut -d / -f 9
++ jq -r '.[0].id'
++ az vm nic list -g docker-machine --vm-name brandon-azure1
ERROR: Please run 'az login' to setup account.
+ local az_vm_nic=
++ az vm nic show -g docker-machine --vm-name brandon-azure1 --nic
++ cut -d/ -f 11
++ jq -r '.ipConfigurations[0].subnet.id'
ERROR: az vm nic show: error: argument --nic: expected one argument
usage: az vm nic show [-h] [--verbose] [--debug]
                      [--output {json,jsonc,table,tsv}] [--query JMESPATH]
                      --resource-group RESOURCE_GROUP_NAME --vm-name NAME
                      --nic NIC
+ local az_subnet_name=
++ az vm nic show -g docker-machine --vm-name brandon-azure1 --nic
++ cut -d/ -f 9
++ jq -r '.ipConfigurations[0].subnet.id'
ERROR: az vm nic show: error: argument --nic: expected one argument
usage: az vm nic show [-h] [--verbose] [--debug]
                      [--output {json,jsonc,table,tsv}] [--query JMESPATH]
                      --resource-group RESOURCE_GROUP_NAME --vm-name NAME
                      --nic NIC
+ local az_vnet_name=
++ az vm nic show -g docker-machine --vm-name brandon-azure1 --nic
++ cut -d/ -f 5
++ jq -r '.ipConfigurations[0].subnet.id'
ERROR: az vm nic show: error: argument --nic: expected one argument
usage: az vm nic show [-h] [--verbose] [--debug]
                      [--output {json,jsonc,table,tsv}] [--query JMESPATH]
                      --resource-group RESOURCE_GROUP_NAME --vm-name NAME
                      --nic NIC
+ local az_vnet_resource_group=
+ az logout
ERROR: There are no active accounts.
+ '[' -z <subscription> ']'
+ '[' -z westus ']'
+ '[' -z docker-machine ']'
+ '[' -z '' ']'
+ echo 'Some variables were not populated correctly, using the passed config!'
+ '[' kubelet = kubelet ']'

failed to run Kubelet: could not init cloud provider "azure": No credentials provided for AAD application
@bmdepesa

This comment has been minimized.

bmdepesa commented May 2, 2018

Validated against rancher/racher:master (5/2).

Azure w/ cloud provider provisioned correctly.
Volumes, workloads, ingress, service discovery all functioning correctly.

@bmdepesa bmdepesa closed this May 2, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment