New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Azure MSI auth http 400 error #56167

Closed
r7vme opened this Issue Nov 21, 2017 · 5 comments

Comments

Projects
None yet
5 participants
@r7vme

r7vme commented Nov 21, 2017

/kind bug

What happened:
I want to use Azure MSI for k8s cloud provider. But kubelet produces errors like this:

Nov 21 17:43:41 worker-3 sh[10372]: I1121 17:43:41.907901   10395 kubelet.go:1779] skipping pod synchronization - [Kubelet failed to get node info: failed to get external ID from cloud provider: azure.BearerAuthorizer#WithAuthorization: Failed to refresh the Token for request to https://management.azure.com/subscriptions/xxx/resourceGroups/gollum/providers/Microsoft.Compute/virtualMachines/worker-3?api-version=2016-04-30-preview: StatusCode=0 -- Original Error: adal: Refresh request failed. Status Code = '400']

JFYI: everything works fine w/o MSI, with precreated service principal.

What you expected to happen:
I expect kubelet will be able to authenticate in Azure API and get necessary info.

How to reproduce it (as minimally and precisely as possible):

  • Create k8s cluster on Azure availability set VMs (Probably acs-engine can be used, we are using own terraform)
  • Enable Azure MSI in VM like described here
  • Assign "Contributor" role for VM
  • Restart kubelet

Anything else we need to know?:
cloud-provider config

cloud: AZUREPUBLICCLOUD
#tenantId: xxx
subscriptionId: xxx
#aadClientId: xxx
#aadClientSecret: xxx
resourceGroup: gollum
location: westeurope
subnetName: gollum_worker_subnet
securityGroupName: gollum-worker
vnetName: gollum
routeTableName: gollum_worker_rt
useManagedIdentityExtension: true

I can get token with

curl http://localhost:50342/oauth2/token --data "resource=https://management.azure.com/" -H Metadata:true   
{"access_token":"xxx","refresh_token":"","expires_in":"3599","expires_on":"1511290443","not_before":"1511286543","resource":"https://management.azure.com/","token_type":"Bearer"}

But i'm getting error 400, when i'm trying to use full resource url

curl -v http://localhost:50342/oauth2/token --data "resource=https://management.azure.com/subscriptions/xxx/resourceGroups/gollum/providers/Microsoft.Compute/virtualMachines/worker-3?api-version=2016-04-30-preview" -H Metadata:true 
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 50342 (#0)
> POST /oauth2/token HTTP/1.1
> Host: localhost:50342
> User-Agent: curl/7.54.0
> Accept: */*
> Metadata:true
> Content-Length: 194
> Content-Type: application/x-www-form-urlencoded
> 
* upload completely sent off: 194 out of 194 bytes
< HTTP/1.1 400 Bad Request
< Content-Type: text/plain; charset=utf-8
< X-Content-Type-Options: nosniff
< Date: Tue, 21 Nov 2017 18:12:10 GMT
< Content-Length: 855
< 
{"error":"invalid_resource","error_description":"AADSTS50001: The application named https://management.azure.com/subscriptions/xxx/resourceGroups/gollum/providers/Microsoft.Compute/virtualMachines/worker-3?api-version=2016-04-30-preview was not found in the tenant named 31f75bf9-3d8c-4691-95c0-83dd71613db8.  This can happen if the application has not been installed by the administrator of the tenant or consented to by any user in the tenant.  You might have sent your authentication request to the wrong tenant.\r\nTrace ID: 2c5d4de5-927e-45b4-8050-cd3a95941000\r\nCorrelation ID: cdcda3e8-aa0f-42f5-85e9-4979311d5880\r\nTimestamp: 2017-11-21 18:12:10Z","error_codes":[50001],"timestamp":"2017-11-21 18:12:10Z","trace_id":"2c5d4de5-927e-45b4-8050-cd3a95941000","correlation_id":"cdcda3e8-aa0f-42f5-85e9-4979311d5880"}
* Connection #0 to host localhost left intact

Environment:

  • Kubernetes version 1.7.5 or 1.8.1
  • Cloud provider or hardware configuration: azure
  • OS (e.g. from /etc/os-release): CoreOS 1520.8.0
  • Kernel (e.g. uname -a): Linux worker-3 4.13.9-coreos #1 SMP Thu Oct 26 03:21:00 UTC 2017 x86_64 Intel(R) Xeon(R) CPU E5-2673 v3 @ 2.40GHz GenuineIntel GNU/Linux
  • Install tools: own terraform manifests (source)
  • Others:
@r7vme

This comment has been minimized.

r7vme commented Nov 21, 2017

/sig azure

@andyzhangx

This comment has been minimized.

Member

andyzhangx commented Nov 22, 2017

@karataliu

This comment has been minimized.

Contributor

karataliu commented Nov 22, 2017

Same cause: #55837

MSI is still a preview feature now. The azure sdk used by k8s v1.8 did not add the required metadata header. You can try v1.9 and it should work.

@r7vme

This comment has been minimized.

r7vme commented Nov 22, 2017

thanks @karataliu this works with 1.9.0-beta.

WDYT? will this be cherry-picked/fixed for older versions? or this bug can be closed?

@r7vme

This comment has been minimized.

r7vme commented Nov 22, 2017

I see your last comment.

Closing as duplicate of #55837 . Expected to be fixed in 1.8.

@r7vme r7vme closed this Nov 22, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment