RBAC on AKS: kubectl prompts for login for every command

[vijayrajagopalan-hmcts]

What happened:
Executing kubectl command prompts for authentication every time, even within the same cluster context.

Output: To sign in, use a web browser to open the page https://microsoft.com/devicelogin and enter the code XXXXXXXXX to authenticate.

Authentication succeeds. However, it prompts for login again, when executing the next kubectl command, within the same cluster context.

NOTE:

1. Have checked the workaround of a similar issue #1057. I can confirm there is no config.lock file under ~/.kube/ folder.

2. Have tried deleting the config file, cache and http-cache folders. This did not resolve the issue

What you expected to happen:
After successful login, executing kubectl command should not prompt for authentication within the same cluster context.

How to reproduce it (as minimally and precisely as possible):
az aks get-credentials --resource-group xxx--rg --name xxx-aks --subscription xxx-sub --overwrite

kubectl get pods -n xx

Anything else we need to know?:

Environment:

• Kubernetes version (use kubectl version):
  Client Version: version.Info{Major:"1", Minor:"16+", GitVersion:"v1.16.6-beta.0", GitCommit:"e7f962ba86f4ce7033828210ca3556393c377bcc", GitTreeState:"clean", BuildDate:"2020-01-15T08:26:26Z", GoVersion:"go1.13.5", Compiler:"gc", Platform:"darwin/amd64"}
  Server Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.7", GitCommit:"5737fe2e0b8e92698351a853b0d07f9c39b96736", GitTreeState:"clean", BuildDate:"2020-06-24T19:54:11Z", GoVersion:"go1.13.6", Compiler:"gc", Platform:"linux/amd64"}

[ghost]

Triage required from @Azure/aks-pm

[TomGeske]

@vijayrajagopalan-hmcts: can you try to update your kubectl to latest version?

[vijayrajagopalan-hmcts]

@TomGeske thanks for your feedback. I updated kubectl version, but that hasn't changed the behaviour. It still prompts for authentication for every kubectl command. Here are the versions after the update.

Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.6", GitCommit:"dff82dc0de47299ab66c83c626e08b245ab19037", GitTreeState:"clean", BuildDate:"2020-07-16T00:04:31Z", GoVersion:"go1.14.4", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.7", GitCommit:"5737fe2e0b8e92698351a853b0d07f9c39b96736", GitTreeState:"clean", BuildDate:"2020-06-24T19:54:11Z", GoVersion:"go1.13.6", Compiler:"gc", Platform:"linux/amd64"}

[TomGeske]

are you using managed aad experience? You can figure out by running:

az aks show --resource-group xxx--rg --name xxx-aks --query "aadProfile.managed"

[vijayrajagopalan-hmcts]

@TomGeske @weinong Executed the above command and got no results. Please let me know if you need any other details to help diagnose this issue.

[TomGeske]

Did this behavior happen all the time or did it start to happen recently?

[vijayrajagopalan-hmcts]

@TomGeske this started just over a week ago. Earlier it was working as expected. I did not make any change to the local environment (like kubectl upgrade etc). I can see the access-token, refresh-token and other config data in ~/.kube/config file.

[vijayrajagopalan-hmcts]

@TomGeske do you have any feedback?

[ghost]

Hi there 👋 AKS bot here. This issue has been tagged as needing a support request so that the AKS support and engineering teams have a look into this particular cluster/issue.

Follow the steps here to create a support ticket for Azure Kubernetes Service and the cluster discussed in this issue.

Please do mention this issue in the case description so our teams can coordinate to help you.

Thank you!

[TomGeske]

@vijayrajagopalan-hmcts: Please, share your ticket no. once you created the ticket

[SliderCO-007]

I am also observing the same issue which just started recently.

I updated the azure-cli using Homebrew but I don't believe that this issue is related to the cli.

$az aks show --resource-group my_rg --name my_aks_cluster --query "aadProfile.managed"
false

$kubectl version
To sign in, use a web browser to open the page https://microsoft.com/devicelogin and enter the code ELXNDYE7C to authenticate.
Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.6", GitCommit:"dff82dc0de47299ab66c83c626e08b245ab19037", GitTreeState:"clean", BuildDate:"2020-07-15T16:58:53Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.5", GitCommit:"e0fccafd69541e3750d460ba0f9743b90336f24f", GitTreeState:"clean", BuildDate:"2020-04-16T23:22:07Z", GoVersion:"go1.13.6", Compiler:"gc", Platform:"linux/amd64"}

[palma21]

1. Make sure you have latest and stable version of kubectl (I see some cases above where that is not the case)
2. Please validate that your AAD does not have any new policies
3. As Thomas asked, please do open a support ticket and share it so we may investigate

Thank you!

[TomGeske]

kubectl upstream bug is impacting AKS managed-AAD. You can fix by ensuring you run kubectl > 1.8.1. If you are using WSL please ensure to update kubectl in windows and wsl. We noticed odd cases in such scenarios. Ensure you re-run az aks get-credentials --resource-group my_rg --name my_aks_cluster, if you update kubectl.

If you are running AAD Integration legacy and you experience unexpected re-auth requests, please open a ticket with us.

You can verify your version by running:
$az aks show --resource-group my_rg --name my_aks_cluster --query "aadProfile.managed"

AAD Integration legacy
false

AKS-managed AAD
true

@SliderCO-007: Please, share you ticket no.

[SliderCO-007]

@TomGeske - I posted my output of aadProfile.managed query above which is 'false'.

I also posted my kubectl versions above. Brew has shown that I am using the most recent.
brew upgrade kubectl Updating Homebrew... Warning: kubectl 1.18.6 already installed

As for the ticket... I tried to create one but apparently there is a policy at the enterprise preventing me from creating one. I will need to get someone else at our company to create one.

[TomGeske]

@SliderCO-007: Your issue is most likely not related to kubectl version. Send me your cluster FQDN to thomas dot geske at company dot com

[andyzhangx]

it looks like related to PR(kubernetes/kubernetes#87507) which is fixed in 1.16.7, pls try 1.16.7 kubectl first, thanks.

or then use same kubectl version with api-server(agent node version)

[andyzhangx]

@TomGeske thanks for your feedback. I updated kubectl version, but that hasn't changed the behaviour. It still prompts for authentication for every kubectl command. Here are the versions after the update.

Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.6", GitCommit:"dff82dc0de47299ab66c83c626e08b245ab19037", GitTreeState:"clean", BuildDate:"2020-07-16T00:04:31Z", GoVersion:"go1.14.4", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.7", GitCommit:"5737fe2e0b8e92698351a853b0d07f9c39b96736", GitTreeState:"clean", BuildDate:"2020-06-24T19:54:11Z", GoVersion:"go1.13.6", Compiler:"gc", Platform:"linux/amd64"}

@vijayrajagopalan-hmcts in your case, can you try kubectl v1.17.7 since your api-server is v1.17.7? thanks.

[andyzhangx]

pls try exactly same kubectl version with api-server version

[vijayrajagopalan-hmcts]

@andyzhangx thanks for your inputs. I downgraded the version to v1.17.0 and v1.17.7, but this did not help. Any thoughts?

[vijayrajagopalan-hmcts]

@andyzhangx, an MS Engineer helped resolve this issue. Turned out, the issue is with the ~/.kube folder. We deleted the folder and the next az get cred command created this folder. After this, kubectl commands did not prompt me for authentication, just authenticate once per cluster.

This issue can be closed.