Dashboard login fails on Chrome/Firefox

[ITRON-kpalmberg]

Environment

**Cluster info:**
Kubernetes version: AKS v1.16.7
Dashboard version: v2.0.0-beta8

**Client info:**
Client Operating System: Windows 10
Kubectl version: 1.18.0
Az CLI version: 2.5.1
Chrome version: Version 81.0.4044.138 (Official Build) (64-bit)
Edge version: 44.18362.449.0
Firefox version: 76.0.1 (64-bit)

Steps to reproduce

Deploy v1.16.7 AKS cluster. By default dashboard version v2.0.0-beta8 is deployed with this cluster via AddOnManager. Try to access dashboard via kubectl proxy or az aks browse. When clicking the login button on Chrome/Firefox after entering a valid user token nothing occurs and you are left at the login page.

Observed result

After upgrading our AKS cluster to v1.16.7 this updated the dashboard version. Users were no longer able to login via Chrome/Firefox. This is due to the new login page which required config/token input to authenticate. After entering a valid token and clicking on the login button nothing occurs. When looking in browser debugging tools, 200 status codes can be seen after clicking the login button. No error messages are returned whereas if I try an intentionally invalid token I receive an error message. However, when I try the login process on Edge with the valid token the login works as expected and the dashboard loads!

Expected result

Expect to be able to login to dashboard on Chrome/Firefox/Edge.

Comments

• The login process works flawlessly on Microsoft Edge.
• Issue is present on multiple separate AKS clusters (not isolated to 1 cluster).
• Issue is seen on multiple developer machines (not isolated to 1 client).
• Originally we thought there might be an RBAC issue (even though no errors are returned after clicking login) until we discovered Edge worked just fine.
• Per the comment above, we have RBAC enabled clusters.
• The AKS admin account and service account tokens work properly when logging in via Chrome/Firefox. But when using a user account token this only works on Edge.
• I tried manually deploying newer versions of the dashboard such as v.2.0.0-rc3 and v2.0.0 and encountered the same login issues on Chrome/Firefox.

[floreks]

This is either related to #5126 which makes it a duplicate or already fixed by #5124 and will be available in the next minor release (hopefully tomorrow).

/close

[k8s-ci-robot]

@floreks: Closing this issue.

In response to this:

This is either related to #5126 which makes it a duplicate or already fixed by #5124 and will be available in the next minor release (hopefully tomorrow).

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[SayakMukhopadhyay]

@OnAzureCloud9 are you using the auth-provider.config.access-token generated after logging in through Azure AD? If so you might be facing an issue wherein the access-token provided by Azure is so long that the generated jweToken exceeds the 4KB size limit of browsers cookies. Just check the size of the token that you are entering and if it is around 2k characters long. See #3081

[ITRON-kpalmberg]

@SayakMukhopadhyay: No, I'm retrieving an access token with az account get-access-token --query accessToken -o tsv. Based on your response I checked the length of the token and it was over 3000 characters. I know the token is valid because I was able to login on Edge. I'll do some investigating on the issue you linked.

[ITRON-kpalmberg]

@SayakMukhopadhyay I thought you were referring to the size of the Azure token used to sign into the dashboard. I see now that you are referring to size of the jweToken cookie. From my successful login attempts using Edge I can see that the jweToken cookie appears after successful login. However, I don't see this cookie ever appear when I get stuck on the login screen on Chrome/Firefox in browser debugging tools so I'm not sure this is applicable. I also do not see any cookie related error output in the console such as in the post you linked.

[ITRON-kpalmberg]

@SayakMukhopadhyay Interestingly enough I think you might be on the right track. I was using Chrome as my main browser and didn't receive any console output regarding the jweToken cookie being too large. However, since I was also having issues with Firefox I decided to get that a shot. I can in fact see a warning in the console on Firefox (but not Chrome):

I'll follow up on your linked issues to see if there's any kind of resolution. I'm not sure why this error isn't appearing on Chrome but I'm assuming it's probably the same issue. Not sure why Edge accepts this yet.

[SayakMukhopadhyay]

I must say that the browser behaviour is very unreliable around this. I have used Chrome, Firefox and Edge (all latest as of this post) and none of them accepts my 2k long access_token (which creates a more than 4k long jweToken. Moreover I never got an error message in the console which is why I actually spent 3 days straight troubleshooting this issue before I literally stumbled across the issue I linked.

[ITRON-kpalmberg]

@SayakMukhopadhyay I noticed that on the "new" Edge browser it didn't work.

Left is old Edge browser logo and right is new Edge browser:

You can see my Edge version in original post (44.18362.449.0) on the old Edge browser where login succeeds. On Microsoft's new Edge browser (Version 81.0.416.77 (Official build) (64-bit)) I am also unable to login, but on the old Edge browser login succeeds. Maybe there's some difference in the cookie size limitation on the old Edge browser?

Side note: I only see the cookie size warning in debug console on Firefox. I'm unsure why this is not displaying on Chrome/Edge but I'm hoping this doesn't mean that there's another underlying issue on those browsers. It's odd that the warning isn't displayed on Chrome/Edge.

[floreks]

Max cookie size depends on the browser implementation, however, most browsers will accept a cookie only up to ~4096 bytes. As headers do not share this restriction, it's better to switch to a reverse proxy setup (in this case) and inject the token into authorization header.

[SayakMukhopadhyay]

Could be, I have tested on the chromium based edge itself as I no longer have the old edge. I don't really know of a workaround other than using a oauth proxy in the cluster, something that I am yet to try as that would need dashboard to be accessible from the internet. I don't know enough about the security of a dashboard connected to the internet and protected by a oauth proxy.

[floreks]

Proxy does not mean that you need to expose Dashboard. You can also setup some reverse proxy in your local cluster and expose it only inside the local network. Everything depends on your configuration and what you need.

[floreks]

@SayakMukhopadhyay try out a new release once travis pushes the images.

[ITRON-kpalmberg]

@floreks thanks, keeping a lookout for the release.

[floreks]

@OnAzureCloud9 it won't help with the cookie size issue but should fix a couple of other issues.

[SayakMukhopadhyay]

@SayakMukhopadhyay try out a new release once travis pushes the images.

I have tested and its working perfectly. I have commented in the original issue with some details

Proxy does not mean that you need to expose Dashboard. You can also setup some reverse proxy in your local cluster and expose it only inside the local network. Everything depends on your configuration and what you need.

By "local cluster" if you mean setting up a reverse proxy at the kubectl level using a plugin, then yeah, I have also considered it. I am pretty much weighing between allowing access to the dashboard only via oauth+proxy at kubectl vs oauth+reverse proxy at the ingress. I can imagine that local oauth+proxy at kubectl would be safer but it is not as user friendly as the oauth+reverse proxy at ingress option is.

[floreks]

The best option is to configure oauth reverse proxy with some IdP plugin that will first force you to log in using i.e. Google/Github and then K8S API will return user-specific token directly to your proxy. Proxy then takes it and appends it to authorization header before forwarding request to Dashboard. This way even if someone somehow manages to bypass proxy and access Dashboard without logging in, there will be nothing to see as Dashboard itself does not have basically any privileges. Remember to also expose your ingress over HTTPS only. Dashboard then does not need custom certificates. It can be exposed only inside the cluster.

[robbiezhang]

@floreks, how do you think about splitting the jwe and saving them into multiple cookies when it exceeds 4k?

[floreks]

I'd first try some kind of string compression to reduce cookie size. It should save us about 25% space.

[tgdfool2]

I'd first try some kind of string compression to reduce cookie size. It should save us about 25% space.

Hi @floreks. Do you plan to implement this string compression in one of the upcoming Dashboard releases? If so, then where can we follow the status? Thanks!

[floreks]

We are focused now on defining the architecture for the new API. Not sure when I'll have time to look into this.

[tgdfool2]

Great, thanks for your answer. Nice to see that it will be addressed.