Set-Cookie header is ignored due to size limit after the Kibana upgrade to 1.10.1

[arnitolog]

OpenID integration stopped working after the upgarde of Kibana to version 1.10.1.
Checking the headers I found that the set-cookie header is ignored due to the size limit.
Here is a message from the browser dev tools:
Set-Cookie header is ignored in response from url: https://logs-test.us.dev.kube.company.com/auth/openid/login?state=abKJ_B2F8UGxIw7HWbhWIp&code=a950100f-19d1-4bb6-8b50-0557362be14b.1e86c530-392c-4bd2-81fa-ca31a25db5f2.208d33ca-1735-423c-8075-8860fa0a8c34. Cookie length should be less than or equal to 4096

[zengyan-amazon]

Since this problem looks specific to OIDC, as the JWT token might be very large we might provide an config to make the refresh token optional.

A more decent solution might be encode the cookie with gzip or other compression encoding to reduce the overall cookie size, before sending it back to browser. but it requires encoding/decoding at server side, and potentially impact server side performance. And this optimization also requires bigger effort to refactor the cookie handling code.

[zengyan-amazon]

@arnitolog do you have an idea of how long your JWT tokens are?

[arnitolog]

@zengyan-amazon it is 11769 characters.

[zengyan-amazon]

@arnitolog if the id_token by itself is already more than 11k characters, and it worke with old versions like ODFE 1.9, maybe hapi/iron, the Kibana cookie encryption library has done some sort of encoding/compression when encrypting the cookie.

We need to look deeper into it and see how to optimize it to support big JWT token case.

[arnitolog]

thank you @zengyan-amazon. Meantime we will try to decrease the size of our JWT token

[sushant-pradhan-tm]

@zengyan-amazon is there some update on this issue? Its kind of a blocker

[zengyan-amazon]

@arnitolog @sushant-pradhan We tried to compress the JWT token, but it turned out that the compress ratio is not ideal.

The problem is we are using id_token as the bearer token, which includes user profile ( if you request profile scope ), that can be very large.

In your case, is your accessToken also a JWT token? if so do you know how long is your access token? If your access_token is much smaller than id_token, maybe we can consider to use access_token with custom claims to replace id_token to mitigate this issue

[arnitolog]

@zengyan-amazon I already solved this issue. I found that we sent all available groups from Keycloak in JWT token. But for ODFE we need only 1 (it can be a user role or admin role), So, I created a custom mapping in keycloak client to keep only this role and filter out everything else. And that did the trick.

[zengyan-amazon]

@arnitolog Thanks for your update.

@sushant-pradhan can you share some detail of your use case? like how long are your JWT tokens, and whether you are able to reduce the size of your token from IDP?

[mrhorvath]

We are having the same issue, though we're going against an adfs openid server. The cookie size is over 6000 characters and has been preventing us from being able to upgrade past 1.9.0. We end up seeing the set-cookie fail and get redirected back to the idp until the max redirects gets hit. Really looking forward to a fix, and I'm willing to help by answering any questions you might have about our setup.

[zengyan-amazon]

@mrhorvath thanks!

Can you share

• which OIDC IDP you are using? Does it support custom claims in access token?
• what scopes do you configured in opendistro_security.openid.scope.

and it will be very helpful if you can share the length of your id_token and access_token in your case. We are considering to allow customer use access_token (usually much smaller than id_token) with custom claim as the bearer token to solve this issue.

[sushant-pradhan-tm]

@arnitolog Thanks for your update.

@sushant-pradhan can you share some detail of your use case? like how long are your JWT tokens, and whether you are able to reduce the size of your token from IDP?

@zengyan-amazon apologies for delayed response. My setup is AzureAD tenant. I checked my access_token is about 2270 characters (1481 characters after decoding) and is not a a token with roles from IDP: so this can't be used. id_token is 1392 characters encoded (and 820 characters after decoding jwt.io) and has role info. And since I am part of 5 roles for the Azure application, it comes back with those 5 roles. 5 roles is very low, it can sometimes be pretty big (and used to work pre upgrade). Please let me know if you need more data

I am wondering if the new feature to merge id_token and refresh_token as described here be made a configurable parameter? This way, we can have a capability to turn it off and at-least have the login enabled. For now our users are totally locked out

[zengyan-amazon]

@sushant-pradhan yes we can make it configurable. It may save some space for cookie, but if you keep adding more attributes in the id_token, it may eventually go beyond the limit again.

So we'd like to understand how many roles a user can have at most, and why does one user need that many roles? Are all the roles useful to determine the user's security role?

[sushant-pradhan-tm]

@sushant-pradhan yes we can make it configurable. It may save some space for cookie, but if you keep adding more attributes in the id_token, it may eventually go beyond the limit again.

So we'd like to understand how many roles a user can have at most, and why does one user need that many roles? Are all the roles useful to determine the user's security role?

So here is my high level usecase, I serve the Kibana dashboard/visualizations per team. This is to ensure teams can view/modify their visualizations. The team level RBAC is achieved using roles from IDP. So a person can logically be part of more than one teams and he would like to see/edit dashboard for all the teams/roles he is part of. Its a typical use-case for any IDP/AD integration. Just having 5 roles is exceeding limit? Do you support only 1 role in id_token?

[zengyan-amazon]

@sushant-pradhan having 5 roles should not be a problem unless the role names extremely long. as you said your id_token is 1392 characters, and usually a refresh_token is just around a few hundred characters for most IDP, which should not cause cookie to be longer than 4KB.

do you have other cookies for the same domain? browsers have cookie size limit per domain, so if you have other cookie for your domain, it is more likely to exceed the limit. Can you try clear your browser cookie for the domain or try in incognito mode? If it doesn't help, maybe we can setup a call to sync up on your case

[sushant-pradhan-tm]

@zengyan-amazon yes I have tried that already, cleaned up browser cookies, and tried from incognito but no help. I am still confused that a small id_token, how is it exceeding limits now? Earlier (prior to upgrade) it used to work even after considering cookies from same domain.

Yes I am open for a call to show my env/configuration and IDP response. Let me know how to proceed with setting up a call?

[sushant-pradhan-tm]

@zengyan-amazon Thanks for taking up the enhancement and making code change. I see the PR is merged but still there is no new release. Do you know when we will have this available? Is it part of the next major release? If so, any timeline?

[sushant-pradhan-tm]

@zengyan-amazon is this released part of 1.12.0? I cant find as per the release notes. Nor is there a release for this repo/plugin

[yuxitang-amzn]

@sushant-pradhan-tm Yes it is part of 1.12. We didn't put into release note since we also merge that into main (still 1.11 right now) so it is not a new feature of 1.12.

[sushant-pradhan-tm]

@yizheliu-amazon thanks for the update. Can you share the usage also? like how can the flag be disabled/enabled? I am using helm charts for deploying opendistro

[yizheliu-amazon]

Hi @sushant-pradhan-tm, I guess you might want to ask @yuxitang-amzn or @zengyan-amazon, rather than me.

[yuxitang-amzn]

@sushant-pradhan-tm Please add opendistro_security.openid.refresh_tokens: false to your kibana.yml config file.

[pplotka]

Hi!
I'm using Gitlab OpenId and I still have an issue "Too many redirects", even after added opendistro_security.openid.refresh_tokens: false. Are there any tips to debug why this problem occurs in my environment.
I have tested on version 1.13.0 (Docker). The last version that works is 1.8.

[hari819]

hi ,
i have a situation like , the same version of kibana with security plugin version 1.10.1.2 running fine in production and secondary , but when i try to setup same stuff in another environment i am continuously getting this "too many redirects" error even with same Azure AD configuration params , too much confusion whats heppening inside Kiabana,

i checked kibana logs , Elastic logs , nginx ingress logs , only error i see is ,

{"type":"log","@timestamp":"2021-03-05T05:57:05Z","tags":["error","plugins","opendistroSecurityKibana"],"pid":10,"message":"OpenId authentication failed: Error: Response Error: 401 Unauthorized"}

the same Azure AD credentials with same RBAC settings is working fine in other environments,

[mjiderhamn]

Looks like this issue is being addressed in https://git.floragunn.com/search-guard/search-guard-kibana-plugin/-/issues/307

[dobharweim]

Hi team - has there been any work/workarounds on this?

[indiealexh]

Has there been any work toward a full solution to this issue? The size of my token should not be a factor here.

[dobharweim]

@zengyan-amazon We are seeing the same issue. Our IdP sends back all group memberships by default - they can return all groups or no groups but not a custom selection. Some of our members (me included 😅) are members of 70+ groups!

I was wondering if we could look at a minimum impact fix, maybe add an parameter to the config which would contain group names that we are interested in for opensearch, then when we are adding the roles to the token only add roles from that parameter. If the parameter is not set proceed as normal.

I believe that logic is handled server side in the HTTPJwtAuthenticator class?

If you think this would be acceptable I can do a draft PR but would like your input on it!

I have currently gotten a workaround in place where I disable groups from the IdP set the roles_key to emailAddress and use helm charts with a workaround Jenkins reconciler job to update membership. It is not ideal.

[csashi]

can you pls share tentative release date for below?

#1352

[davidlago]

@csashi 2.7.0 is scheduled to go out tomorrow 4/24

[csashi]

@davidlago thanks for the update. 2.7 is not released yet. can you share the new release date?

[davidlago]

Currently we are targeting 5/2, apologies for the delay.