Bad request - Cookies exceeds server limit for header size (bad browsers doesn't delete expired cookies)

[norrs]

We have a few users who always seem to end up in the state where they have to delete all the state cookies that is generated by mod_auth_openidc to get back in.

The web server ends up answering:

"Bad Request

Your browser sent a request that this server could not understand.
Size of a request header field exceeds server limit.

Cookie"

I think this happens due to a time out of the session, and the state cookies seems to pile up and not getting deleted even tho the cookies have a timeout. At least the problem is related to cookies getting piled up and not deleted when a new state cookie is created.

Our user which is most annoyed by this is using Firefox 42 on Ubuntu 14.04.

Is anyone else experiencing this?
I also think this might be a Firefox bug… ( https://bugzilla.mozilla.org/show_bug.cgi?id=576347 )

https://groups.google.com/forum/#!searchin/mod_auth_openidc/cookies/mod_auth_openidc/qRLKZwddpqo/nmT1ezVzCwAJ

[zandbelt]

just to clarify: the fact that expired cookies are not purged in the browser does not mean that they get sent on every request to the server; so the actual error is not directly caused by the browser retaining expired cookies in its own cache (though there's some relation since the cookies need to be generated in the first place)

the cause of cookies piling up in a short period of time is possibly due to some other misconfiguration or mismatch that is going on; it may for example happen when a large number of unauthenticated requests are fired off from the same browser (e.g. perhaps when opening a large number of pages in parallel tabs); it would be helpful to investigate under which circumstances this occurs to see if this needs mitigation in the module code itself or in the configuration of the protected paths; any info is appreciated

[bertramn]

@norrs I can certainly attest you are not the only one with this problem. Although I have never seen this problem in production, I do experience this during development occasionally. Clearing the cookies will typically fix it. I noticed when this happens that there are a large number of cookies containing a different nounce (can't recall on top of my head what the cookie name was). I'll post the set of cookies when it happens again but it sure looks like a pattern in which these accumulate.

[zandbelt]

@bertramn do you see it with Firefox only or other browsers as well?

This happening in development may occur because you fire off a large number of authentication requests that never complete (upon receiving an authentication response the cookie should be cleared).

[bertramn]

I primarily use Chrome (latest) and IE 11 during development and I encountered the problem in both from time to time.

[zandbelt]

so that actually makes sense and proves the point I was making in the first comment: the expired cookies piling up in Firefox browsers and the "header size limit" exceeded are different problems; can you look in to the circumstances under which this happens (presumably a large number of unauthenticated parallel requests)

[zandbelt]

to mitigate the Firefox issue I've changed the state cookie behavior in a new branch: https://github.com/pingidentity/mod_auth_openidc/tree/fix-firefox-cookie-storage

the state cookies are session cookies now so they won't pile up in the browser; also I'm now performing a cleaning cycle on expired cookies when setting and receiving state cookies

this does not prevent a browser from DOSing itself by firing off a large number of unauthenticated requests (in fact this mechanism prevents the server from being DOSed) but it should fix the Firefox issue and reduce the overhead on bursty unauthenticated request traffic

it would be great if you could test this and provide your feedback here

[zandbelt]

any news on this one?

[norrs]

Our colleague ( @hogneh ) which is using firefox actually got this again, but he didn't record any HAR file from firefox when it happen. I've instructed him that he needs to record this if this happens again and contact you @zandbelt with the logs to get help to diagnose this issue.

But I'll be honest; @hogneh is having a million tabs open in firefox, so it could be there is some issues related to how many tabs he keeps open and/or when session times out (firefox putting tabs to sleep?), maybe it doesn't delete the cookies as it should and it ends up in a weird state. But this is all guesses... as I simply have no clue.

(I don't have this problem when using Chrome on my Debian Sid boxes)

Maybe @bertramn is able to provide a request log for you to diagnose?

[zandbelt]

@norrs are you using the "fix" branch?

[norrs]

@zandbelt : nope, we are not. We're using the latest 1.8.7 release, but we can test the fix branch. Are you able to ship the RPM for centos7 for me if you have a build env up? We only had build setup for mod_auth_openidc against a customized apache build for centos6, but we've moved on to centos7 and migrated to the official packages since then. (Just send em to my work email, which I assume you have in your mailbox history somewhere :))

[zandbelt]

Here's a build:
https://mod-auth-openidc.org/downloads/mod_auth_openidc-1.8.8cf-1.el7.centos.x86_64.rpm

(cf stands for cookie fix...)

[mattmoyer]

I'm also seeing this kind of behavior. I'm running Apache w/mod_auth_openidc as a reverse proxy in front of some Javascript-heavy web apps (e.g., uchiwa).

My understanding of the situation is basically the same described above:

• The browser app is left running in a tab for a long time (overnight) and continues to make background requests via JS.
• At some point, the session times out and it starts receiving redirects to refresh its session. Each of these requests is setting a new state cookie.
• The Javascript code doesn't handle this case, so it just keeps blindly sending requests and accumulating state cookies until it exceeds the max request cookie size limit and starts getting 401 errors as mentioned at the top of this issue.

Clearing cookies in the browser is a working stopgap but it's been a big annoyance for users. It looks like 1724d01 is likely a fix and I'm happy to do some testing as well.

[zandbelt]

It would be great if you could test the new behaviour but I don't think it will do what you're looking for: state cookies will still be set for each of the requests, except they won't be persistent and pile up in a Firefox browser. This fix is basically a workaround for the Firefox bug.

It seems that you are better of protecting the Javascript paths with OIDCUnAuthAction 401, see: https://github.com/pingidentity/mod_auth_openidc/blob/master/auth_openidc.conf#L612

[mattmoyer]

It seems that you are better of protecting the Javascript paths with OIDCUnAuthAction 401

Ah, thanks for the tip!

[norrs]

@mattmoyer : I've suspected this as well, but using OIDCUnAuthAction is only viable if you configure it properly for all applications that you protect with mod_auth_openidc. It doesn't make it easier when apps have same endpoint for JS endpoints and normal webbrowsing by analyzing content-type in the request.

Anyhow, we're now running the "cf (cookie fix)" release, and @hogneh will report back how it goes. But we might just deploy and use server side caching with redis probably ourselves in the office; we've just been lazy and haven't bothered deploying it. :-)

[ghost]

Hi

Using the fix branch I am seeing the following behaviour on servers where we use client cookies and no server side cache with redis:

• While my session is valid I am only seeing the session cookie as I expect and no state cookies
• When my session times out and I haven't refreshed it, my session cookie goes away and state cookies start to pile up fast.
• This continues until I again refresh my session and the state cookies stop piling up. I usually have to remove them to be able to refresh my session as I get the cookie error in the browser
• After refreshing my session I again have a session cookie and whatever amount of state cookies that had piled up in the period before I could get the session refreshed and I had removed the state cookies
• If I then again remove all left-over state cookies they don't come back again until my session times out

I am not sure what is the intended behaviour. Am I supposed to get the state cookie pile-up when I don't have a session cookie?

[zandbelt]

It seems you're running in to the same issue as @mattmoyer; Javascript that fires off a lot of parallel requests. In that case, as noted in the earlier comment, you should be using OIDCUnAuthAction 401 on the Javascript paths.

This fix is for the Firefox issue of expired cookies still piling up.

[norrs]

@zandbelt : so they will only stop piling up if you ensure to configure proper OIDCUnAuthAction?

I guess this won't be a problem if we move over to server side cache with redis right?

[zandbelt]

using OIDCUnAuthAction on paths that wouldn't appropriately redirect the user and succesfully authenticate him anyhow (e.g. Javascript paths) would stop state cookies piling up

Redis caching or any other type of caching won't make a difference since that is about the session cookie, not the state oookie and moreover, this problem about the number of cookies, not the size.

in general I don't think this issue is something that can (or should) be fixed in code: using OIDC redirects on paths that cannot authenticate the user in a browser just doesn't make sense and trying to workaround it in code just makes things difficult; using OIDCUnAuthAction 401 on those paths is the proper solution

[norrs]

Maybe this should be mentioned in the README as well that if you don't configure up all your ajax paths with proper OIDCUnAuthAction 401 you will most likely end up in this situation.

It does indeed make sense that Google doesn't allow to refresh session from XHR requests. (dejavu)

But would it be an idea to support «X-Requested-With» to check if it's set to XMLHttpRequest to set OIDCUnAuthAction 401 automatically?

It's a defacto standard AFAIK, and for example used pr default in jQuery. ( http://api.jquery.com/jquery.ajax/ , see settings option headers)

https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers

But yeah, I guess we could close this issue as there really is no fix for it it seems as cookies still piles up when session times out and requests are issued by javascript.

a mitigation would be to support the X-Requested-With, but that just helps it to prevent the cookie pile up if javascript requests is issued with this header

[zandbelt]

I totally agree with those suggestions: I'll create a Wiki page to describe this issue and the use of OIDCUnAuthAction, point to it from the README and I'll look in to X-Requested-With separately.

[zandbelt]

please see if the previous commit works for you

[norrs]

@zandbelt : Could I request a new RPM? :') (still doesn't have a build env towards centos7)

[zandbelt]

https://mod-auth-openidc.org/downloads/mod_auth_openidc-1.8.8rc3-1.el7.centos.x86_64.rpm

[norrs]

@zandbelt : I think something is missing:

AH00548: NameVirtualHost has no effect and will be removed in the next release /etc/httpd/conf.d/admin.conf:1
Feb 22 13:21:55  httpd[17121]: AH00526: Syntax error on line 43 of /etc/httpd/conf.d/oauth.conf:
Feb 22 13:21:55  httpd[17121]: parameter must be one of 'auth', 'pass', '401' or '401xrw'

OIDCUnAuthAction "401xrw"  was in config. 

[zandbelt]

yeah sorry, that was a leftover of an earlier tryout; I removed all references to 401xrw in the latest commit; you should be able just stick with the default (OIDCUnAuthAction auth) since the suggested behavior is the default now (so forget about OIDCUnAuthAction 401xrw)

[zandbelt]

I merged the Firefox fix in to the trunk so that the state cookies are session cookies now and expired cookies are cleaned explicitly. That mitigates the Firefox bug.

As commented before, the X-Requested-With header is used to stop creating state cookies on Javascript calls.

I've added a Wiki page on the use of cookies here: https://github.com/pingidentity/mod_auth_openidc/wiki/Cookies

If there's any other problem I suggest creating a new issue.

[zandbelt]

@norrs I'd still like to see you confirming that the desired effect is working for calls that contain X-Requested-With header, so in essence that the 1.8.8rc3 build works for you OOTB (no reconfig should be required); I'll then release 1.8.8

[zandbelt]

Note that the upcoming 2.1.6 release will tighten up the behavior wrt. X-Requested-With and only treat the XMLHttpRequest value as a non-browser client whereas before (>=1.8.8 <= 2.1.5) it would handle any X-Requested-With header as such regardless of its value.