You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I, as a user with admin powers, log in. After a while, I'll notice I'm logged in as normal user "singer". I have to log out and log back in to regain my normal account and admin powers.
"bambi" and "singer" can become me by conjuring the new hover thingies and then going to a new page. This includes admin powers.
"sniper" can hit refresh repeatedly on a page and become "singer" or logged out. "sniper" is also an admin.
Note: The usernames have been changed to protect the guilty.
A better description of symptoms:
I, as a user with admin powers, log in. After a while, I'll notice I'm logged in as normal user "singer". I have to log out and log back in to regain my normal account and admin powers.
"bambi" and "singer" can become me by conjuring the new hover thingies and then going to a new page. This includes admin powers.
"sniper" can hit refresh repeatedly on a page and become "singer" or logged out. "sniper" is also an admin.
Note: The usernames have been changed to protect the guilty.
Also: I'd be happy to help troubleshoot/debug this. I can get some of the Java experts here to help me look at thread dumps or whatever else might be useful.
Also: I'd be happy to help troubleshoot/debug this. I can get some of the Java experts here to help me look at thread dumps or whatever else might be useful.
For our setup, we use Tomcat. We use LDAP exclusively, so we don't have another realm.
We are behind an apache reverse proxy.
I had "sniper" try using :8080 (the non-reverse-proxied url) and he isn't losing his login. This may be because a) he has never had a cookie/session for that URL or b) the proxy is messing things up (but how?).
PS: I'm 'docwhat' on IRC.
For our setup, we use Tomcat. We use LDAP exclusively, so we don't have another realm.
We are behind an apache reverse proxy.
I had "sniper" try using :8080 (the non-reverse-proxied url) and he isn't losing his login. This may be because a) he has never had a cookie/session for that URL or b) the proxy is messing things up (but how?).
We had a long troubleshooting session on IRC today and here is our summary (please correct if I make a mistake):
On my server, we use Tomcat.
It is behind an apache, running a reverse proxy with mod_cache and mod_*_cache.
Tomcat sends out JSESSIONID cookie when it thinks a session needs to be set and the previous JSESSIONID is expired or no JSESSIONID is set.
Our thought is that some cacheable resources had also had the "Set-Cookie: JSESSIONID" header set. So when someone else loaded that cached resource, they had their JSESSIONID set to someone elses.
This is born out by the fact that adding "CacheIgnoreHeaders Set-Cookie" to apache seems to have fixed the problem.
We had a long troubleshooting session on IRC today and here is our summary (please correct if I make a mistake):
On my server, we use Tomcat.
It is behind an apache, running a reverse proxy with mod_cache and mod_*_cache.
Tomcat sends out JSESSIONID cookie when it thinks a session needs to be set and the previous JSESSIONID is expired or no JSESSIONID is set.
Our thought is that some cacheable resources had also had the "Set-Cookie: JSESSIONID" header set. So when someone else loaded that cached resource, they had their JSESSIONID set to someone elses.
This is born out by the fact that adding "CacheIgnoreHeaders Set-Cookie" to apache seems to have fixed the problem.
Thanks to Christian for helping the trouble-shooting session.
We couldn't exactly verify our hypothesis during the session. The hypothesis hinges upon having Jenkins send "Set-Cookie" header in a static resource that then gets cached by the reverse proxy, yet the code path that serves those files do not touch HTTP sessions before committing the output, and the Acegi's attempt to create a session afterward resulted in no-op, at least with Winstone.
With that said, toward the end, our discussion was mainly around how to fix this, assuming the hypothesis is correct. Since Jenkins as a webapp doesn't exactly know when the 'Set-Cookie' header is set on the response by the container, the solution has to be done on every single response we produce.
One thought was to add "Vary: Cookie", and the other was to add "Cache-Control: private". Relevant read here, which also says IE<9 fails to locally cache the resouce if the Vary header is set (and instead if requires conditional GET.)
While driving after the trouble-shooting session, another reason "Vary: Cookie" header is wrong occured to me, which is that if a request does not contain a cookie header (say it expired) and if the response contains "Set-Cookie", "Vary: Cookie" will do no good as the said resource ends up getting cached by the intermediary.
So I'm inclined to start serving everything with "Cache-Control: private" unless I hear otherwise.
Incidentally, http://ci.jenkins-ci.org/ appears to have this header set on the response, although Jenkins doesn't do this. I suspect haproxy or Apache are doing it.
Thanks to Christian for helping the trouble-shooting session.
We couldn't exactly verify our hypothesis during the session. The hypothesis hinges upon having Jenkins send "Set-Cookie" header in a static resource that then gets cached by the reverse proxy, yet the code path that serves those files do not touch HTTP sessions before committing the output, and the Acegi's attempt to create a session afterward resulted in no-op, at least with Winstone.
With that said, toward the end, our discussion was mainly around how to fix this, assuming the hypothesis is correct. Since Jenkins as a webapp doesn't exactly know when the 'Set-Cookie' header is set on the response by the container, the solution has to be done on every single response we produce.
One thought was to add "Vary: Cookie", and the other was to add "Cache-Control: private". Relevant read here, which also says IE<9 fails to locally cache the resouce if the Vary header is set (and instead if requires conditional GET.)
While driving after the trouble-shooting session, another reason "Vary: Cookie" header is wrong occured to me, which is that if a request does not contain a cookie header (say it expired) and if the response contains "Set-Cookie", "Vary: Cookie" will do no good as the said resource ends up getting cached by the intermediary.
So I'm inclined to start serving everything with "Cache-Control: private" unless I hear otherwise.
Incidentally, http://ci.jenkins-ci.org/ appears to have this header set on the response, although Jenkins doesn't do this. I suspect haproxy or Apache are doing it.
Just a clarification. It doesn't have to be a static asset.
Tomcat could have (according to the brief description on some stackoverflow article I can't find now) sent it on any page. It doesn't send JSESSIONID on every URL requested, but has some logic for when to do it.
The only thing needed to poison apache's cache would be that:
a) A page that is cacheable. This isn't always just pages with an "Expires" or "Last-Modified" header. The rules are... a little esoteric at times, and who knows how apache interprets them.
b) A page that sent the Set-Cookie header.
Also, as you mentioned in IRC, there is no point trying to "recover" a poisoned cache. Your only real choice is to turn off the proxy-cache, or reset it with either a fix (like CacheIgnoreHeader Set-Cache) or the new as-yet-unwritten Jenkins.
Just a clarification. It doesn't have to be a static asset.
Tomcat could have (according to the brief description on some stackoverflow article I can't find now) sent it on any page. It doesn't send JSESSIONID on every URL requested, but has some logic for when to do it.
The only thing needed to poison apache's cache would be that:
a) A page that is cacheable. This isn't always just pages with an "Expires" or "Last-Modified" header. The rules are... a little esoteric at times, and who knows how apache interprets them.
b) A page that sent the Set-Cookie header.
Also, as you mentioned in IRC, there is no point trying to "recover" a poisoned cache. Your only real choice is to turn off the proxy-cache, or reset it with either a fix (like CacheIgnoreHeader Set-Cache) or the new as-yet-unwritten Jenkins.
I forgot to mention yesterday, I compared one jenkins instance with logged-in access only to one that has anonymous, both set a cookie on the start page, the logged-in instance has 403 and no no-cache header, the anonymous instance has Cache-Control: no-cache,must-revalidate.
Both looks correct and the static files do not send a set-cookie header.
I forgot to mention yesterday, I compared one jenkins instance with logged-in access only to one that has anonymous, both set a cookie on the start page, the logged-in instance has 403 and no no-cache header, the anonymous instance has Cache-Control: no-cache,must-revalidate.
Both looks correct and the static files do not send a set-cookie header.
Code changed in jenkins
User: Kohsuke Kawaguchi
Path:
changelog.html
core/src/main/java/hudson/security/AuthenticationProcessingFilter2.java
core/src/main/resources/lib/layout/layout.jelly
war/src/main/webapp/WEB-INF/security/SecurityFilters.groovy
http://jenkins-ci.org/commit/jenkins/7a4858d65f2431396c2f4dadbc3d654712bc02a8
Log:
[FIXED JENKINS-12585] restrict where sessions are created.
If a resource with 'Set-Cookie' header is cached (either by intermediary
like HTTP proxy and reverse proxy, or by the browser), it'll cause
identity swap / session mix-up as discussed in this ticket.
I suspect this was caused by HttpSessionContextIntegrationFilter2, which
is the only code path that attempts to create a session when a request
to a static resource is made.
So I'm disabling the creation of session in
HttpSessionContextIntegrationFilter2. This in turn requires that we
have sessions already created when the authentication was successful and
people need to login (or else the login will have no effect.)
We already do so in layout.jelly, so any request that renders a Jenkins
page would have a session, but I've also added it in
AuthenticationProcessingFilter2, which ensures that a successful login
does have a session.
Code changed in jenkins
User: Kohsuke Kawaguchi
Path:
changelog.html
core/src/main/java/hudson/security/AuthenticationProcessingFilter2.java
core/src/main/resources/lib/layout/layout.jelly
war/src/main/webapp/WEB-INF/security/SecurityFilters.groovy http://jenkins-ci.org/commit/jenkins/7a4858d65f2431396c2f4dadbc3d654712bc02a8
Log: [FIXED JENKINS-12585] restrict where sessions are created.
If a resource with 'Set-Cookie' header is cached (either by intermediary
like HTTP proxy and reverse proxy, or by the browser), it'll cause
identity swap / session mix-up as discussed in this ticket.
I suspect this was caused by HttpSessionContextIntegrationFilter2, which
is the only code path that attempts to create a session when a request
to a static resource is made.
So I'm disabling the creation of session in
HttpSessionContextIntegrationFilter2. This in turn requires that we
have sessions already created when the authentication was successful and
people need to login (or else the login will have no effect.)
We already do so in layout.jelly, so any request that renders a Jenkins
page would have a session, but I've also added it in
AuthenticationProcessingFilter2, which ensures that a successful login
does have a session.
Integrated in jenkins_ui-changes_branch #30[FIXED JENKINS-12585] restrict where sessions are created. (Revision 7a4858d65f2431396c2f4dadbc3d654712bc02a8)
Activity
jenkins-infra-bot commentedon Mar 27, 2012
Raw content of original comment:
I have the same issue. One user can hit refresh repeatedly and get different users or even logged out and then back in!
My setup is:
Jenkins 1.456
JRE 7u3
Plugins of interest: Role-based Authorization Strategy
Authentication: LDAP
jenkins-infra-bot commentedon Mar 27, 2012
Raw content of original comment:
+1
This is causing major issues as users are gaining unauthorized access to controls.
jenkins-infra-bot commentedon Mar 27, 2012
Raw content of original comment:
Note: The usernames have been changed to protect the guilty.
A better description of symptoms:
Note: The usernames have been changed to protect the guilty.
jenkins-infra-bot commentedon Mar 27, 2012
Raw content of original comment:
Related bug: JENKINS-13203">
JENKINS-13203This is the symptoms described with "bambi" and "singer" above (number 2).
jenkins-infra-bot commentedon Mar 27, 2012
Raw content of original comment:
Also: I'd be happy to help troubleshoot/debug this. I can get some of the Java experts here to help me look at thread dumps or whatever else might be useful.
jenkins-infra-bot commentedon Mar 27, 2012
Raw content of original comment:
Looking into this.
For other people seeing this, please report
jenkins-infra-bot commentedon Mar 27, 2012
Raw content of original comment:
For those who are seeing this issue, are you all running behind a reverse-proxy?
jenkins-infra-bot commentedon Mar 27, 2012
Raw content of original comment:
For our setup, we use Tomcat. We use LDAP exclusively, so we don't have another realm.
We are behind an apache reverse proxy.
I had "sniper" try using :8080 (the non-reverse-proxied url) and he isn't losing his login. This may be because a) he has never had a cookie/session for that URL or b) the proxy is messing things up (but how?).
PS: I'm 'docwhat' on IRC.
jenkins-infra-bot commentedon Mar 27, 2012
Raw content of original comment:
We had a long troubleshooting session on IRC today and here is our summary (please correct if I make a mistake):
On my server, we use Tomcat.
It is behind an apache, running a reverse proxy with mod_cache and mod_*_cache.
Tomcat sends out JSESSIONID cookie when it thinks a session needs to be set and the previous JSESSIONID is expired or no JSESSIONID is set.
Our thought is that some cacheable resources had also had the "Set-Cookie: JSESSIONID" header set. So when someone else loaded that cached resource, they had their JSESSIONID set to someone elses.
This is born out by the fact that adding "CacheIgnoreHeaders Set-Cookie" to apache seems to have fixed the problem.
jenkins-infra-bot commentedon Mar 27, 2012
Raw content of original comment:
Thanks to Christian for helping the trouble-shooting session.
We couldn't exactly verify our hypothesis during the session. The hypothesis hinges upon having Jenkins send "Set-Cookie" header in a static resource that then gets cached by the reverse proxy, yet the code path that serves those files do not touch HTTP sessions before committing the output, and the Acegi's attempt to create a session afterward resulted in no-op, at least with Winstone.
With that said, toward the end, our discussion was mainly around how to fix this, assuming the hypothesis is correct. Since Jenkins as a webapp doesn't exactly know when the 'Set-Cookie' header is set on the response by the container, the solution has to be done on every single response we produce.
One thought was to add "Vary: Cookie", and the other was to add "Cache-Control: private". Relevant read here, which also says IE<9 fails to locally cache the resouce if the Vary header is set (and instead if requires conditional GET.)
While driving after the trouble-shooting session, another reason "Vary: Cookie" header is wrong occured to me, which is that if a request does not contain a cookie header (say it expired) and if the response contains "Set-Cookie", "Vary: Cookie" will do no good as the said resource ends up getting cached by the intermediary.
So I'm inclined to start serving everything with "Cache-Control: private" unless I hear otherwise.
Incidentally, http://ci.jenkins-ci.org/ appears to have this header set on the response, although Jenkins doesn't do this. I suspect haproxy or Apache are doing it.
jenkins-infra-bot commentedon Mar 28, 2012
Raw content of original comment:
Just a clarification. It doesn't have to be a static asset.
Tomcat could have (according to the brief description on some stackoverflow article I can't find now) sent it on any page. It doesn't send JSESSIONID on every URL requested, but has some logic for when to do it.
The only thing needed to poison apache's cache would be that:
a) A page that is cacheable. This isn't always just pages with an "Expires" or "Last-Modified" header. The rules are... a little esoteric at times, and who knows how apache interprets them.
b) A page that sent the Set-Cookie header.
Also, as you mentioned in IRC, there is no point trying to "recover" a poisoned cache. Your only real choice is to turn off the proxy-cache, or reset it with either a fix (like CacheIgnoreHeader Set-Cache) or the new as-yet-unwritten Jenkins.
jenkins-infra-bot commentedon Mar 28, 2012
Raw content of original comment:
I forgot to mention yesterday, I compared one jenkins instance with logged-in access only to one that has anonymous, both set a cookie on the start page, the logged-in instance has 403 and no no-cache header, the anonymous instance has Cache-Control: no-cache,must-revalidate.
Both looks correct and the static files do not send a set-cookie header.
jenkins-infra-bot commentedon May 4, 2012
Raw content of original comment:
Code changed in jenkins
User: Kohsuke Kawaguchi
Path:
changelog.html
core/src/main/java/hudson/security/AuthenticationProcessingFilter2.java
core/src/main/resources/lib/layout/layout.jelly
war/src/main/webapp/WEB-INF/security/SecurityFilters.groovy
http://jenkins-ci.org/commit/jenkins/7a4858d65f2431396c2f4dadbc3d654712bc02a8
Log:
[FIXED JENKINS-12585] restrict where sessions are created.
If a resource with 'Set-Cookie' header is cached (either by intermediary
like HTTP proxy and reverse proxy, or by the browser), it'll cause
identity swap / session mix-up as discussed in this ticket.
I suspect this was caused by HttpSessionContextIntegrationFilter2, which
is the only code path that attempts to create a session when a request
to a static resource is made.
So I'm disabling the creation of session in
HttpSessionContextIntegrationFilter2. This in turn requires that we
have sessions already created when the authentication was successful and
people need to login (or else the login will have no effect.)
We already do so in layout.jelly, so any request that renders a Jenkins
page would have a session, but I've also added it in
AuthenticationProcessingFilter2, which ensures that a successful login
does have a session.
jenkins-infra-bot commentedon May 4, 2012
Raw content of original comment:
Integrated in
jenkins_main_trunk #1701
[FIXED JENKINS-12585] restrict where sessions are created. (Revision 7a4858d)
Result = UNSTABLE
Kohsuke Kawaguchi : 7a4858d65f2431396c2f4dadbc3d654712bc02a8
Files :
jenkins-infra-bot commentedon Jun 8, 2012
Raw content of original comment:
Integrated in
jenkins_ui-changes_branch #30
[FIXED JENKINS-12585] restrict where sessions are created. (Revision 7a4858d)
Result = SUCCESS
Kohsuke Kawaguchi : 7a4858d65f2431396c2f4dadbc3d654712bc02a8
Files :