New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cannot get AuthContext without subject (Invalid session?) #2710

Closed
alanjjenkins opened this Issue Aug 25, 2017 · 13 comments

Comments

Projects
None yet
@alanjjenkins

alanjjenkins commented Aug 25, 2017

Issue type: Bug report/Enhancement Request

My Rundeck detail

  • Rundeck version: 2.9.2
  • install type: RPM
  • OS Name/version: CentOS Linux release 7.2.1511 (Core)
  • DB Type/version: h2

Expected Behavior

Not to get an exception when using Rundeck.

Actual Behavior

While using Rundeck we receive this exception:

getAuthContextForSubject: Cannot get AuthContext without subject. Stacktrace follows:
java.lang.RuntimeException: getAuthContextForSubject: Cannot get AuthContext without subject
at rundeck.services.FrameworkService.getAuthContextForSubject(FrameworkService.groovy:603)
  at rundeck.services.FrameworkService.userAuthContext(FrameworkService.groovy:590)
  at rundeck.filters.ProjectSelectFilters$_closure1$_closure2$_closure3.doCall(ProjectSelectFilters.groovy:77)
  at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:696)
  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1515)
  at grails.plugin.cache.web.filter.PageFragmentCachingFilter.doFilter(PageFragmentCachingFilter.java:198)
  at grails.plugin.cache.web.filter.AbstractFilter.doFilter(AbstractFilter.java:63)
  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1486)
  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1486)
  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1486)
  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1486)
  at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:519)
  at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:138)
  at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:582)
  at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:213)
  at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1097)
  at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:448)
  at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:175)
  at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1031)
  at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:136)
  at org.eclipse.jetty.server.Dispatcher.forward(Dispatcher.java:261)
  at org.eclipse.jetty.server.Dispatcher.forward(Dispatcher.java:101)
  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1486)
  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1486)
  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1486)
  at com.codahale.metrics.servlet.AbstractInstrumentedFilter.doFilter(AbstractInstrumentedFilter.java:97)
  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1486)
  at com.dtolabs.rundeck.server.filters.AuthFilter.doFilter(AuthFilter.java:74)
  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1486)
  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1486)
  at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:519)
  at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:138)
  at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:529)
  at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:213)
  at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1097)
  at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:448)
  at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:175)
  at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1031)
  at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:136)
  at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
  at org.eclipse.jetty.server.Server.handle(Server.java:446)
  at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:271)
  at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:246)
  at org.eclipse.jetty.io.AbstractConnection$ReadCallback.run(AbstractConnection.java:358)
  at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:601)
  at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:532)
  at java.lang.Thread.run(Thread.java:748)

This then causes Rundeck to cease working correctly until the service is restarted. It appears this is a result of a corrupted / invalid session object that does not contain a subject.

How to reproduce Behavior

Unfortunately we are not certain what causes this bug to trigger but I have found others discussing it at:

https://groups.google.com/forum/#!topic/rundeck-discuss/oOlp0XQ1bys

I tried to trace the exception back and these are the lines of code I found:

The exception that is triggered is here:

https://github.com/rundeck/rundeck/blob/master/rundeckapp/grails-app/services/rundeck/services/FrameworkService.groovy#L601-L606

And the function that calls it is here:

https://github.com/rundeck/rundeck/blob/master/rundeckapp/grails-app/services/rundeck/services/FrameworkService.groovy#L588-L593

But the problem seems to start here:

https://github.com/rundeck/rundeck/blob/master/rundeckapp/grails-app/conf/rundeck/filters/ProjectSelectFilters.groovy#L75-L77

However I was not able to trace it back to the session creation code to be able to determine how a session could exist and not contain a subject.

@gschueler

This comment has been minimized.

Show comment
Hide comment
@gschueler

gschueler Aug 25, 2017

Member

the only case where i've seen that happen is if the web app container auth constraint is disabled (e.g. for pre-authenticated mode), in which case the container isn't enforcing a log in before loading application pages, and if the pre-auth mode is not configured correctly, a HTTP request to the app might occur without the appropriate subject in the session.

how is your authentication configured? what pages/urls seem to cause the problem?

Member

gschueler commented Aug 25, 2017

the only case where i've seen that happen is if the web app container auth constraint is disabled (e.g. for pre-authenticated mode), in which case the container isn't enforcing a log in before loading application pages, and if the pre-auth mode is not configured correctly, a HTTP request to the app might occur without the appropriate subject in the session.

how is your authentication configured? what pages/urls seem to cause the problem?

@alanjjenkins

This comment has been minimized.

Show comment
Hide comment
@alanjjenkins

alanjjenkins Sep 4, 2017

Hey @gschueler,

Sorry for the delay in replying I was on holiday.

The authentication system in use is default (using: org.eclipse.jetty.jaas.spi.PropertyFileLoginModule) and the users and passwords are stored in the realm.properties file with CRYPT passwords and the groups that the user should be assigned to.

We are unsure what page triggers the exception but once it has been triggered it causes everyone to be unable to login to the server. Would it be possible to add a check to the code so that if a session is found without subject then the session is invalidated (effectively logging out that user) which would prevent this from causing everyone to be unable to login.

Thanks,

Alan Jenkins

alanjjenkins commented Sep 4, 2017

Hey @gschueler,

Sorry for the delay in replying I was on holiday.

The authentication system in use is default (using: org.eclipse.jetty.jaas.spi.PropertyFileLoginModule) and the users and passwords are stored in the realm.properties file with CRYPT passwords and the groups that the user should be assigned to.

We are unsure what page triggers the exception but once it has been triggered it causes everyone to be unable to login to the server. Would it be possible to add a check to the code so that if a session is found without subject then the session is invalidated (effectively logging out that user) which would prevent this from causing everyone to be unable to login.

Thanks,

Alan Jenkins

@dirkniblick

This comment has been minimized.

Show comment
Hide comment
@dirkniblick

dirkniblick Sep 20, 2017

I'd like to bump this one up too. I had pretty much the same error and behavior of the application - the website stopped functioning for those of us already logged in, new users couldn't get logged in, and, most importantly, jobs weren't executing anymore. You can see an error below similar to the one reported by @alanjjenkins (from service.log):

INFO  ExecutionUtilService: Execution successful: 77452 in project XXX
INFO  ExecutionService: updated scheduled Execution
INFO  ExecutionUtilService: Execution successful: 77453 in project XXX
INFO  ExecutionService: updated scheduled Execution
ERROR ExecutionUtilService: Execution failed: 77455 in project XXX: [Workflow result: , step failures: {1=Dispatch faile
d on 1 nodes: [xxx: NonZeroResultCode: Result code was 127 + {dataContext=MultiDataContextImpl(map={ContextView(no
de:xxx)=BaseDataContext{{exec={exitCode=0}}}, ContextView(step:1, node:xxx)=BaseDataContext{{exec={exitCode=
0}}}}, base=null)} ]}, Node failures: {xxx=[NonZeroResultCode: Result code was 127 + {dataContext=MultiDataContext
Impl(map={ContextView(node:scheduler)=BaseDataContext{{exec={exitCode=0}}}, ContextView(step:1, node:xxx)=BaseData
Context{{exec={exitCode=0}}}}, base=null)} ]}, status: failed]
INFO  ExecutionUtilService: Execution successful: 77456 in project XXX
INFO  ExecutionUtilService: Execution successful: 77460 in project XXX
INFO  ExecutionUtilService: Execution successful: 77454 in project XXX
INFO  ExecutionUtilService: Execution successful: 77459 in project XXX
INFO  ExecutionUtilService: Execution successful: 77457 in project XXX
INFO  ExecutionUtilService: Execution successful: 77458 in project XXX
2017-09-20 10:48:22.543:INFO:cdrjj.JettyCachingLdapLoginModule:qtp1480010240-9619: Login attempts: 2, Hits: 0, Ratio: 0%
.
2017-09-20 10:48:22.543:DBUG:cdrjj.JettyCachingLdapLoginModule:qtp1480010240-9619: Cache Miss for jlp1.
2017-09-20 10:48:22.543:DBUG:cdrjj.JettyCachingLdapLoginModule:qtp1480010240-9619: Searching for users with filter: '(&(
objectClass={0})({1}={2}))' from base dn: DC=xxx,DC=xxx,DC=com
2017-09-20 10:48:22.546:DBUG:cdrjj.JettyCachingLdapLoginModule:qtp1480010240-9619: Found user?: true
2017-09-20 10:48:22.546:INFO:cdrjj.JettyCachingLdapLoginModule:qtp1480010240-9619: Attempting authentication: CN=xxx
L xxx,OU=xxx,OU=xxx,DC=xxx,DC=xxx,DC=com
2017-09-20 10:48:22.795:DBUG:cdrjj.JettyCachingLdapLoginModule:qtp1480010240-9619: JettyCachingLdapLoginModule: User 'jl
p1' has roles: [xxx, RunDeckUsers, RunDeckAdministrators, xxx]
2017-09-20 10:48:22.795:DBUG:cdrjj.JettyCachingLdapLoginModule:qtp1480010240-9619: Adding jlp1 set to expire: 1505922502
79530000
ERROR GrailsExceptionResolver: RuntimeException occurred when processing request: [GET] /
getAuthContextForSubject: Cannot get AuthContext without subject. Stacktrace follows:
java.lang.RuntimeException: getAuthContextForSubject: Cannot get AuthContext without subject
        at rundeck.services.FrameworkService.getAuthContextForSubject(FrameworkService.groovy:604)
        at rundeck.services.FrameworkService.userAuthContext(FrameworkService.groovy:591)
        at rundeck.filters.ProjectSelectFilters$_closure1$_closure2$_closure3.doCall(ProjectSelectFilters.groovy:77)
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:696)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1515)
        at grails.plugin.cache.web.filter.PageFragmentCachingFilter.doFilter(PageFragmentCachingFilter.java:198)
        at grails.plugin.cache.web.filter.AbstractFilter.doFilter(AbstractFilter.java:63)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1486)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1486)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1486)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1486)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:519)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:138)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:582)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:213)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1097)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:448)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:175)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1031)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:136)
        at org.eclipse.jetty.server.Dispatcher.forward(Dispatcher.java:261)
        at org.eclipse.jetty.server.Dispatcher.forward(Dispatcher.java:101)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1486)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1486)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1486)
        at com.codahale.metrics.servlet.AbstractInstrumentedFilter.doFilter(AbstractInstrumentedFilter.java:97)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1486)
        at com.dtolabs.rundeck.server.filters.AuthFilter.doFilter(AuthFilter.java:74)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1486)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1486)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:519)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:138)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:529)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:213)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1097)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:448)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:175)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1031)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:136)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
        at org.eclipse.jetty.server.Server.handle(Server.java:446)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:271)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:246)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.run(AbstractConnection.java:358)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:601)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:532)
        at java.lang.Thread.run(Thread.java:748)

Prior to the error, you'll notice that execution 77455 failed. This happened at 9:50 am. The other six jobs scheduled to execute at the same time as 77455 (e.g., 77456, 77460, etc.) completed their work in a second or two, but they didn't "finish" until I stopped and started Rundeck, which then marked them as "incomplete" after 1h 31m 27s. You can see that no other jobs were executed after 77455 failed and nothing else was logged to service.log until one of my coworkers tried to login at 10:48 am. They got ahold of me and I restarted the app.

We didn't notice anything had happened so I'll be writing a log monitor for rundeck... Anyhow, did you have any failed jobs prior to your freeze, @alanjjenkins?

Edit: I'm on rundeck 2.9.3-1 via RPMs on CentOS 7.

dirkniblick commented Sep 20, 2017

I'd like to bump this one up too. I had pretty much the same error and behavior of the application - the website stopped functioning for those of us already logged in, new users couldn't get logged in, and, most importantly, jobs weren't executing anymore. You can see an error below similar to the one reported by @alanjjenkins (from service.log):

INFO  ExecutionUtilService: Execution successful: 77452 in project XXX
INFO  ExecutionService: updated scheduled Execution
INFO  ExecutionUtilService: Execution successful: 77453 in project XXX
INFO  ExecutionService: updated scheduled Execution
ERROR ExecutionUtilService: Execution failed: 77455 in project XXX: [Workflow result: , step failures: {1=Dispatch faile
d on 1 nodes: [xxx: NonZeroResultCode: Result code was 127 + {dataContext=MultiDataContextImpl(map={ContextView(no
de:xxx)=BaseDataContext{{exec={exitCode=0}}}, ContextView(step:1, node:xxx)=BaseDataContext{{exec={exitCode=
0}}}}, base=null)} ]}, Node failures: {xxx=[NonZeroResultCode: Result code was 127 + {dataContext=MultiDataContext
Impl(map={ContextView(node:scheduler)=BaseDataContext{{exec={exitCode=0}}}, ContextView(step:1, node:xxx)=BaseData
Context{{exec={exitCode=0}}}}, base=null)} ]}, status: failed]
INFO  ExecutionUtilService: Execution successful: 77456 in project XXX
INFO  ExecutionUtilService: Execution successful: 77460 in project XXX
INFO  ExecutionUtilService: Execution successful: 77454 in project XXX
INFO  ExecutionUtilService: Execution successful: 77459 in project XXX
INFO  ExecutionUtilService: Execution successful: 77457 in project XXX
INFO  ExecutionUtilService: Execution successful: 77458 in project XXX
2017-09-20 10:48:22.543:INFO:cdrjj.JettyCachingLdapLoginModule:qtp1480010240-9619: Login attempts: 2, Hits: 0, Ratio: 0%
.
2017-09-20 10:48:22.543:DBUG:cdrjj.JettyCachingLdapLoginModule:qtp1480010240-9619: Cache Miss for jlp1.
2017-09-20 10:48:22.543:DBUG:cdrjj.JettyCachingLdapLoginModule:qtp1480010240-9619: Searching for users with filter: '(&(
objectClass={0})({1}={2}))' from base dn: DC=xxx,DC=xxx,DC=com
2017-09-20 10:48:22.546:DBUG:cdrjj.JettyCachingLdapLoginModule:qtp1480010240-9619: Found user?: true
2017-09-20 10:48:22.546:INFO:cdrjj.JettyCachingLdapLoginModule:qtp1480010240-9619: Attempting authentication: CN=xxx
L xxx,OU=xxx,OU=xxx,DC=xxx,DC=xxx,DC=com
2017-09-20 10:48:22.795:DBUG:cdrjj.JettyCachingLdapLoginModule:qtp1480010240-9619: JettyCachingLdapLoginModule: User 'jl
p1' has roles: [xxx, RunDeckUsers, RunDeckAdministrators, xxx]
2017-09-20 10:48:22.795:DBUG:cdrjj.JettyCachingLdapLoginModule:qtp1480010240-9619: Adding jlp1 set to expire: 1505922502
79530000
ERROR GrailsExceptionResolver: RuntimeException occurred when processing request: [GET] /
getAuthContextForSubject: Cannot get AuthContext without subject. Stacktrace follows:
java.lang.RuntimeException: getAuthContextForSubject: Cannot get AuthContext without subject
        at rundeck.services.FrameworkService.getAuthContextForSubject(FrameworkService.groovy:604)
        at rundeck.services.FrameworkService.userAuthContext(FrameworkService.groovy:591)
        at rundeck.filters.ProjectSelectFilters$_closure1$_closure2$_closure3.doCall(ProjectSelectFilters.groovy:77)
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:696)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1515)
        at grails.plugin.cache.web.filter.PageFragmentCachingFilter.doFilter(PageFragmentCachingFilter.java:198)
        at grails.plugin.cache.web.filter.AbstractFilter.doFilter(AbstractFilter.java:63)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1486)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1486)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1486)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1486)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:519)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:138)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:582)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:213)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1097)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:448)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:175)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1031)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:136)
        at org.eclipse.jetty.server.Dispatcher.forward(Dispatcher.java:261)
        at org.eclipse.jetty.server.Dispatcher.forward(Dispatcher.java:101)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1486)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1486)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1486)
        at com.codahale.metrics.servlet.AbstractInstrumentedFilter.doFilter(AbstractInstrumentedFilter.java:97)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1486)
        at com.dtolabs.rundeck.server.filters.AuthFilter.doFilter(AuthFilter.java:74)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1486)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1486)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:519)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:138)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:529)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:213)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1097)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:448)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:175)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1031)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:136)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
        at org.eclipse.jetty.server.Server.handle(Server.java:446)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:271)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:246)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.run(AbstractConnection.java:358)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:601)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:532)
        at java.lang.Thread.run(Thread.java:748)

Prior to the error, you'll notice that execution 77455 failed. This happened at 9:50 am. The other six jobs scheduled to execute at the same time as 77455 (e.g., 77456, 77460, etc.) completed their work in a second or two, but they didn't "finish" until I stopped and started Rundeck, which then marked them as "incomplete" after 1h 31m 27s. You can see that no other jobs were executed after 77455 failed and nothing else was logged to service.log until one of my coworkers tried to login at 10:48 am. They got ahold of me and I restarted the app.

We didn't notice anything had happened so I'll be writing a log monitor for rundeck... Anyhow, did you have any failed jobs prior to your freeze, @alanjjenkins?

Edit: I'm on rundeck 2.9.3-1 via RPMs on CentOS 7.

@akh13

This comment has been minimized.

Show comment
Hide comment
@akh13

akh13 Sep 26, 2017

Bump. Ran into this error too. Same software version, Amazon Linux 2017.03. The software ran fine for about 4-5 hours, then stopped processing jobs. Attempts to log in would result in the error. After restarting the process, Rundeck would again execute jobs and allow login. I ended up exporting all of my projects and importing them into a clean build. This seems to have solved the issue for myself.

ERROR GrailsExceptionResolver: RuntimeException occurred when processing request: [GET] / getAuthContextForSubject: Cannot get AuthContext without subject. Stacktrace follows: java.lang.RuntimeException: getAuthContextForSubject: Cannot get AuthContext without subject at rundeck.services.FrameworkService.getAuthContextForSubject(FrameworkService.groovy:603) at rundeck.services.FrameworkService.userAuthContext(FrameworkService.groovy:590) at rundeck.filters.ProjectSelectFilters$_closure1$_closure2$_closure3.doCall(ProjectSelectFilters.groovy:77) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:696) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1515) at grails.plugin.cache.web.filter.PageFragmentCachingFilter.doFilter(PageFragmentCachingFilter.java:198) at grails.plugin.cache.web.filter.AbstractFilter.doFilter(AbstractFilter.java:63) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1486) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1486) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1486) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1486) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:519) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:138) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:582) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:213) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1097) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:448) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:175) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1031) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:136) at org.eclipse.jetty.server.Dispatcher.forward(Dispatcher.java:261) at org.eclipse.jetty.server.Dispatcher.forward(Dispatcher.java:101) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1486) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1486) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1486) at com.codahale.metrics.servlet.AbstractInstrumentedFilter.doFilter(AbstractInstrumentedFilter.java:97) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1486) at com.dtolabs.rundeck.server.filters.AuthFilter.doFilter(AuthFilter.java:74) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1486) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1486) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:519) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:138) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:529) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:213) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1097) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:448) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:175) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1031) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:136) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) at org.eclipse.jetty.server.Server.handle(Server.java:446) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:271) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:246) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.run(AbstractConnection.java:358) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:601) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:532) at java.lang.Thread.run(Thread.java:748) ERROR GrailsExceptionResolver: RuntimeException occurred when processing request: [GET] / getAuthContextForSubject: Cannot get AuthContext without subject. Stacktrace follows: java.lang.RuntimeException: getAuthContextForSubject: Cannot get AuthContext without subject at rundeck.services.FrameworkService.getAuthContextForSubject(FrameworkService.groovy:603) at rundeck.services.FrameworkService.userAuthContext(FrameworkService.groovy:590) at rundeck.filters.ProjectSelectFilters$_closure1$_closure2$_closure3.doCall(ProjectSelectFilters.groovy:77) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:696) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1515) at grails.plugin.cache.web.filter.PageFragmentCachingFilter.doFilter(PageFragmentCachingFilter.java:198) at grails.plugin.cache.web.filter.AbstractFilter.doFilter(AbstractFilter.java:63) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1486) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1486) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1486) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1486) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:519) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:138) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:582) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:213) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1097) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:448) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:175) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1031) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:136) at org.eclipse.jetty.server.Dispatcher.forward(Dispatcher.java:261) at org.eclipse.jetty.server.Dispatcher.forward(Dispatcher.java:101) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1486) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1486) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1486) at com.codahale.metrics.servlet.AbstractInstrumentedFilter.doFilter(AbstractInstrumentedFilter.java:97) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1486) at com.dtolabs.rundeck.server.filters.AuthFilter.doFilter(AuthFilter.java:74) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1486) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1486) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:519) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:138) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:529) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:213) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1097) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:448) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:175) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1031) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:136) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) at org.eclipse.jetty.server.Server.handle(Server.java:446) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:271) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:246) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.run(AbstractConnection.java:358) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:601) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:532) at java.lang.Thread.run(Thread.java:748)

akh13 commented Sep 26, 2017

Bump. Ran into this error too. Same software version, Amazon Linux 2017.03. The software ran fine for about 4-5 hours, then stopped processing jobs. Attempts to log in would result in the error. After restarting the process, Rundeck would again execute jobs and allow login. I ended up exporting all of my projects and importing them into a clean build. This seems to have solved the issue for myself.

ERROR GrailsExceptionResolver: RuntimeException occurred when processing request: [GET] / getAuthContextForSubject: Cannot get AuthContext without subject. Stacktrace follows: java.lang.RuntimeException: getAuthContextForSubject: Cannot get AuthContext without subject at rundeck.services.FrameworkService.getAuthContextForSubject(FrameworkService.groovy:603) at rundeck.services.FrameworkService.userAuthContext(FrameworkService.groovy:590) at rundeck.filters.ProjectSelectFilters$_closure1$_closure2$_closure3.doCall(ProjectSelectFilters.groovy:77) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:696) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1515) at grails.plugin.cache.web.filter.PageFragmentCachingFilter.doFilter(PageFragmentCachingFilter.java:198) at grails.plugin.cache.web.filter.AbstractFilter.doFilter(AbstractFilter.java:63) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1486) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1486) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1486) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1486) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:519) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:138) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:582) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:213) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1097) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:448) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:175) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1031) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:136) at org.eclipse.jetty.server.Dispatcher.forward(Dispatcher.java:261) at org.eclipse.jetty.server.Dispatcher.forward(Dispatcher.java:101) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1486) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1486) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1486) at com.codahale.metrics.servlet.AbstractInstrumentedFilter.doFilter(AbstractInstrumentedFilter.java:97) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1486) at com.dtolabs.rundeck.server.filters.AuthFilter.doFilter(AuthFilter.java:74) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1486) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1486) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:519) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:138) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:529) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:213) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1097) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:448) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:175) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1031) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:136) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) at org.eclipse.jetty.server.Server.handle(Server.java:446) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:271) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:246) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.run(AbstractConnection.java:358) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:601) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:532) at java.lang.Thread.run(Thread.java:748) ERROR GrailsExceptionResolver: RuntimeException occurred when processing request: [GET] / getAuthContextForSubject: Cannot get AuthContext without subject. Stacktrace follows: java.lang.RuntimeException: getAuthContextForSubject: Cannot get AuthContext without subject at rundeck.services.FrameworkService.getAuthContextForSubject(FrameworkService.groovy:603) at rundeck.services.FrameworkService.userAuthContext(FrameworkService.groovy:590) at rundeck.filters.ProjectSelectFilters$_closure1$_closure2$_closure3.doCall(ProjectSelectFilters.groovy:77) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:696) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1515) at grails.plugin.cache.web.filter.PageFragmentCachingFilter.doFilter(PageFragmentCachingFilter.java:198) at grails.plugin.cache.web.filter.AbstractFilter.doFilter(AbstractFilter.java:63) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1486) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1486) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1486) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1486) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:519) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:138) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:582) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:213) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1097) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:448) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:175) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1031) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:136) at org.eclipse.jetty.server.Dispatcher.forward(Dispatcher.java:261) at org.eclipse.jetty.server.Dispatcher.forward(Dispatcher.java:101) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1486) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1486) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1486) at com.codahale.metrics.servlet.AbstractInstrumentedFilter.doFilter(AbstractInstrumentedFilter.java:97) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1486) at com.dtolabs.rundeck.server.filters.AuthFilter.doFilter(AuthFilter.java:74) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1486) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1486) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:519) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:138) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:529) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:213) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1097) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:448) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:175) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1031) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:136) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) at org.eclipse.jetty.server.Server.handle(Server.java:446) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:271) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:246) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.run(AbstractConnection.java:358) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:601) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:532) at java.lang.Thread.run(Thread.java:748)

@dirkniblick

This comment has been minimized.

Show comment
Hide comment
@dirkniblick

dirkniblick Sep 28, 2017

The bug struck us again. No warning and no failed jobs this time. @gschueler, is there a work around or fix coming for this?

ERROR GrailsExceptionResolver: RuntimeException occurred when processing request: [GET] /
getAuthContextForSubject: Cannot get AuthContext without subject. Stacktrace follows:
java.lang.RuntimeException: getAuthContextForSubject: Cannot get AuthContext without subject
        at rundeck.services.FrameworkService.getAuthContextForSubject(FrameworkService.groovy:604)
        at rundeck.services.FrameworkService.userAuthContext(FrameworkService.groovy:591)
        at rundeck.filters.ProjectSelectFilters$_closure1$_closure2$_closure3.doCall(ProjectSelectFilters.groovy:77)
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:696)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1515)
        at grails.plugin.cache.web.filter.PageFragmentCachingFilter.doFilter(PageFragmentCachingFilter.java:198)
        at grails.plugin.cache.web.filter.AbstractFilter.doFilter(AbstractFilter.java:63)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1486)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1486)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1486)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1486)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:519)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:138)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:582)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:213)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1097)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:448)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:175)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1031)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:136)
        at org.eclipse.jetty.server.Dispatcher.forward(Dispatcher.java:261)
        at org.eclipse.jetty.server.Dispatcher.forward(Dispatcher.java:101)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1486)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1486)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1486)
        at com.codahale.metrics.servlet.AbstractInstrumentedFilter.doFilter(AbstractInstrumentedFilter.java:97)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1486)
        at com.dtolabs.rundeck.server.filters.AuthFilter.doFilter(AuthFilter.java:74)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1486)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1486)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:519)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:138)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:529)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:213)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1097)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:448)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:175)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1031)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:136)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
        at org.eclipse.jetty.server.Server.handle(Server.java:446)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:271)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:246)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.run(AbstractConnection.java:358)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:601)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:532)
        at java.lang.Thread.run(Thread.java:748)

dirkniblick commented Sep 28, 2017

The bug struck us again. No warning and no failed jobs this time. @gschueler, is there a work around or fix coming for this?

ERROR GrailsExceptionResolver: RuntimeException occurred when processing request: [GET] /
getAuthContextForSubject: Cannot get AuthContext without subject. Stacktrace follows:
java.lang.RuntimeException: getAuthContextForSubject: Cannot get AuthContext without subject
        at rundeck.services.FrameworkService.getAuthContextForSubject(FrameworkService.groovy:604)
        at rundeck.services.FrameworkService.userAuthContext(FrameworkService.groovy:591)
        at rundeck.filters.ProjectSelectFilters$_closure1$_closure2$_closure3.doCall(ProjectSelectFilters.groovy:77)
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:696)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1515)
        at grails.plugin.cache.web.filter.PageFragmentCachingFilter.doFilter(PageFragmentCachingFilter.java:198)
        at grails.plugin.cache.web.filter.AbstractFilter.doFilter(AbstractFilter.java:63)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1486)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1486)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1486)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1486)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:519)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:138)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:582)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:213)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1097)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:448)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:175)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1031)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:136)
        at org.eclipse.jetty.server.Dispatcher.forward(Dispatcher.java:261)
        at org.eclipse.jetty.server.Dispatcher.forward(Dispatcher.java:101)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1486)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1486)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1486)
        at com.codahale.metrics.servlet.AbstractInstrumentedFilter.doFilter(AbstractInstrumentedFilter.java:97)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1486)
        at com.dtolabs.rundeck.server.filters.AuthFilter.doFilter(AuthFilter.java:74)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1486)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1486)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:519)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:138)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:529)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:213)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1097)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:448)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:175)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1031)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:136)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
        at org.eclipse.jetty.server.Server.handle(Server.java:446)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:271)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:246)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.run(AbstractConnection.java:358)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:601)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:532)
        at java.lang.Thread.run(Thread.java:748)
@javaenthugm

This comment has been minimized.

Show comment
Hide comment
@javaenthugm

javaenthugm Oct 10, 2017

Is there any update on this issue? We are also facing this issue.

javaenthugm commented Oct 10, 2017

Is there any update on this issue? We are also facing this issue.

@drbeka

This comment has been minimized.

Show comment
Hide comment
@drbeka

drbeka Oct 10, 2017

Hi, we have same issue with rundeck 2.6.1, any update on that issue?

drbeka commented Oct 10, 2017

Hi, we have same issue with rundeck 2.6.1, any update on that issue?

@ahonor ahonor added the bug label Nov 17, 2017

@ibrentlam

This comment has been minimized.

Show comment
Hide comment
@ibrentlam

ibrentlam Nov 20, 2017

Still present on 2.10.0-1. Please bump priority. No, we're not bypassing the auth, standard rundeck login screen.

ibrentlam commented Nov 20, 2017

Still present on 2.10.0-1. Please bump priority. No, we're not bypassing the auth, standard rundeck login screen.

@gschueler gschueler added this to the 2.10.1 milestone Nov 21, 2017

@gschueler gschueler closed this in 78172c2 Nov 21, 2017

gschueler added a commit that referenced this issue Nov 21, 2017

@jtexier

This comment has been minimized.

Show comment
Hide comment
@jtexier

jtexier Mar 8, 2018

Hi, unfortunately, the bug is still present on 2.10.2-1.
Maybe, this issue should not have been closed so quickly

jtexier commented Mar 8, 2018

Hi, unfortunately, the bug is still present on 2.10.2-1.
Maybe, this issue should not have been closed so quickly

@Hackeruncle

This comment has been minimized.

Show comment
Hide comment
@Hackeruncle

Hackeruncle Mar 23, 2018

the bug is still present on 2.10.7-1.

Hackeruncle commented Mar 23, 2018

the bug is still present on 2.10.7-1.

@jipipayo

This comment has been minimized.

Show comment
Hide comment
@jipipayo

jipipayo May 18, 2018

Hited the bug on 2.10.7-1. version.
If we clean cookies the error message disappears ,but never ends trying log in.
In our case i think is related to high load of this rundeck instance.
i changed the memory settings on

/etc/rundeck/profile file

RDECK_JVM_SETTINGS="${RDECK_JVM_SETTINGS:- -Xmx3072m -Xms3072m -XX:MaxMetaspaceSize=1024m -server}"

and restarted the server.
Please , reopen the issue

jipipayo commented May 18, 2018

Hited the bug on 2.10.7-1. version.
If we clean cookies the error message disappears ,but never ends trying log in.
In our case i think is related to high load of this rundeck instance.
i changed the memory settings on

/etc/rundeck/profile file

RDECK_JVM_SETTINGS="${RDECK_JVM_SETTINGS:- -Xmx3072m -Xms3072m -XX:MaxMetaspaceSize=1024m -server}"

and restarted the server.
Please , reopen the issue

@muad-dweeb

This comment has been minimized.

Show comment
Hide comment
@muad-dweeb

muad-dweeb May 30, 2018

I'm hitting this exact error on Rundeck 2.10.5-1. Began happening suddenly last night. No configuration changes were made, no upgrades, etc. Because of this error our Production is down right now.

@jipipayo Did your memory setting change solve your issue?

muad-dweeb commented May 30, 2018

I'm hitting this exact error on Rundeck 2.10.5-1. Began happening suddenly last night. No configuration changes were made, no upgrades, etc. Because of this error our Production is down right now.

@jipipayo Did your memory setting change solve your issue?

@jipipayo

This comment has been minimized.

Show comment
Hide comment
@jipipayo

jipipayo May 31, 2018

@muad-dweeb yes, it worked great, its up and running without issues since then.

jipipayo commented May 31, 2018

@muad-dweeb yes, it worked great, its up and running without issues since then.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment