Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Parallel/Concurrent jobs fail #180

Closed
lcasey001 opened this issue Jun 28, 2012 · 22 comments
Closed

Parallel/Concurrent jobs fail #180

lcasey001 opened this issue Jun 28, 2012 · 22 comments
Assignees
Labels
Milestone

Comments

@lcasey001
Copy link

Issue Type: Bug
Rundeck version: rundeck-1.4.3-1.2
JDK Version: 1.6.0_33, sun
Server OS: Red Hat Enterprise Linux Server release 5.8 (Tikanga)
Install type: Yum-rpm
Issue:

When jenkins-rundeck plugin kicks off the parallel job it runs fine within rundeck(both jobs finish successful) however a 500 gets thrown back to jenkins on one of the jobs. This happens on HSQLDB, H2DB, and MYSQL.

Login failed on http://rundeck.notme.com : Invalid HTTP response 'HTTP/1.1 500 Internal Server Error' for http://rundeck.notme.com/run
Build step 'RunDeck' marked build as failure
Notifying upstream projects of job completion
Finished: FAILURE

2012-06-27 07:26:23.565::INFO: Login attempts: 1046, Hits: 1024, Ratio: 97.9%.
ERROR GroovyPageView: Error processing GroovyPageView: Error executing tag <g:form>: Error executing tag <g:render>: Error executing tag <g:render>: Error executing tag <g:select>: null at /WEB-INF/grails-app/views/framework/nodeFilterField.gsp:26 at /WEB-INF/grails-app/views/framework/nodeFilterInputs.gsp:70 at /WEB-INF/grails-app/views/framework/nodes.gsp:701 at /WEB-INF/grails-app/views/framework/nodes.gsp:710
org.codehaus.groovy.grails.web.taglib.exceptions.GrailsTagException: Error executing tag <g:form>: Error executing tag <g:render>: Error executing tag <g:render>: Error executing tag <g:select>: null at /WEB-INF/grails-app/views/framework/nodeFilterField.gsp:26 at /WEB-INF/grails-app/views/framework/nodeFilterInputs.gsp:70 at /WEB-INF/grails-app/
views/framework/nodes.gsp:701 at /WEB-INF/grails-app/views/framework/nodes.gsp:710

at gsp_rundeck_frameworknodes_gsp$_run_closure2.doCall(gsp_rundeck_frameworknodes_gsp.groovy:221)
at gsp_rundeck_frameworknodes_gsp$_run_closure2.doCall(gsp_rundeck_frameworknodes_gsp.groovy)
at gsp_rundeck_frameworknodes_gsp.run(gsp_rundeck_frameworknodes_gsp.groovy:278)
Caused by: org.codehaus.groovy.grails.web.taglib.exceptions.GrailsTagException: Error executing tag <g:render>: Error executing tag <g:render>: Error executing tag <g:select>:
null at /WEB-INF/grails-app/views/framework/nodeFilterField.gsp:26 at /WEB-INF/grails-app/views/framework/nodeFilterInputs.gsp:70 at /WEB-INF/grails-app/views/framework/nod es.gsp:701

at gsp_rundeck_frameworknodes_gsp$_run_closure2_closure6.doCall(gsp_rundeck_frameworknodes_gsp.groovy:214)
at gsp_rundeck_frameworknodes_gsp$_run_closure2_closure6.doCall(gsp_rundeck_frameworknodes_gsp.groovy)
... 3 more
Caused by: org.codehaus.groovy.grails.web.taglib.exceptions.GrailsTagException: Error executing tag <g:render>: Error executing tag <g:select>: null at /WEB-INF/grails-app/views/framework/nodeFilterField.gsp:26 at /WEB-INF/grails-app/views/framework/nodeFilterInputs.gsp:70

at gsp_rundeck_framework_nodeFilterInputs_gsp$_run_closure3.doCall(gsp_rundeck_framework_nodeFilterInputs_gsp.groovy:52)
at gsp_rundeck_framework_nodeFilterInputs_gsp.run(gsp_rundeck_framework_nodeFilterInputs_gsp.groovy:50)
... 5 more
Caused by: org.codehaus.groovy.grails.web.taglib.exceptions.GrailsTagException: Error executing tag <g:select>: null at /WEB-INF/grails-app/views/framework/_nodeFilterField.gsp:26

at gsp_rundeck_framework_nodeFilterField_gsp.run(gsp_rundeck_framework_nodeFilterField_gsp.groovy:46)
... 7 more
Caused by: java.lang.NullPointerException

... 8 more
ERROR GrailsExceptionResolver: Exception occurred when processing request: [POST] /run
Stacktrace follows:
org.codehaus.groovy.grails.web.pages.exceptions.GroovyPagesException: Error processing GroovyPageView: Error executing tag <g:form>: Error executing tag <g:render>: Error executing tag <g:render>: Error executing tag <g:select>: null at /WEB-INF/grails-app/views/framework/nodeFilterField.gsp:26 at /WEB-INF/grails-app/views/framework/nodeFilterInp
uts.gsp:70 at /WEB-INF/grails-app/views/framework/nodes.gsp:701 at /WEB-INF/grails-app/views/framework/nodes.gsp:710

at org.codehaus.groovy.grails.web.servlet.view.GroovyPageView.handleException(GroovyPageView.java:180)
at org.codehaus.groovy.grails.web.servlet.view.GroovyPageView.renderWithTemplateEngine(GroovyPageView.java:153)
at org.codehaus.groovy.grails.web.servlet.view.GroovyPageView.renderMergedOutputModel(GroovyPageView.java:81)
at org.springframework.web.servlet.view.AbstractView.render(AbstractView.java:250)
at org.springframework.web.servlet.DispatcherServlet.render(DispatcherServlet.java:1047)
at org.codehaus.groovy.grails.web.servlet.GrailsDispatcherServlet.doDispatch(GrailsDispatcherServlet.java:329)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:719)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:644)
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:560)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1166)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:70)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:70)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:70)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:388)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:418)
at org.mortbay.jetty.servlet.Dispatcher.forward(Dispatcher.java:327)
at org.mortbay.jetty.servlet.Dispatcher.forward(Dispatcher.java:126)
at org.codehaus.groovy.grails.web.util.WebUtils.forwardRequestForUrlMappingInfo(WebUtils.java:298)
at org.codehaus.groovy.grails.web.util.WebUtils.forwardRequestForUrlMappingInfo(WebUtils.java:264)
at org.codehaus.groovy.grails.web.util.WebUtils.forwardRequestForUrlMappingInfo(WebUtils.java:255)
at org.codehaus.groovy.grails.web.mapping.filter.UrlMappingsFilter.doFilterInternal(UrlMappingsFilter.java:183)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
at org.codehaus.groovy.grails.web.sitemesh.GrailsPageFilter.obtainContent(GrailsPageFilter.java:245)
at org.codehaus.groovy.grails.web.sitemesh.GrailsPageFilter.doFilter(GrailsPageFilter.java:134)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
at org.codehaus.groovy.grails.web.servlet.mvc.GrailsWebRequestFilter.doFilterInternal(GrailsWebRequestFilter.java:69)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
at org.codehaus.groovy.grails.web.filters.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:65)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:88)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:388)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:418)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:536)
at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:930)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:747)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:405)
at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
Caused by: org.codehaus.groovy.grails.web.taglib.exceptions.GrailsTagException: Error executing tag <g:form>: Error executing tag <g:render>: Error executing tag <g:render>: Error executing tag <g:select>: null at /WEB-INF/grails-app/views/framework/nodeFilterField.gsp:26 at /WEB-INF/grails-app/views/framework/nodeFilterInputs.gsp:70 at /WEB-INF/grails-app/views/framework/nodes.gsp:701 at /WEB-INF/grails-app/views/framework/nodes.gsp:710

at gsp_rundeck_frameworknodes_gsp$_run_closure2.doCall(gsp_rundeck_frameworknodes_gsp.groovy:221)
at gsp_rundeck_frameworknodes_gsp$_run_closure2.doCall(gsp_rundeck_frameworknodes_gsp.groovy)
at gsp_rundeck_frameworknodes_gsp.run(gsp_rundeck_frameworknodes_gsp.groovy:278)
Caused by: org.codehaus.groovy.grails.web.taglib.exceptions.GrailsTagException: Error executing tag <g:render>: Error executing tag <g:render>: Error executing tag <g:select>: null at /WEB-INF/grails-app/views/framework/nodeFilterField.gsp:26 at /WEB-INF/grails-app/views/framework/nodeFilterInputs.gsp:70 at /WEB-INF/grails-app/views/framework/nodes.gsp:701

at gsp_rundeck_frameworknodes_gsp$_run_closure2_closure6.doCall(gsp_rundeck_frameworknodes_gsp.groovy:214)
at gsp_rundeck_frameworknodes_gsp$_run_closure2_closure6.doCall(gsp_rundeck_frameworknodes_gsp.groovy)
... 3 more
Caused by: org.codehaus.groovy.grails.web.taglib.exceptions.GrailsTagException: Error executing tag <g:render>: Error executing tag <g:select>: null at /WEB-INF/grails-app/views/framework/nodeFilterField.gsp:26 at /WEB-INF/grails-app/views/framework/nodeFilterInputs.gsp:70

at gsp_rundeck_framework_nodeFilterInputs_gsp$_run_closure3.doCall(gsp_rundeck_framework_nodeFilterInputs_gsp.groovy:52)
at gsp_rundeck_framework_nodeFilterInputs_gsp.run(gsp_rundeck_framework_nodeFilterInputs_gsp.groovy:50)
... 5 more
Caused by: org.codehaus.groovy.grails.web.taglib.exceptions.GrailsTagException: Error executing tag <g:select>: null at /WEB-INF/grails-app/views/framework/_nodeFilterField.gsp:26

at gsp_rundeck_framework_nodeFilterField_gsp.run(gsp_rundeck_framework_nodeFilterField_gsp.groovy:46)
... 7 more
Caused by: java.lang.NullPointerException

... 8 more
INFO ScheduledExecutionService: scheduling immediate job run: TEMP:deploy:26:3646
INFO ExecutionService: Execution successful: 3646
INFO ExecutionService: updated scheduled Execution

With Mysql you also get the following:

INFO ExecutionService: updated scheduled Execution
ERROR PatchedDefaultFlushEventListener: Could not synchronize database state with session
org.hibernate.StaleObjectStateException: Row was updated or deleted by another transaction (or unsaved-value mapping was incorrect): [ScheduledExecution#9]
at org.hibernate.persister.entity.AbstractEntityPersister.check(AbstractEntityPersister.java:1792)
at org.hibernate.persister.entity.AbstractEntityPersister.update(AbstractEntityPersister.java:2435)
at org.hibernate.persister.entity.AbstractEntityPersister.updateOrInsert(AbstractEntityPersister.java:2335)
at org.hibernate.persister.entity.AbstractEntityPersister.update(AbstractEntityPersister.java:2635)
at org.hibernate.action.EntityUpdateAction.execute(EntityUpdateAction.java:115)
at org.hibernate.engine.ActionQueue.execute(ActionQueue.java:279)
at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:263)
at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:168)
at org.codehaus.groovy.grails.orm.hibernate.events.PatchedDefaultFlushEventListener.performExecutions(PatchedDefaultFlushEventListener.java:46)
at org.hibernate.event.def.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:50)
at org.hibernate.impl.SessionImpl.flush(SessionImpl.java:1027)
at sun.reflect.GeneratedMethodAccessor929.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.springframework.orm.hibernate3.HibernateTemplate$CloseSuppressingInvocationHandler.invoke(HibernateTemplate.java:1282)
at $Proxy10.flush(Unknown Source)
at org.springframework.orm.hibernate3.HibernateTemplate$28.doInHibernate(HibernateTemplate.java:883)
at org.springframework.orm.hibernate3.HibernateTemplate.doExecute(HibernateTemplate.java:406)
at org.springframework.orm.hibernate3.HibernateTemplate.executeWithNativeSession(HibernateTemplate.java:374)
at org.springframework.orm.hibernate3.HibernateTemplate.flush(HibernateTemplate.java:881)
at org.codehaus.groovy.grails.orm.hibernate.metaclass.SavePersistentMethod$1.doInHibernate(SavePersistentMethod.java:58)
at org.springframework.orm.hibernate3.HibernateTemplate.doExecute(HibernateTemplate.java:406)
at org.springframework.orm.hibernate3.HibernateTemplate.execute(HibernateTemplate.java:339)
at org.codehaus.groovy.grails.orm.hibernate.metaclass.SavePersistentMethod.performSave(SavePersistentMethod.java:53)
at org.codehaus.groovy.grails.orm.hibernate.metaclass.AbstractSavePersistentMethod.doInvokeInternal(AbstractSavePersistentMethod.java:179)
at org.codehaus.groovy.grails.orm.hibernate.metaclass.AbstractDynamicPersistentMethod.invoke(AbstractDynamicPersistentMethod.java:59)
at sun.reflect.GeneratedMethodAccessor187.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoCachedMethodSite.invoke(PojoMetaMethodSite.java:188)
at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:52)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:132)
at org.codehaus.groovy.grails.plugins.orm.hibernate.HibernatePluginSupport$_addBasicPersistenceMethods_closure71.doCall(HibernatePluginSupport.groovy:806)
at sun.reflect.GeneratedMethodAccessor883.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
at org.codehaus.groovy.runtime.metaclass.ClosureMetaMethod.invoke(ClosureMetaMethod.java:80)
at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoMetaMethodSiteNoUnwrapNoCoerce.invoke(PogoMetaMethodSite.java:307)
at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.call(PogoMetaMethodSite.java:63)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:124)
at ExecutionService$_updateScheduledExecState_closure31.doCall(ExecutionService.groovy:1459)
at sun.reflect.GeneratedMethodAccessor1205.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1058)
at groovy.lang.ExpandoMetaClass.invokeMethod(ExpandoMetaClass.java:1070)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:886)
at groovy.lang.Closure.call(Closure.java:282)
at groovy.lang.Closure.call(Closure.java:295)
at groovy.lang.Closure$call$0.call(Unknown Source)
at org.codehaus.groovy.grails.plugins.orm.hibernate.HibernatePluginSupport$_addTransactionalMethods_closure24_closure99.doCall(HibernatePluginSupport.groovy:526)
at sun.reflect.GeneratedMethodAccessor1057.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1058)
at groovy.lang.ExpandoMetaClass.invokeMethod(ExpandoMetaClass.java:1070)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:886)
at groovy.lang.Closure.call(Closure.java:282)
at org.codehaus.groovy.runtime.ConvertedClosure.invokeCustom(ConvertedClosure.java:51)
at org.codehaus.groovy.runtime.ConversionHandler.invoke(ConversionHandler.java:82)
at $Proxy11.doInHibernate(Unknown Source)
at org.springframework.orm.hibernate3.HibernateTemplate.doExecute(HibernateTemplate.java:406)
at org.springframework.orm.hibernate3.HibernateTemplate.execute(HibernateTemplate.java:339)
at sun.reflect.GeneratedMethodAccessor1056.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoCachedMethodSite.invoke(PojoMetaMethodSite.java:188)
at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:52)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:124)
at org.codehaus.groovy.grails.plugins.orm.hibernate.HibernatePluginSupport$_addTransactionalMethods_closure24.doCall(HibernatePluginSupport.groovy:517)
at sun.reflect.GeneratedMethodAccessor1053.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1058)
at groovy.lang.ExpandoMetaClass.invokeMethod(ExpandoMetaClass.java:1070)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:886)
at groovy.lang.Closure.call(Closure.java:282)
at org.codehaus.groovy.runtime.metaclass.ClosureStaticMetaMethod.invoke(ClosureStaticMetaMethod.java:59)
at org.codehaus.groovy.runtime.callsite.StaticMetaMethodSite$StaticMetaMethodSiteNoUnwrapNoCoerce.invoke(StaticMetaMethodSite.java:148)
at org.codehaus.groovy.runtime.callsite.StaticMetaMethodSite.call(StaticMetaMethodSite.java:88)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:124)
at ExecutionService.updateScheduledExecState(ExecutionService.groovy:1434)
at ExecutionService$updateScheduledExecState.callCurrent(Unknown Source)
at ExecutionService.saveExecutionState(ExecutionService.groovy:1383)
at ExecutionService$$FastClassByCGLIB$$38316c9d.invoke()
at net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:149)
at org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation.invokeJoinpoint(Cglib2AopProxy.java:688)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:110)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:621)
at ExecutionService$$EnhancerByCGLIB$$704ec811.saveExecutionState()
at ExecutionService$saveExecutionState.call(Unknown Source)
at ExecutionJob.saveState(ExecutionJob.groovy:231)
at ExecutionJob$saveState.callCurrent(Unknown Source)
at ExecutionJob.execute(ExecutionJob.groovy:54)
at org.quartz.core.JobRunShell.run(JobRunShell.java:199)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:546)

Reproduce: (Describe steps to reproduce bug here.)

@lcasey001
Copy link
Author

Using the latest dev version "Run Deck 1.4.4-dev" now allows for parallel jobs and also seems to have fixed the "hung" start job issue. The only thing I noticed was if the same job runs more than twice in parallel an error is thrown on occasion. The jobs still finish successfully. Using mysql backend. Here is a patebin http://pastebin.com/Nqi1PYet

@lcasey001
Copy link
Author

Attaching raw output incase something happens to pastebin.

ERROR PatchedDefaultFlushEventListener: Could not synchronize database state with session
org.hibernate.StaleObjectStateException: Row was updated or deleted by another transaction (or unsaved-value mapping was incorrect): [ScheduledExecution#36]
at org.hibernate.persister.entity.AbstractEntityPersister.check(AbstractEntityPersister.java:1792)
at org.hibernate.persister.entity.AbstractEntityPersister.update(AbstractEntityPersister.java:2435)
at org.hibernate.persister.entity.AbstractEntityPersister.updateOrInsert(AbstractEntityPersister.java:2335)
at org.hibernate.persister.entity.AbstractEntityPersister.update(AbstractEntityPersister.java:2635)
at org.hibernate.action.EntityUpdateAction.execute(EntityUpdateAction.java:115)
at org.hibernate.engine.ActionQueue.execute(ActionQueue.java:279)
at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:263)
at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:168)
at org.codehaus.groovy.grails.orm.hibernate.events.PatchedDefaultFlushEventListener.performExecutions(PatchedDefaultFlushEventListener.java:46)
at org.hibernate.event.def.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:50)
at org.hibernate.impl.SessionImpl.flush(SessionImpl.java:1027)
at sun.reflect.GeneratedMethodAccessor758.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.springframework.orm.hibernate3.HibernateTemplate$CloseSuppressingInvocationHandler.invoke(HibernateTemplate.java:1282)
at $Proxy10.flush(Unknown Source)
at org.springframework.orm.hibernate3.HibernateTemplate$28.doInHibernate(HibernateTemplate.java:883)
at org.springframework.orm.hibernate3.HibernateTemplate.doExecute(HibernateTemplate.java:406)
at org.springframework.orm.hibernate3.HibernateTemplate.executeWithNativeSession(HibernateTemplate.java:374)
at org.springframework.orm.hibernate3.HibernateTemplate.flush(HibernateTemplate.java:881)
at org.codehaus.groovy.grails.orm.hibernate.metaclass.SavePersistentMethod$1.doInHibernate(SavePersistentMethod.java:58)
at org.springframework.orm.hibernate3.HibernateTemplate.doExecute(HibernateTemplate.java:406)
at org.springframework.orm.hibernate3.HibernateTemplate.execute(HibernateTemplate.java:339)
at org.codehaus.groovy.grails.orm.hibernate.metaclass.SavePersistentMethod.performSave(SavePersistentMethod.java:53)
at org.codehaus.groovy.grails.orm.hibernate.metaclass.AbstractSavePersistentMethod.doInvokeInternal(AbstractSavePersistentMethod.java:179)
at org.codehaus.groovy.grails.orm.hibernate.metaclass.AbstractDynamicPersistentMethod.invoke(AbstractDynamicPersistentMethod.java:59)
at sun.reflect.GeneratedMethodAccessor432.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoCachedMethodSite.invoke(PojoMetaMethodSite.java:188)
at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:52)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:132)
at org.codehaus.groovy.grails.plugins.orm.hibernate.HibernatePluginSupport$_addBasicPersistenceMethods_closure71.doCall(HibernatePluginSupport.groovy:806)
at sun.reflect.GeneratedMethodAccessor724.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
at org.codehaus.groovy.runtime.metaclass.ClosureMetaMethod.invoke(ClosureMetaMethod.java:80)
at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoMetaMethodSiteNoUnwrapNoCoerce.invoke(PogoMetaMethodSite.java:307)
at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.call(PogoMetaMethodSite.java:63)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:124)
at ExecutionService$_updateScheduledExecState_closure31.doCall(ExecutionService.groovy:1460)
at sun.reflect.GeneratedMethodAccessor1149.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1058)
at groovy.lang.ExpandoMetaClass.invokeMethod(ExpandoMetaClass.java:1070)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:886)
at groovy.lang.Closure.call(Closure.java:282)
at groovy.lang.Closure.call(Closure.java:295)
at groovy.lang.Closure$call$0.call(Unknown Source)
at org.codehaus.groovy.grails.plugins.orm.hibernate.HibernatePluginSupport$_addTransactionalMethods_closure24_closure99.doCall(HibernatePluginSupport.groovy:526)
at sun.reflect.GeneratedMethodAccessor1043.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1058)
at groovy.lang.ExpandoMetaClass.invokeMethod(ExpandoMetaClass.java:1070)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:886)
at groovy.lang.Closure.call(Closure.java:282)
at org.codehaus.groovy.runtime.ConvertedClosure.invokeCustom(ConvertedClosure.java:51)
at org.codehaus.groovy.runtime.ConversionHandler.invoke(ConversionHandler.java:82)
at $Proxy11.doInHibernate(Unknown Source)
at org.springframework.orm.hibernate3.HibernateTemplate.doExecute(HibernateTemplate.java:406)
at org.springframework.orm.hibernate3.HibernateTemplate.execute(HibernateTemplate.java:339)
at sun.reflect.GeneratedMethodAccessor1042.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoCachedMethodSite.invoke(PojoMetaMethodSite.java:188)
at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:52)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:124)
at org.codehaus.groovy.grails.plugins.orm.hibernate.HibernatePluginSupport$_addTransactionalMethods_closure24.doCall(HibernatePluginSupport.groovy:517)
at sun.reflect.GeneratedMethodAccessor1039.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1058)
at groovy.lang.ExpandoMetaClass.invokeMethod(ExpandoMetaClass.java:1070)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:886)
at groovy.lang.Closure.call(Closure.java:282)
at org.codehaus.groovy.runtime.metaclass.ClosureStaticMetaMethod.invoke(ClosureStaticMetaMethod.java:59)
at org.codehaus.groovy.runtime.callsite.StaticMetaMethodSite$StaticMetaMethodSiteNoUnwrapNoCoerce.invoke(StaticMetaMethodSite.java:148)
at org.codehaus.groovy.runtime.callsite.StaticMetaMethodSite.call(StaticMetaMethodSite.java:88)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:124)
at ExecutionService.updateScheduledExecState(ExecutionService.groovy:1435)
at ExecutionService$updateScheduledExecState.callCurrent(Unknown Source)
at ExecutionService.saveExecutionState(ExecutionService.groovy:1384)
at ExecutionService$$FastClassByCGLIB$$38316c9d.invoke()
at net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:149)
at org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation.invokeJoinpoint(Cglib2AopProxy.java:688)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:110)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:621)
at ExecutionService$$EnhancerByCGLIB$$2db5091e.saveExecutionState()
at ExecutionService$saveExecutionState.call(Unknown Source)
at ExecutionJob.saveState(ExecutionJob.groovy:245)
at ExecutionJob$saveState.callCurrent(Unknown Source)
at ExecutionJob.execute(ExecutionJob.groovy:54)
at org.quartz.core.JobRunShell.run(JobRunShell.java:199)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:546)

@lcasey001
Copy link
Author

So I think this may actually be related to dynamic node filtering. Jenkins is passing that info onto rundeck which is where we get the error. If we have static nodes it works fine.

@lcasey001
Copy link
Author

This has been fixed by using the jenkins-rundeck plugin node filter option vs dynamic node filter via options.

@lcasey001 lcasey001 reopened this Aug 21, 2012
@lcasey001
Copy link
Author

nvm same thing happened. Just doesn't happen as often.

@lcasey001
Copy link
Author

The NPE seems to go away for a while after a restart. It also occurred while no users were logged into the UI. Wonder if it could be related to the rundeck-jenkins plugin regarding authentication. The authenticated rundeck-jenkins user is unable to retrieve a specific view/template and thus causing the job to fail.

Also of note I have seen the NPE both before and after jobs. Which again leads me to think its an issue with logged in or cached user session.

@chalettu
Copy link

chalettu commented Oct 4, 2012

I am also having this exact same issue. I am interested in hearing any and all ideas people have for solutions

@lcasey001
Copy link
Author

So we looked at this further. We have temporarily resolved this issue by editing the rundeck-jenkins plugin.
https://github.com/vbehar/jenkins-rundeck-plugin/blob/master/src/main/java/org/jenkinsci/plugins/rundeck/RundeckNotifier.java

Thread.sleep(5000) to Thread.sleep(2000).

We have not gotten any NPE in the last 1500 job executions. We normally start getting it after couple hundred or so.

@chalettu
Copy link

chalettu commented Oct 4, 2012

hmm. Do you have any directions on how to go about editing this and recompiling the plugin?

@lcasey001
Copy link
Author

Grab the plugin via github, make your change, build it via maven, then follow the instructions here https://wiki.jenkins-ci.org/display/JENKINS/Plugin+tutorial for deploying the custom plugin.

Again this has temporarily resolved our issue for our environment. Your's might be different. Why this resolved our issue I can not answer.

@lcasey001
Copy link
Author

So this did not resolve our issue completely. We got the NPE again. We are adding tons of retries on the plugin side to account for this NPE. Huge band aid for whatever is causing the NPE. The rundeck logs will get spammed with the NPE but the jenkins-rundeck plugin will keep retrying for X times to get a response back from rundeck before failing. We put that in because the job within rundeck still runs.

@gschueler
Copy link
Member

I will hopefully get a chance to look into this issue soon

but just to verify, you are using the latest dev branch code?

@ghost ghost assigned gschueler Oct 5, 2012
@lcasey001
Copy link
Author

That would be great. We are currently running rundeck-1.4.4-0.94.dev. The current "work around" within the plugin to loop through that exception has worked. Normally once it would hit the 500 exception jenkins would mark the job as failed but rundeck would continue on and run the job. Here is an example of the output within Jenkins with some debugs to it. http://pastie.org/4916523
Let me know if you want additional debug logs from the rundeck side.

@chalettu
Copy link

chalettu commented Oct 8, 2012

So I have a little bit of info that might be helpful in troubleshooting this bug. I am still new to rundeck but I have set up two jobs that I run from Jenkins. One job runs for 5 minutes and the other averages about 15 minutes. Both jobs essentially are the same but operate against different servers. The 15 minute job is the one that seems to consistently have this issue of failing. The shorter running job hasn't failed even once.

@gschueler
Copy link
Member

ok, after some hunting, I think it is related to this Grails bug: http://jira.grails.org/browse/GRAILS-7656

I'm not sure why it was so intermittent, but the NPE seems to be caused simply by a missing attribute in one of the gsp tags. This was causing the jenkins-rundeck plugin to fail on some of the periodic polling or login checks.

Adding the missing attribute to the tag should fix it. Doing some more testing, but I think this will fix the issue once and for all.

Note: the "StaleObjectStateException" and "HibernateOptimisticLockingFailureException" in the log is a side-effect of the locking model, and the Job executions will still succeed because the current code expects it and retries when it encounters that exception. you can effectively ignore this message although it is ugly.

@lcasey001
Copy link
Author

Thanks !!! Just upgraded and will keep a look out for the NPE. Thanks again !!!!

@connaryscott
Copy link

just happened to me as well. I'm on a recent version using jenkins/rundeck plugin:

[cscott2@centgoldrk1v trunk]$ rpm -q rundeck
rundeck-1.5.1-0.221.SNAPSHOT.noarch

which fails the jenkins build

Notification succeeded ! Execution #58, at {}/execution/follow/58 (status : RUNNING)
Waiting for RunDeck execution to finish...
Login failed on http://localhost:4440/rundeck : Invalid HTTP response 'HTTP/1.1 500 Internal Server Error' for http://localhost:4440/rundeck/;jsessionid=415pg2i6ke6r
Build step 'RunDeck' marked build as failure
Finished: FAILURE

@gschueler
Copy link
Member

hi chuck, do you have any log info from the server?

@connaryscott
Copy link

Hey Greg,

Here's what I got:

I'm wondering if there was another rundeck process running. I recall that
the service startup script (sudo service rundeckd start) does not check for
existing process running and wondering if I got nailed by that.

I am rerunning a long running job and will provide more detail shortly.

2013-04-23 16:18:27,632 [main] INFO grails.spring.BeanBuilder -
[RuntimeConfiguration] Configuring data source for environment: PRODUCTION
2013-04-23 16:18:33,033 [1211850091@qtp-405223709-21] ERROR
org.hibernate.transaction.JDBCTransaction - JDBC begin failed
org.h2.jdbc.JdbcSQLException: Database may be already in use: Locked by
another process. Possible solutions: close all other connection(s); use the
server mode [90020-112]
at org.h2.message.Message.getSQLException(Message.java:107)
at org.h2.message.Message.getSQLException(Message.java:118)
at org.h2.message.Message.getSQLException(Message.java:77)
at org.h2.store.FileLock.getExceptionAlreadyInUse(FileLock.java:475)
at org.h2.store.FileLock.lockFile(FileLock.java:338)
at org.h2.store.FileLock.lock(FileLock.java:130)
at org.h2.engine.Database.open(Database.java:557)
at org.h2.engine.Database.openDatabase(Database.java:221)
at org.h2.engine.Database.(Database.java:216)
at org.h2.engine.Engine.openSession(Engine.java:57)
at org.h2.engine.Engine.openSession(Engine.java:139)
at org.h2.engine.Engine.getSession(Engine.java:119)
at
org.h2.engine.SessionFactoryEmbedded.createSession(SessionFactoryEmbedded.ja
va:17)
at
org.h2.engine.SessionRemote.connectEmbeddedOrServer(SessionRemote.java:248)
at org.h2.engine.SessionRemote.createSession(SessionRemote.java:226)
at org.h2.jdbc.JdbcConnection.(JdbcConnection.java:106)
at org.h2.jdbc.JdbcConnection.(JdbcConnection.java:90)
at org.h2.Driver.connect(Driver.java:58)
at java.sql.DriverManager.getConnection(DriverManager.java:620)
at java.sql.DriverManager.getConnection(DriverManager.java:169)
at
org.springframework.jdbc.datasource.DriverManagerDataSource.getConnectionFro
mDriverManager(DriverManagerDataSource.java:173)
at
org.springframework.jdbc.datasource.DriverManagerDataSource.getConnectionFro
mDriver(DriverManagerDataSource.java:164)
at
org.springframework.jdbc.datasource.AbstractDriverBasedDataSource.getConnect
ionFromDriver(AbstractDriverBasedDataSource.java:149)
at
org.springframework.jdbc.datasource.AbstractDriverBasedDataSource.getConnect
ion(AbstractDriverBasedDataSource.java:119)
at
org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSour
ceUtils.java:111)
at
org.springframework.jdbc.datasource.TransactionAwareDataSourceProxy$Transact
ionAwareInvocationHandler.invoke(TransactionAwareDataSourceProxy.java:224)
at sun.proxy.$Proxy7.getAutoCommit(Unknown Source)
at
org.hibernate.transaction.JDBCTransaction.begin(JDBCTransaction.java:85)
at
org.hibernate.impl.SessionImpl.beginTransaction(SessionImpl.java:1353)
at
org.springframework.orm.hibernate3.HibernateTransactionManager.doBegin(Hiber
nateTransactionManager.java:555)
at
org.codehaus.groovy.grails.orm.hibernate.GrailsHibernateTransactionManager.s
uper$3$doBegin(GrailsHibernateTransactionManager.groovy)
at sun.reflect.GeneratedMethodAccessor494.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at
org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1058)
at
groovy.lang.ExpandoMetaClass.invokeMethod(ExpandoMetaClass.java:1070)
at
org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuperN(Scrip
tBytecodeAdapter.java:127)
at
org.codehaus.groovy.grails.orm.hibernate.GrailsHibernateTransactionManager.d
oBegin(GrailsHibernateTransactionManager.groovy:32)
at
org.springframework.transaction.support.AbstractPlatformTransactionManager.g
etTransaction(AbstractPlatformTransactionManager.java:371)
at
org.springframework.transaction.interceptor.TransactionAspectSupport.createT
ransactionIfNecessary(TransactionAspectSupport.java:335)
at
org.springframework.transaction.interceptor.TransactionInterceptor.invoke(Tr
ansactionInterceptor.java:105)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(Reflect
iveMethodInvocation.java:172)
at
org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.i
ntercept(Cglib2AopProxy.java:621)
at
rundeck.services.UserService$$EnhancerByCGLIB$$289a4772.getFilterPref()
at rundeck.services.UserService$getFilterPref.call(Unknown Source)
at
rundeck.filters.ProjectSelectFilters$_closure1_closure3_closure5.doCall(Proj
ectSelectFilters.groovy:97)
at sun.reflect.GeneratedMethodAccessor472.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at
org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSite
.invoke(PogoMetaMethodSite.java:225)
at
org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.callCurrent(PogoMeta
MethodSite.java:51)
at
org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCa
llSite.java:149)
at
rundeck.filters.ProjectSelectFilters$_closure1_closure3_closure5.doCall(Proj
ectSelectFilters.groovy)
at sun.reflect.GeneratedMethodAccessor471.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at
org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1058)
at
groovy.lang.ExpandoMetaClass.invokeMethod(ExpandoMetaClass.java:1070)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:886)
at groovy.lang.Closure.call(Closure.java:282)
at groovy.lang.Closure.call(Closure.java:277)
at groovy.lang.Closure$call.call(Unknown Source)
at
org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray
.java:40)
at groovy.lang.Closure$call.call(Unknown Source)
at
org.codehaus.groovy.grails.plugins.web.filters.FilterToHandlerAdapter.preHan
dle(FilterToHandlerAdapter.groovy:110)
at
org.springframework.web.servlet.HandlerInterceptor$preHandle.call(Unknown
Source)
at
org.codehaus.groovy.grails.plugins.web.filters.CompositeInterceptor.preHandl
e(CompositeInterceptor.groovy:42)
at
org.codehaus.groovy.grails.web.servlet.GrailsDispatcherServlet.doDispatch(Gr
ailsDispatcherServlet.java:282)
at
org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServle
t.java:719)
at
org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkSer
vlet.java:644)
at
org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.jav
a:560)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at
org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler
.java:1166)
at
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestF
ilter.java:70)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler
.java:1157)
at
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestF
ilter.java:70)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler
.java:1157)
at
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestF
ilter.java:70)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler
.java:1157)
at
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:388)
at
org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)
at
org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:418)
at org.mortbay.jetty.servlet.Dispatcher.forward(Dispatcher.java:327)
at org.mortbay.jetty.servlet.Dispatcher.forward(Dispatcher.java:126)
at
org.codehaus.groovy.grails.web.util.WebUtils.forwardRequestForUrlMappingInfo
(WebUtils.java:298)
at
org.codehaus.groovy.grails.web.util.WebUtils.forwardRequestForUrlMappingInfo
(WebUtils.java:264)
at
org.codehaus.groovy.grails.web.util.WebUtils.forwardRequestForUrlMappingInfo
(WebUtils.java:255)
at
org.codehaus.groovy.grails.web.mapping.filter.UrlMappingsFilter.doFilterInte
rnal(UrlMappingsFilter.java:183)
at
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestF
ilter.java:76)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler
.java:1157)
at
org.codehaus.groovy.grails.web.sitemesh.GrailsPageFilter.obtainContent(Grail
sPageFilter.java:245)
at
org.codehaus.groovy.grails.web.sitemesh.GrailsPageFilter.doFilter(GrailsPage
Filter.java:134)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler
.java:1157)
at
org.codehaus.groovy.grails.web.servlet.mvc.GrailsWebRequestFilter.doFilterIn
ternal(GrailsWebRequestFilter.java:69)
at
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestF
ilter.java:76)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler
.java:1157)
at
org.codehaus.groovy.grails.web.filters.HiddenHttpMethodFilter.doFilterIntern
al(HiddenHttpMethodFilter.java:65)
at
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestF
ilter.java:76)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler
.java:1157)
at
org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(Char
acterEncodingFilter.java:88)
at
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestF
ilter.java:76)
at
org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(Delegati
ngFilterProxy.java:237)
at
org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilt
erProxy.java:167)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler
.java:1157)
at
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:388)
at
org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)
at
org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:418)
at
org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at
org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:536)
at
org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:
930)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:747)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:405)
at
org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:22
8)
at
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582
)
2013-04-23 16:18:33,089 [1211850091@qtp-405223709-21] ERROR
org.codehaus.groovy.grails.web.errors.GrailsExceptionResolver - Exception
occurred when processing request: [POST] /rundeck/;jsessionid=415pg2i6ke6r
Stacktrace follows:
org.springframework.transaction.CannotCreateTransactionException: Could not
open Hibernate Session for transaction; nested exception is
org.hibernate.TransactionException: JDBC begin failed:
at
rundeck.services.UserService$$EnhancerByCGLIB$$289a4772.getFilterPref()
at rundeck.services.UserService$getFilterPref.call(Unknown Source)
at
rundeck.filters.ProjectSelectFilters$_closure1_closure3_closure5.doCall(Proj
ectSelectFilters.groovy:97)
at
rundeck.filters.ProjectSelectFilters$_closure1_closure3_closure5.doCall(Proj
ectSelectFilters.groovy)
Caused by: org.hibernate.TransactionException: JDBC begin failed:
... 4 more
Caused by: org.h2.jdbc.JdbcSQLException: Database may be already in use:
Locked by another process. Possible solutions: close all other
connection(s); use the server mode [90020-112]
at org.h2.message.Message.getSQLException(Message.java:107)
at org.h2.message.Message.getSQLException(Message.java:118)
at org.h2.message.Message.getSQLException(Message.java:77)
at org.h2.store.FileLock.getExceptionAlreadyInUse(FileLock.java:475)
at org.h2.store.FileLock.lockFile(FileLock.java:338)
at org.h2.store.FileLock.lock(FileLock.java:130)
at org.h2.engine.Database.open(Database.java:557)
at org.h2.engine.Database.openDatabase(Database.java:221)
at org.h2.engine.Database.(Database.java:216)
at org.h2.engine.Engine.openSession(Engine.java:57)
at org.h2.engine.Engine.openSession(Engine.java:139)
at org.h2.engine.Engine.getSession(Engine.java:119)
at
org.h2.engine.SessionFactoryEmbedded.createSession(SessionFactoryEmbedded.ja
va:17)
at
org.h2.engine.SessionRemote.connectEmbeddedOrServer(SessionRemote.java:248)
at org.h2.engine.SessionRemote.createSession(SessionRemote.java:226)
at org.h2.jdbc.JdbcConnection.(JdbcConnection.java:106)
at org.h2.jdbc.JdbcConnection.(JdbcConnection.java:90)
at org.h2.Driver.connect(Driver.java:58)
at java.sql.DriverManager.getConnection(DriverManager.java:620)
at java.sql.DriverManager.getConnection(DriverManager.java:169)
... 4 more
2013-04-23 16:18:35,081 [97612953@qtp-405223709-18] ERROR
org.hibernate.util.JDBCExceptionReporter - Error opening database: Lock file
recently modified [8000-112]
2013-04-23 16:18:37,393 [97612953@qtp-405223709-18] ERROR
org.codehaus.groovy.grails.web.errors.GrailsExceptionResolver - Exception
occurred when processing request: [POST]
/rundeck/execution/tailExecutionOutput.json
Stacktrace follows:
org.h2.jdbc.JdbcSQLException: Error opening database: Lock file recently
modified [8000-112]
at org.h2.message.Message.getSQLException(Message.java:107)
at org.h2.store.FileLock.getExceptionFatal(FileLock.java:471)
at org.h2.store.FileLock.waitUntilOld(FileLock.java:295)
at org.h2.store.FileLock.lockFile(FileLock.java:330)
at org.h2.store.FileLock.lock(FileLock.java:130)
at org.h2.engine.Database.open(Database.java:557)
at org.h2.engine.Database.openDatabase(Database.java:221)
at org.h2.engine.Database.(Database.java:216)
at org.h2.engine.Engine.openSession(Engine.java:57)
at org.h2.engine.Engine.openSession(Engine.java:139)
at org.h2.engine.Engine.getSession(Engine.java:119)
at
org.h2.engine.SessionFactoryEmbedded.createSession(SessionFactoryEmbedded.ja
va:17)
at
org.h2.engine.SessionRemote.connectEmbeddedOrServer(SessionRemote.java:248)
at org.h2.engine.SessionRemote.createSession(SessionRemote.java:226)
at org.h2.jdbc.JdbcConnection.(JdbcConnection.java:106)
at org.h2.jdbc.JdbcConnection.(JdbcConnection.java:90)
at org.h2.Driver.connect(Driver.java:58)
at java.sql.DriverManager.getConnection(DriverManager.java:620)
at java.sql.DriverManager.getConnection(DriverManager.java:169)
at
rundeck.controllers.ExecutionController$_closure10.doCall(ExecutionControlle
r.groovy:218)
at
rundeck.controllers.ExecutionController$_closure10.doCall(ExecutionControlle
r.groovy)
2013-04-23 16:18:37,909 [main] WARN grails.app.bootstrap.BootStrap - using
rdeck.base system property: /etc/rundeck
2013-04-23 16:18:37,911 [main] INFO grails.app.bootstrap.BootStrap -
config: rdeck.base = /etc/rundeck
2013-04-23 16:18:37,922 [main] INFO grails.app.bootstrap.BootStrap - loaded
configuration: /etc/rundeck/framework.properties
2013-04-23 16:18:37,922 [main] INFO grails.app.bootstrap.BootStrap - RSS
feeds enabled
2013-04-23 16:18:41,018 [main] INFO
grails.app.service.rundeck.services.ExecutionService - updated scheduled
Execution
2013-04-23 16:18:41,040 [main] WARN
grails.app.service.rundeck.services.FrameworkService - rdeck.base is:
/etc/rundeck
2013-04-23 16:18:41,381 [main] ERROR
grails.app.service.rundeck.services.ExecutionService - Stale Execution
cleaned up: [58]
2013-04-23 16:18:44,369 [quartzScheduler_Worker-2] INFO
grails.app.service.rundeck.services.ExecutionService - Execution successful:
58
2013-04-23 16:18:44,427 [quartzScheduler_Worker-2] INFO
grails.app.service.rundeck.services.ExecutionService - updated scheduled
Execution
2013-04-23 16:33:48,268 [1331817367@qtp-405223709-29] INFO
grails.app.service.rundeck.services.ScheduledExecutionService - scheduling
temp job: TEMP:cscott2:59
2013-04-23 16:33:49,411 [quartzScheduler_Worker-3] INFO
grails.app.service.rundeck.services.ExecutionService - Execution successful:
59

From: Greg Schueler notifications@github.com
Reply-To: dtolabs/rundeck
<reply+i-5309978-47074443a3d953fd2a67f085127b5d497215ea7f-571449@reply.githu
b.com>
Date: Tuesday, April 23, 2013 4:32 PM
To: dtolabs/rundeck rundeck@noreply.github.com
Cc: Charles Scott connaryscott@gmail.com
Subject: Re: [rundeck] Parallel/Concurrent jobs fail (#180)

hi chuck, do you have any log info from the server?


Reply to this email directly or view it on GitHub
#180 (comment) .

@gschueler
Copy link
Member

yes, that does look like a second process was already locking the H2 database

@connaryscott
Copy link

Just reran the job and rundeck responded with same issueŠ..

Not sure what's causing it . There is only one rundeck process running
(parent and child):

[root@centgoldrk1v trunk]# ps -ef |grep rundeck
root 21578 13166 0 16:47 pts/0 00:00:00 grep rundeck
root 32106 1 0 16:01 pts/0 00:00:00 runuser -l rundeck -c
/usr/bin/java
-Djava.security.auth.login.config=/etc/rundeck/jaas-activedirectory.conf
-Dserver.web.context=/rundeck ?-Dloginmodule.name=activedirectory
?-Drdeck.config=/etc/rundeck ?-Drdeck.base=/etc/rundeck
?-Drundeck.server.configDir=/etc/rundeck
?-Dserver.datastore.path=/var/lib/rundeck/data
?-Drundeck.server.serverDir=/var/lib/rundeck
?-Drdeck.projects=/var/rundeck/projects
?-Drdeck.runlogs=/var/lib/rundeck/logs
?-Drundeck.config.name=/etc/rundeck/rundeck-config.properties
?-Djava.io.tmpdir=/tmp/rundeck -Xmx1024m -Xms256m -cp
/var/lib/rundeck/bootstrap/jetty-util-6.1.21.jar:/var/lib/rundeck/bootstrap/
servlet-api-2.5-20081211.jar:/var/lib/rundeck/bootstrap/rundeck-jetty-server
-1.5.1-SNAPSHOT.jar:/var/lib/rundeck/bootstrap/jetty-plus-6.1.21.jar:/var/li
b/rundeck/bootstrap/jetty-6.1.21.jar:/var/lib/rundeck/bootstrap/jetty-naming
-6.1.21.jar::/etc/rundeck com.dtolabs.rundeck.RunServer /etc/rundeck 4440
rundeck 32112 32106 2 16:01 ? 00:01:22 /usr/bin/java
-Djava.security.auth.login.config=/etc/rundeck/jaas-activedirectory.conf
-Dserver.web.context=/rundeck -Dloginmodule.name=activedirectory
-Drdeck.config=/etc/rundeck -Drdeck.base=/etc/rundeck
-Drundeck.server.configDir=/etc/rundeck
-Dserver.datastore.path=/var/lib/rundeck/data
-Drundeck.server.serverDir=/var/lib/rundeck
-Drdeck.projects=/var/rundeck/projects -Drdeck.runlogs=/var/lib/rundeck/logs
-Drundeck.config.name=/etc/rundeck/rundeck-config.properties
-Djava.io.tmpdir=/tmp/rundeck -Xmx1024m -Xms256m -cp
/var/lib/rundeck/bootstrap/jetty-util-6.1.21.jar:/var/lib/rundeck/bootstrap/
servlet-api-2.5-20081211.jar:/var/lib/rundeck/bootstrap/rundeck-jetty-server
-1.5.1-SNAPSHOT.jar:/var/lib/rundeck/bootstrap/jetty-plus-6.1.21.jar:/var/li
b/rundeck/bootstrap/jetty-6.1.21.jar:/var/lib/rundeck/bootstrap/jetty-naming
-6.1.21.jar::/etc/rundeck com.dtolabs.rundeck.RunServer /etc/rundeck 4440

2013-04-23 16:44:49,093 [1929121341@qtp-405223709-33] INFO
grails.app.service.rundeck.services.ScheduledExecutionService - scheduling
immediate job run: TEMP:ciuser:69:60
2013-04-23 16:45:14,808 [main] INFO grails.spring.BeanBuilder -
[RuntimeConfiguration] Configuring data source for environment: PRODUCTION
2013-04-23 16:45:17,927 [116314147@qtp-405223709-32] ERROR
org.hibernate.transaction.JDBCTransaction - JDBC begin failed
org.h2.jdbc.JdbcSQLException: Database may be already in use: Locked by
another process. Possible solutions: close all other connection(s); use the
server mode [90020-112]
at org.h2.message.Message.getSQLException(Message.java:107)
at org.h2.message.Message.getSQLException(Message.java:118)
at org.h2.message.Message.getSQLException(Message.java:77)
at org.h2.store.FileLock.getExceptionAlreadyInUse(FileLock.java:475)
at org.h2.store.FileLock.lockFile(FileLock.java:338)
at org.h2.store.FileLock.lock(FileLock.java:130)
at org.h2.engine.Database.open(Database.java:557)
at org.h2.engine.Database.openDatabase(Database.java:221)
at org.h2.engine.Database.(Database.java:216)
at org.h2.engine.Engine.openSession(Engine.java:57)
at org.h2.engine.Engine.openSession(Engine.java:139)
at org.h2.engine.Engine.getSession(Engine.java:119)
at
org.h2.engine.SessionFactoryEmbedded.createSession(SessionFactoryEmbedded.ja
va:17)
at
org.h2.engine.SessionRemote.connectEmbeddedOrServer(SessionRemote.java:248)
at org.h2.engine.SessionRemote.createSession(SessionRemote.java:226)
at org.h2.jdbc.JdbcConnection.(JdbcConnection.java:106)
at org.h2.jdbc.JdbcConnection.(JdbcConnection.java:90)
at org.h2.Driver.connect(Driver.java:58)
at java.sql.DriverManager.getConnection(DriverManager.java:620)
at java.sql.DriverManager.getConnection(DriverManager.java:169)
at
org.springframework.jdbc.datasource.DriverManagerDataSource.getConnectionFro
mDriverManager(DriverManagerDataSource.java:173)
at
org.springframework.jdbc.datasource.DriverManagerDataSource.getConnectionFro
mDriver(DriverManagerDataSource.java:164)
at
org.springframework.jdbc.datasource.AbstractDriverBasedDataSource.getConnect
ionFromDriver(AbstractDriverBasedDataSource.java:149)
at
org.springframework.jdbc.datasource.AbstractDriverBasedDataSource.getConnect
ion(AbstractDriverBasedDataSource.java:119)
at
org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSour
ceUtils.java:111)
at
org.springframework.jdbc.datasource.TransactionAwareDataSourceProxy$Transact
ionAwareInvocationHandler.invoke(TransactionAwareDataSourceProxy.java:224)
at sun.proxy.$Proxy7.getAutoCommit(Unknown Source)
at
org.hibernate.transaction.JDBCTransaction.begin(JDBCTransaction.java:85)
at
org.hibernate.impl.SessionImpl.beginTransaction(SessionImpl.java:1353)
at
org.springframework.orm.hibernate3.HibernateTransactionManager.doBegin(Hiber
nateTransactionManager.java:555)
at
org.codehaus.groovy.grails.orm.hibernate.GrailsHibernateTransactionManager.s
uper$3$doBegin(GrailsHibernateTransactionManager.groovy)
at sun.reflect.GeneratedMethodAccessor494.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at
org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1058)
at
groovy.lang.ExpandoMetaClass.invokeMethod(ExpandoMetaClass.java:1070)
at
org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuperN(Scrip
tBytecodeAdapter.java:127)
at
org.codehaus.groovy.grails.orm.hibernate.GrailsHibernateTransactionManager.d
oBegin(GrailsHibernateTransactionManager.groovy:32)
at
org.springframework.transaction.support.AbstractPlatformTransactionManager.g
etTransaction(AbstractPlatformTransactionManager.java:371)
at
org.springframework.transaction.interceptor.TransactionAspectSupport.createT
ransactionIfNecessary(TransactionAspectSupport.java:335)
at
org.springframework.transaction.interceptor.TransactionInterceptor.invoke(Tr
ansactionInterceptor.java:105)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(Reflect
iveMethodInvocation.java:172)
at
org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.i
ntercept(Cglib2AopProxy.java:621)
at
rundeck.services.UserService$$EnhancerByCGLIB$$289a4772.getFilterPref()
at rundeck.services.UserService$getFilterPref.call(Unknown Source)
at
rundeck.filters.ProjectSelectFilters$_closure1_closure3_closure5.doCall(Proj
ectSelectFilters.groovy:97)
at sun.reflect.GeneratedMethodAccessor472.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at
org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSite
.invoke(PogoMetaMethodSite.java:225)
at
org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.callCurrent(PogoMeta
MethodSite.java:51)
at
org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCa
llSite.java:149)
at
rundeck.filters.ProjectSelectFilters$_closure1_closure3_closure5.doCall(Proj
ectSelectFilters.groovy)
at sun.reflect.GeneratedMethodAccessor471.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at
org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1058)
at
groovy.lang.ExpandoMetaClass.invokeMethod(ExpandoMetaClass.java:1070)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:886)
at groovy.lang.Closure.call(Closure.java:282)
at groovy.lang.Closure.call(Closure.java:277)
at groovy.lang.Closure$call.call(Unknown Source)
at
org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray
.java:40)
at groovy.lang.Closure$call.call(Unknown Source)
at
org.codehaus.groovy.grails.plugins.web.filters.FilterToHandlerAdapter.preHan
dle(FilterToHandlerAdapter.groovy:110)
at
org.springframework.web.servlet.HandlerInterceptor$preHandle.call(Unknown
Source)
at
org.codehaus.groovy.grails.plugins.web.filters.CompositeInterceptor.preHandl
e(CompositeInterceptor.groovy:42)
at
org.codehaus.groovy.grails.web.servlet.GrailsDispatcherServlet.doDispatch(Gr
ailsDispatcherServlet.java:282)
at
org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServle
t.java:719)
at
org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkSer
vlet.java:644)
at
org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.jav
a:560)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at
org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler
.java:1166)
at
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestF
ilter.java:70)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler
.java:1157)
at
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestF
ilter.java:70)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler
.java:1157)
at
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestF
ilter.java:70)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler
.java:1157)
at
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:388)
at
org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)
at
org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:418)
at org.mortbay.jetty.servlet.Dispatcher.forward(Dispatcher.java:327)
at org.mortbay.jetty.servlet.Dispatcher.forward(Dispatcher.java:126)
at
org.codehaus.groovy.grails.web.util.WebUtils.forwardRequestForUrlMappingInfo
(WebUtils.java:298)
at
org.codehaus.groovy.grails.web.util.WebUtils.forwardRequestForUrlMappingInfo
(WebUtils.java:264)
at
org.codehaus.groovy.grails.web.util.WebUtils.forwardRequestForUrlMappingInfo
(WebUtils.java:255)
at
org.codehaus.groovy.grails.web.mapping.filter.UrlMappingsFilter.doFilterInte
rnal(UrlMappingsFilter.java:183)
at
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestF
ilter.java:76)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler
.java:1157)
at
org.codehaus.groovy.grails.web.sitemesh.GrailsPageFilter.obtainContent(Grail
sPageFilter.java:245)
at
org.codehaus.groovy.grails.web.sitemesh.GrailsPageFilter.doFilter(GrailsPage
Filter.java:134)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler
.java:1157)
at
org.codehaus.groovy.grails.web.servlet.mvc.GrailsWebRequestFilter.doFilterIn
ternal(GrailsWebRequestFilter.java:69)
at
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestF
ilter.java:76)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler
.java:1157)
at
org.codehaus.groovy.grails.web.filters.HiddenHttpMethodFilter.doFilterIntern
al(HiddenHttpMethodFilter.java:65)
at
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestF
ilter.java:76)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler
.java:1157)
at
org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(Char
acterEncodingFilter.java:88)
at
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestF
ilter.java:76)
at
org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(Delegati
ngFilterProxy.java:237)
at
org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilt
erProxy.java:167)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler
.java:1157)
at
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:388)
at
org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)
at
org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:418)
at
org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at
org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:536)
at
org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:
930)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:747)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:405)
at
org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:22
8)
at
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582
)
2013-04-23 16:45:17,960 [116314147@qtp-405223709-32] ERROR
org.codehaus.groovy.grails.web.errors.GrailsExceptionResolver - Exception
occurred when processing request: [POST] /rundeck/;jsessionid=iy4ikika4t1c
Stacktrace follows:
org.springframework.transaction.CannotCreateTransactionException: Could not
open Hibernate Session for transaction; nested exception is
org.hibernate.TransactionException: JDBC begin failed:
at
rundeck.services.UserService$$EnhancerByCGLIB$$289a4772.getFilterPref()
at rundeck.services.UserService$getFilterPref.call(Unknown Source)
at
rundeck.filters.ProjectSelectFilters$_closure1_closure3_closure5.doCall(Proj
ectSelectFilters.groovy:97)
at
rundeck.filters.ProjectSelectFilters$_closure1_closure3_closure5.doCall(Proj
ectSelectFilters.groovy)
Caused by: org.hibernate.TransactionException: JDBC begin failed:
... 4 more
Caused by: org.h2.jdbc.JdbcSQLException: Database may be already in use:
Locked by another process. Possible solutions: close all other
connection(s); use the server mode [90020-112]
at org.h2.message.Message.getSQLException(Message.java:107)
at org.h2.message.Message.getSQLException(Message.java:118)
at org.h2.message.Message.getSQLException(Message.java:77)
at org.h2.store.FileLock.getExceptionAlreadyInUse(FileLock.java:475)
at org.h2.store.FileLock.lockFile(FileLock.java:338)
at org.h2.store.FileLock.lock(FileLock.java:130)
at org.h2.engine.Database.open(Database.java:557)
at org.h2.engine.Database.openDatabase(Database.java:221)
at org.h2.engine.Database.(Database.java:216)
at org.h2.engine.Engine.openSession(Engine.java:57)
at org.h2.engine.Engine.openSession(Engine.java:139)
at org.h2.engine.Engine.getSession(Engine.java:119)
at
org.h2.engine.SessionFactoryEmbedded.createSession(SessionFactoryEmbedded.ja
va:17)
at
org.h2.engine.SessionRemote.connectEmbeddedOrServer(SessionRemote.java:248)
at org.h2.engine.SessionRemote.createSession(SessionRemote.java:226)
at org.h2.jdbc.JdbcConnection.(JdbcConnection.java:106)
at org.h2.jdbc.JdbcConnection.(JdbcConnection.java:90)
at org.h2.Driver.connect(Driver.java:58)
at java.sql.DriverManager.getConnection(DriverManager.java:620)
at java.sql.DriverManager.getConnection(DriverManager.java:169)
... 4 more
2013-04-23 16:45:19,068 [main] ERROR
org.springframework.web.context.ContextLoader - Context initialization
failed
org.springframework.beans.factory.BeanCreationException: Error creating bean
with name 'messageSource': Initialization of bean failed; nested exception
is org.springframework.beans.factory.BeanCreationException: Error creating
bean with name 'transactionManager': Cannot resolve reference to bean
'sessionFactory' while setting bean property 'sessionFactory'; nested
exception is org.springframework.beans.factory.BeanCreationException: Error
creating bean with name 'sessionFactory': Cannot resolve reference to bean
'lobHandlerDetector' while setting bean property 'lobHandler'; nested
exception is org.springframework.beans.factory.BeanCreationException: Error
creating bean with name 'lobHandlerDetector': Invocation of init method
failed; nested exception is
org.springframework.jdbc.support.MetaDataAccessException: Error while
extracting DatabaseMetaData; nested exception is
org.h2.jdbc.JdbcSQLException: Database may be already in use: Locked by
another process. Possible solutions: close all other connection(s); use the
server mode [90020-112]
at com.dtolabs.rundeck.RunServer.run(RunServer.java:101)
at com.dtolabs.rundeck.RunServer.main(RunServer.java:64)
Caused by: org.springframework.beans.factory.BeanCreationException: Error
creating bean with name 'transactionManager': Cannot resolve reference to
bean 'sessionFactory' while setting bean property 'sessionFactory'; nested
exception is org.springframework.beans.factory.BeanCreationException: Error
creating bean with name 'sessionFactory': Cannot resolve reference to bean
'lobHandlerDetector' while setting bean property 'lobHandler'; nested
exception is org.springframework.beans.factory.BeanCreationException: Error
creating bean with name 'lobHandlerDetector': Invocation of init method
failed; nested exception is
org.springframework.jdbc.support.MetaDataAccessException: Error while
extracting DatabaseMetaData; nested exception is
org.h2.jdbc.JdbcSQLException: Database may be already in use: Locked by
another process. Possible solutions: close all other connection(s); use the
server mode [90020-112]
... 2 more
Caused by: org.springframework.beans.factory.BeanCreationException: Error
creating bean with name 'sessionFactory': Cannot resolve reference to bean
'lobHandlerDetector' while setting bean property 'lobHandler'; nested
exception is org.springframework.beans.factory.BeanCreationException: Error
creating bean with name 'lobHandlerDetector': Invocation of init method
failed; nested exception is
org.springframework.jdbc.support.MetaDataAccessException: Error while
extracting DatabaseMetaData; nested exception is
org.h2.jdbc.JdbcSQLException: Database may be already in use: Locked by
another process. Possible solutions: close all other connection(s); use the
server mode [90020-112]
... 2 more
Caused by: org.springframework.beans.factory.BeanCreationException: Error
creating bean with name 'lobHandlerDetector': Invocation of init method
failed; nested exception is
org.springframework.jdbc.support.MetaDataAccessException: Error while
extracting DatabaseMetaData; nested exception is
org.h2.jdbc.JdbcSQLException: Database may be already in use: Locked by
another process. Possible solutions: close all other connection(s); use the
server mode [90020-112]
... 2 more
Caused by: org.springframework.jdbc.support.MetaDataAccessException: Error
while extracting DatabaseMetaData; nested exception is
org.h2.jdbc.JdbcSQLException: Database may be already in use: Locked by
another process. Possible solutions: close all other connection(s); use the
server mode [90020-112]
... 2 more
Caused by: org.h2.jdbc.JdbcSQLException: Database may be already in use:
Locked by another process. Possible solutions: close all other
connection(s); use the server mode [90020-112]
at org.h2.message.Message.getSQLException(Message.java:107)
at org.h2.message.Message.getSQLException(Message.java:118)
at org.h2.message.Message.getSQLException(Message.java:77)
at org.h2.store.FileLock.getExceptionAlreadyInUse(FileLock.java:475)
at org.h2.store.FileLock.lockFile(FileLock.java:338)
at org.h2.store.FileLock.lock(FileLock.java:130)
at org.h2.engine.Database.open(Database.java:557)
at org.h2.engine.Database.openDatabase(Database.java:221)
at org.h2.engine.Database.(Database.java:216)
at org.h2.engine.Engine.openSession(Engine.java:57)
at org.h2.engine.Engine.openSession(Engine.java:139)
at org.h2.engine.Engine.getSession(Engine.java:119)
at
org.h2.engine.SessionFactoryEmbedded.createSession(SessionFactoryEmbedded.ja
va:17)
at
org.h2.engine.SessionRemote.connectEmbeddedOrServer(SessionRemote.java:248)
at org.h2.engine.SessionRemote.createSession(SessionRemote.java:226)
at org.h2.jdbc.JdbcConnection.(JdbcConnection.java:106)
at org.h2.jdbc.JdbcConnection.(JdbcConnection.java:90)
at org.h2.Driver.connect(Driver.java:58)
at java.sql.DriverManager.getConnection(DriverManager.java:620)
at java.sql.DriverManager.getConnection(DriverManager.java:169)
... 2 more
2013-04-23 16:45:32,911 [quartzScheduler_Worker-4] INFO
grails.app.service.rundeck.services.ExecutionService - Execution successful:
60
2013-04-23 16:45:32,958 [quartzScheduler_Worker-4] INFO
grails.app.service.rundeck.services.ExecutionService - updated scheduled
Execution

From: Greg Schueler notifications@github.com
Reply-To: dtolabs/rundeck
<reply+i-5309978-47074443a3d953fd2a67f085127b5d497215ea7f-571449@reply.githu
b.com>
Date: Tuesday, April 23, 2013 4:32 PM
To: dtolabs/rundeck rundeck@noreply.github.com
Cc: Charles Scott connaryscott@gmail.com
Subject: Re: [rundeck] Parallel/Concurrent jobs fail (#180)

hi chuck, do you have any log info from the server?


Reply to this email directly or view it on GitHub
#180 (comment) .

@connaryscott
Copy link

WeirdŠ

I tried to restart it but it is definitely hosed as restart here shows that
the old procs were not killed so in weird state

[root@centgoldrk1v trunk]# service rundeckd restart
Stopping rundeckd: [FAILED]
Starting rundeckd: nohup: ignoring input and redirecting stderr to stdout
[ OK ]
[root@centgoldrk1v trunk]# ps -ef |grep rundeck
root 21860 1 0 16:49 pts/0 00:00:00 runuser -l rundeck -c
/usr/bin/java
-Djava.security.auth.login.config=/etc/rundeck/jaas-activedirectory.conf
-Dserver.web.context=/rundeck ?-Dloginmodule.name=activedirectory
?-Drdeck.config=/etc/rundeck ?-Drdeck.base=/etc/rundeck
?-Drundeck.server.configDir=/etc/rundeck
?-Dserver.datastore.path=/var/lib/rundeck/data
?-Drundeck.server.serverDir=/var/lib/rundeck
?-Drdeck.projects=/var/rundeck/projects
?-Drdeck.runlogs=/var/lib/rundeck/logs
?-Drundeck.config.name=/etc/rundeck/rundeck-config.properties
?-Djava.io.tmpdir=/tmp/rundeck -Xmx1024m -Xms256m -cp
/var/lib/rundeck/bootstrap/jetty-util-6.1.21.jar:/var/lib/rundeck/bootstrap/
servlet-api-2.5-20081211.jar:/var/lib/rundeck/bootstrap/rundeck-jetty-server
-1.5.1-SNAPSHOT.jar:/var/lib/rundeck/bootstrap/jetty-plus-6.1.21.jar:/var/li
b/rundeck/bootstrap/jetty-6.1.21.jar:/var/lib/rundeck/bootstrap/jetty-naming
-6.1.21.jar::/etc/rundeck com.dtolabs.rundeck.RunServer /etc/rundeck 4440
rundeck 21862 21860 99 16:49 ? 00:00:07 /usr/bin/java
-Djava.security.auth.login.config=/etc/rundeck/jaas-activedirectory.conf
-Dserver.web.context=/rundeck -Dloginmodule.name=activedirectory
-Drdeck.config=/etc/rundeck -Drdeck.base=/etc/rundeck
-Drundeck.server.configDir=/etc/rundeck
-Dserver.datastore.path=/var/lib/rundeck/data
-Drundeck.server.serverDir=/var/lib/rundeck
-Drdeck.projects=/var/rundeck/projects -Drdeck.runlogs=/var/lib/rundeck/logs
-Drundeck.config.name=/etc/rundeck/rundeck-config.properties
-Djava.io.tmpdir=/tmp/rundeck -Xmx1024m -Xms256m -cp
/var/lib/rundeck/bootstrap/jetty-util-6.1.21.jar:/var/lib/rundeck/bootstrap/
servlet-api-2.5-20081211.jar:/var/lib/rundeck/bootstrap/rundeck-jetty-server
-1.5.1-SNAPSHOT.jar:/var/lib/rundeck/bootstrap/jetty-plus-6.1.21.jar:/var/li
b/rundeck/bootstrap/jetty-6.1.21.jar:/var/lib/rundeck/bootstrap/jetty-naming
-6.1.21.jar::/etc/rundeck com.dtolabs.rundeck.RunServer /etc/rundeck 4440
root 21906 13166 0 16:49 pts/0 00:00:00 grep rundeck
root 32106 1 0 16:01 pts/0 00:00:00 runuser -l rundeck -c
/usr/bin/java
-Djava.security.auth.login.config=/etc/rundeck/jaas-activedirectory.conf
-Dserver.web.context=/rundeck ?-Dloginmodule.name=activedirectory
?-Drdeck.config=/etc/rundeck ?-Drdeck.base=/etc/rundeck
?-Drundeck.server.configDir=/etc/rundeck
?-Dserver.datastore.path=/var/lib/rundeck/data
?-Drundeck.server.serverDir=/var/lib/rundeck
?-Drdeck.projects=/var/rundeck/projects
?-Drdeck.runlogs=/var/lib/rundeck/logs
?-Drundeck.config.name=/etc/rundeck/rundeck-config.properties
?-Djava.io.tmpdir=/tmp/rundeck -Xmx1024m -Xms256m -cp
/var/lib/rundeck/bootstrap/jetty-util-6.1.21.jar:/var/lib/rundeck/bootstrap/
servlet-api-2.5-20081211.jar:/var/lib/rundeck/bootstrap/rundeck-jetty-server
-1.5.1-SNAPSHOT.jar:/var/lib/rundeck/bootstrap/jetty-plus-6.1.21.jar:/var/li
b/rundeck/bootstrap/jetty-6.1.21.jar:/var/lib/rundeck/bootstrap/jetty-naming
-6.1.21.jar::/etc/rundeck com.dtolabs.rundeck.RunServer /etc/rundeck 4440
rundeck 32112 32106 2 16:01 ? 00:01:24 /usr/bin/java
-Djava.security.auth.login.config=/etc/rundeck/jaas-activedirectory.conf
-Dserver.web.context=/rundeck -Dloginmodule.name=activedirectory
-Drdeck.config=/etc/rundeck -Drdeck.base=/etc/rundeck
-Drundeck.server.configDir=/etc/rundeck
-Dserver.datastore.path=/var/lib/rundeck/data
-Drundeck.server.serverDir=/var/lib/rundeck
-Drdeck.projects=/var/rundeck/projects -Drdeck.runlogs=/var/lib/rundeck/logs
-Drundeck.config.name=/etc/rundeck/rundeck-config.properties
-Djava.io.tmpdir=/tmp/rundeck -Xmx1024m -Xms256m -cp
/var/lib/rundeck/bootstrap/jetty-util-6.1.21.jar:/var/lib/rundeck/bootstrap/
servlet-api-2.5-20081211.jar:/var/lib/rundeck/bootstrap/rundeck-jetty-server
-1.5.1-SNAPSHOT.jar:/var/lib/rundeck/bootstrap/jetty-plus-6.1.21.jar:/var/li
b/rundeck/bootstrap/jetty-6.1.21.jar:/var/lib/rundeck/bootstrap/jetty-naming
-6.1.21.jar::/etc/rundeck com.dtolabs.rundeck.RunServer /etc/rundeck 4440

Killing processes and starting over again:
[root@centgoldrk1v trunk]# ps -ef |grep rundeck |awk '{print $2}' |xargs
kill
kill 21984: No such process

[root@centgoldrk1v trunk]# service rundeckd start
Starting rundeckd: nohup: redirecting stderr to stdout
[ OK ]

Will retry again
From: Greg Schueler notifications@github.com
Reply-To: dtolabs/rundeck
<reply+i-5309978-47074443a3d953fd2a67f085127b5d497215ea7f-571449@reply.githu
b.com>
Date: Tuesday, April 23, 2013 4:46 PM
To: dtolabs/rundeck rundeck@noreply.github.com
Cc: Charles Scott connaryscott@gmail.com
Subject: Re: [rundeck] Parallel/Concurrent jobs fail (#180)

yes, that does look like a second process was already locking the H2
database


Reply to this email directly or view it on GitHub
#180 (comment) .

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

4 participants