Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Possible resource leak or MySQL concurrent connection issue #76

Open
ansell opened this issue Nov 8, 2017 · 10 comments
Open

Possible resource leak or MySQL concurrent connection issue #76

ansell opened this issue Nov 8, 2017 · 10 comments
Assignees
@temi

Comments

@ansell
Copy link
Contributor

ansell commented Nov 8, 2017

Both biocache-b4-a and biocache-b4-b, which are behind a load balancer, are showing symptoms of a possible resource leak or a MySQL concurrent connection issue. The symptoms reappeared after a full system restart.

Some stacktraces when it fails looks like:

org.apache.tomcat.jdbc.pool.PoolExhaustedException: [http-bio-8080-exec-360] Timeout: Pool empty. Unable to fetch a connection in 30 seconds, none available[size:6; busy:5; idle:0; lastwait:30000]. at org.apache.tomcat.jdbc.pool.ConnectionPool.borrowConnection(ConnectionPool.java:706) at org.apache.tomcat.jdbc.pool.ConnectionPool.getConnection(ConnectionPool.java:198) at org.apache.tomcat.jdbc.pool.DataSourceProxy.getConnection(DataSourceProxy.java:132) at org.springframework.jdbc.datasource.LazyConnectionDataSourceProxy$LazyConnectionInvocationHandler.getTargetConnection(LazyConnectionDataSourceProxy.java:403) at org.springframework.jdbc.datasource.LazyConnectionDataSourceProxy$LazyConnectionInvocationHandler.invoke(LazyConnectionDataSourceProxy.java:376) at com.sun.proxy.$Proxy89.prepareStatement(Unknown Source) at sun.reflect.GeneratedMethodAccessor213.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.jdbc.datasource.TransactionAwareDataSourceProxy$TransactionAwareInvocationHandler.invoke(TransactionAwareDataSourceProxy.java:240) at com.sun.proxy.$Proxy89.prepareStatement(Unknown Source) at org.hibernate.engine.jdbc.internal.StatementPreparerImpl$5.doPrepare(StatementPreparerImpl.java:146) at org.hibernate.engine.jdbc.internal.StatementPreparerImpl$StatementPreparationTemplate.prepareStatement(StatementPreparerImpl.java:172) at org.hibernate.engine.jdbc.internal.StatementPreparerImpl.prepareQueryStatement(StatementPreparerImpl.java:148) at org.hibernate.loader.Loader.prepareQueryStatement(Loader.java:1934) at org.hibernate.loader.Loader.executeQueryStatement(Loader.java:1903) at org.hibernate.loader.Loader.executeQueryStatement(Loader.java:1881) at org.hibernate.loader.Loader.doQuery(Loader.java:925) at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:342) at org.hibernate.loader.Loader.doList(Loader.java:2622) at org.hibernate.loader.Loader.doList(Loader.java:2605) at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2434) at org.hibernate.loader.Loader.list(Loader.java:2429) at org.hibernate.loader.criteria.CriteriaLoader.list(CriteriaLoader.java:109) at org.hibernate.internal.SessionImpl.list(SessionImpl.java:1787) at org.hibernate.internal.CriteriaImpl.list(CriteriaImpl.java:363) at org.grails.orm.hibernate.query.AbstractHibernateQuery.singleResultViaListCall(AbstractHibernateQuery.java:785) at org.grails.orm.hibernate.query.AbstractHibernateQuery.singleResult(AbstractHibernateQuery.java:775) at org.grails.datastore.gorm.finders.AbstractFindByFinder.invokeQuery(AbstractFindByFinder.java:35) at org.grails.datastore.gorm.finders.AbstractFindByFinder$1.doInSession(AbstractFindByFinder.java:29) at org.grails.datastore.mapping.core.DatastoreUtils.execute(DatastoreUtils.java:319) at org.grails.datastore.gorm.finders.AbstractFinder.execute(AbstractFinder.java:42) at org.grails.datastore.gorm.finders.AbstractFindByFinder.doInvokeInternal(AbstractFindByFinder.java:27) at org.grails.datastore.gorm.finders.DynamicFinder.invoke(DynamicFinder.java:174) at org.grails.datastore.gorm.finders.DynamicFinder.invoke(DynamicFinder.java:374) at org.grails.datastore.gorm.finders.FinderMethod$invoke$1.call(Unknown Source) at org.grails.datastore.gorm.GormStaticApi$_methodMissing_closure2.doCall(GormStaticApi.groovy:182) at sun.reflect.GeneratedMethodAccessor489.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93) at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325) at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:294) at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022) at groovy.lang.Closure.call(Closure.java:414) at org.codehaus.groovy.runtime.metaclass.ClosureStaticMetaMethod.invoke(ClosureStaticMetaMethod.java:62) at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325) at org.codehaus.groovy.runtime.callsite.StaticMetaMethodSite.invoke(StaticMetaMethodSite.java:46) at org.codehaus.groovy.runtime.callsite.StaticMetaMethodSite.call(StaticMetaMethodSite.java:91) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125) at au.org.ala.specieslist.SecurityUtil.checkListAccess(SecurityUtil.groovy:14) at au.org.ala.specieslist.SecurityUtil$checkListAccess$0.call(Unknown Source) at au.org.ala.specieslist.SecurityInterceptor.checkSecurity(SecurityInterceptor.groovy:27) at sun.reflect.GeneratedMethodAccessor538.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce.invoke(PogoMetaMethodSite.java:210) at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.callCurrent(PogoMetaMethodSite.java:59) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:182) at au.org.ala.specieslist.SecurityInterceptor.before(SecurityInterceptor.groovy:18) at org.grails.plugins.web.interceptors.GrailsInterceptorHandlerInterceptorAdapter.preHandle(GrailsInterceptorHandlerInterceptorAdapter.groovy:73) at org.springframework.web.servlet.HandlerExecutionChain.applyPreHandle(HandlerExecutionChain.java:133) at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:962) at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901) at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970) at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:861) at javax.servlet.http.HttpServlet.service(HttpServlet.java:624) at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846) at javax.servlet.http.HttpServlet.service(HttpServlet.java:731) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.springframework.boot.web.filter.ApplicationContextHeaderFilter.doFilterInternal(ApplicationContextHeaderFilter.java:55) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.springframework.boot.actuate.trace.WebRequestTraceFilter.doFilterInternal(WebRequestTraceFilter.java:105) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.grails.web.servlet.mvc.GrailsWebRequestFilter.doFilterInternal(GrailsWebRequestFilter.java:77) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.grails.web.filters.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:67) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at au.org.ala.cas.client.UriFilter.doFilter(UriFilter.java:199) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at au.org.ala.cas.client.UriFilter.doFilter(UriFilter.java:199) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at au.org.ala.cas.client.UriFilter.doFilter(UriFilter.java:199) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.jasig.cas.client.session.SingleSignOutFilter.doFilter(SingleSignOutFilter.java:97) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:208) at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:177) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:262) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.springframework.web.filter.CorsFilter.doFilterInternal(CorsFilter.java:96) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.springframework.boot.actuate.autoconfigure.MetricsFilter.doFilterInternal(MetricsFilter.java:106) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.springframework.boot.web.support.ErrorPageFilter.doFilter(ErrorPageFilter.java:117) at org.springframework.boot.web.support.ErrorPageFilter.access$000(ErrorPageFilter.java:61) at org.springframework.boot.web.support.ErrorPageFilter$1.doFilterInternal(ErrorPageFilter.java:92) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.springframework.boot.web.support.ErrorPageFilter.doFilter(ErrorPageFilter.java:110) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:221) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:505) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103) at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:956) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:436) at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1078) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:625) at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:748)

org.hibernate.exception.GenericJDBCException: could not prepare statement at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:47) at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:111) at org.hibernate.engine.jdbc.internal.StatementPreparerImpl$StatementPreparationTemplate.prepareStatement(StatementPreparerImpl.java:182) at org.hibernate.engine.jdbc.internal.StatementPreparerImpl.prepareQueryStatement(StatementPreparerImpl.java:148) at org.hibernate.loader.Loader.prepareQueryStatement(Loader.java:1934) at org.hibernate.loader.Loader.executeQueryStatement(Loader.java:1903) at org.hibernate.loader.Loader.executeQueryStatement(Loader.java:1881) at org.hibernate.loader.Loader.doQuery(Loader.java:925) at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:342) at org.hibernate.loader.Loader.doList(Loader.java:2622) at org.hibernate.loader.Loader.doList(Loader.java:2605) at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2434) at org.hibernate.loader.Loader.list(Loader.java:2429) at org.hibernate.loader.criteria.CriteriaLoader.list(CriteriaLoader.java:109) at org.hibernate.internal.SessionImpl.list(SessionImpl.java:1787) at org.hibernate.internal.CriteriaImpl.list(CriteriaImpl.java:363) at grails.orm.PagedResultList.<init>(PagedResultList.java:45) at grails.orm.HibernateCriteriaBuilder.createPagedResultList(HibernateCriteriaBuilder.java:161) at org.grails.orm.hibernate.query.AbstractHibernateCriteriaBuilder.invokeMethod(AbstractHibernateCriteriaBuilder.java:1684) at org.grails.orm.hibernate.query.AbstractHibernateCriteriaBuilder.list(AbstractHibernateCriteriaBuilder.java:1563) at au.org.ala.specieslist.QueryService.getFilterListItemResult(QueryService.groovy:183) at au.org.ala.specieslist.WebServiceController.getListItemsForSpecies(WebServiceController.groovy:73) at org.grails.core.DefaultGrailsControllerClass$MethodHandleInvoker.invoke(DefaultGrailsControllerClass.java:223) at org.grails.core.DefaultGrailsControllerClass.invoke(DefaultGrailsControllerClass.java:188) at org.grails.web.mapping.mvc.UrlMappingsInfoHandlerAdapter.handle(UrlMappingsInfoHandlerAdapter.groovy:90) at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967) at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901) at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970) at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:861) at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846) at org.springframework.boot.web.filter.ApplicationContextHeaderFilter.doFilterInternal(ApplicationContextHeaderFilter.java:55) at org.grails.web.servlet.mvc.GrailsWebRequestFilter.doFilterInternal(GrailsWebRequestFilter.java:77) at org.grails.web.filters.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:67) at au.org.ala.cas.client.UriFilter.doFilter(UriFilter.java:199) at au.org.ala.cas.client.UriFilter.doFilter(UriFilter.java:199) at au.org.ala.cas.client.UriFilter.doFilter(UriFilter.java:199) at org.jasig.cas.client.session.SingleSignOutFilter.doFilter(SingleSignOutFilter.java:97) at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:208) at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:177) at org.springframework.boot.web.support.ErrorPageFilter.doFilter(ErrorPageFilter.java:117) at org.springframework.boot.web.support.ErrorPageFilter.access$000(ErrorPageFilter.java:61) at org.springframework.boot.web.support.ErrorPageFilter$1.doFilterInternal(ErrorPageFilter.java:92) at org.springframework.boot.web.support.ErrorPageFilter.doFilter(ErrorPageFilter.java:110) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.tomcat.jdbc.pool.PoolExhaustedException: [http-bio-8080-exec-343] Timeout: Pool empty. Unable to fetch a connection in 30 seconds, none available[size:6; busy:5; idle:0; lastwait:30000]. at org.springframework.jdbc.datasource.LazyConnectionDataSourceProxy$LazyConnectionInvocationHandler.getTargetConnection(LazyConnectionDataSourceProxy.java:403) at org.springframework.jdbc.datasource.LazyConnectionDataSourceProxy$LazyConnectionInvocationHandler.invoke(LazyConnectionDataSourceProxy.java:376) at org.springframework.jdbc.datasource.TransactionAwareDataSourceProxy$TransactionAwareInvocationHandler.invoke(TransactionAwareDataSourceProxy.java:240) at org.hibernate.engine.jdbc.internal.StatementPreparerImpl$5.doPrepare(StatementPreparerImpl.java:146) at org.hibernate.engine.jdbc.internal.StatementPreparerImpl$StatementPreparationTemplate.prepareStatement(StatementPreparerImpl.java:172) ... 43 common frames omitted

org.hibernate.exception.GenericJDBCException: could not prepare statement at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:47) at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:111) at org.hibernate.engine.jdbc.internal.StatementPreparerImpl$StatementPreparationTemplate.prepareStatement(StatementPreparerImpl.java:182) at org.hibernate.engine.jdbc.internal.StatementPreparerImpl.prepareQueryStatement(StatementPreparerImpl.java:148) at org.hibernate.loader.Loader.prepareQueryStatement(Loader.java:1934) at org.hibernate.loader.Loader.executeQueryStatement(Loader.java:1903) at org.hibernate.loader.Loader.executeQueryStatement(Loader.java:1881) at org.hibernate.loader.Loader.doQuery(Loader.java:925) at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:342) at org.hibernate.loader.Loader.doList(Loader.java:2622) at org.hibernate.loader.Loader.doList(Loader.java:2605) at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2434) at org.hibernate.loader.Loader.list(Loader.java:2429) at org.hibernate.loader.criteria.CriteriaLoader.list(CriteriaLoader.java:109) at org.hibernate.internal.SessionImpl.list(SessionImpl.java:1787) at org.hibernate.internal.CriteriaImpl.list(CriteriaImpl.java:363) at grails.orm.PagedResultList.<init>(PagedResultList.java:45) at grails.orm.HibernateCriteriaBuilder.createPagedResultList(HibernateCriteriaBuilder.java:161) at org.grails.orm.hibernate.query.AbstractHibernateCriteriaBuilder.invokeMethod(AbstractHibernateCriteriaBuilder.java:1684) at org.grails.orm.hibernate.query.AbstractHibernateCriteriaBuilder.list(AbstractHibernateCriteriaBuilder.java:1563) at au.org.ala.specieslist.QueryService.getFilterListItemResult(QueryService.groovy:183) at au.org.ala.specieslist.WebServiceController.getListItemsForSpecies(WebServiceController.groovy:73) at org.grails.core.DefaultGrailsControllerClass$MethodHandleInvoker.invoke(DefaultGrailsControllerClass.java:223) at org.grails.core.DefaultGrailsControllerClass.invoke(DefaultGrailsControllerClass.java:188) at org.grails.web.mapping.mvc.UrlMappingsInfoHandlerAdapter.handle(UrlMappingsInfoHandlerAdapter.groovy:90) at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967) at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901) at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970) at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:861) at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846) at org.springframework.boot.web.filter.ApplicationContextHeaderFilter.doFilterInternal(ApplicationContextHeaderFilter.java:55) at org.grails.web.servlet.mvc.GrailsWebRequestFilter.doFilterInternal(GrailsWebRequestFilter.java:77) at org.grails.web.filters.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:67) at au.org.ala.cas.client.UriFilter.doFilter(UriFilter.java:199) at au.org.ala.cas.client.UriFilter.doFilter(UriFilter.java:199) at au.org.ala.cas.client.UriFilter.doFilter(UriFilter.java:199) at org.jasig.cas.client.session.SingleSignOutFilter.doFilter(SingleSignOutFilter.java:97) at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:208) at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:177) at org.springframework.boot.web.support.ErrorPageFilter.doFilter(ErrorPageFilter.java:117) at org.springframework.boot.web.support.ErrorPageFilter.access$000(ErrorPageFilter.java:61) at org.springframework.boot.web.support.ErrorPageFilter$1.doFilterInternal(ErrorPageFilter.java:92) at org.springframework.boot.web.support.ErrorPageFilter.doFilter(ErrorPageFilter.java:110) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.tomcat.jdbc.pool.PoolExhaustedException: [http-bio-8080-exec-343] Timeout: Pool empty. Unable to fetch a connection in 30 seconds, none available[size:6; busy:5; idle:0; lastwait:30000]. at org.springframework.jdbc.datasource.LazyConnectionDataSourceProxy$LazyConnectionInvocationHandler.getTargetConnection(LazyConnectionDataSourceProxy.java:403) at org.springframework.jdbc.datasource.LazyConnectionDataSourceProxy$LazyConnectionInvocationHandler.invoke(LazyConnectionDataSourceProxy.java:376) at org.springframework.jdbc.datasource.TransactionAwareDataSourceProxy$TransactionAwareInvocationHandler.invoke(TransactionAwareDataSourceProxy.java:240) at org.hibernate.engine.jdbc.internal.StatementPreparerImpl$5.doPrepare(StatementPreparerImpl.java:146) at org.hibernate.engine.jdbc.internal.StatementPreparerImpl$StatementPreparationTemplate.prepareStatement(StatementPreparerImpl.java:172) ... 43 common frames omitted

On restart, the following occurred

2017-11-09 09:19:19.589  WARN --- [.au-startStop-1] ationConfigEmbeddedWebApplicationContext : Exception encountered during context initialization - cancelling refresh attempt: org.springframework.context.ApplicationContextException: Unable to start embedded container; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'grailsCacheFilter': Cannot create inner bean '(inner bean)#abec3a' of type [grails.plugin.cache.web.filter.simple.MemoryPageFragmentCachingFilter] while setting bean property 'filter'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name '(inner bean)#abec3a': Unsatisfied dependency expressed through method 'setUrlMappingsHandlerMapping' parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'urlMappingsHandlerMapping': Unsatisfied dependency expressed through method 'setHandlerInterceptors' parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'grailsInterceptorMappedInterceptor': Cannot create inner bean '(inner bean)#4a1c1880' of type [org.grails.plugins.web.interceptors.GrailsInterceptorHandlerInterceptorAdapter] while setting constructor argument; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name '(inner bean)#4a1c1880': Unsatisfied dependency expressed through method 'setInterceptors' parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'alaSecuredInterceptor': Initialization of bean failed; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'grailsLinkGenerator': Unsatisfied dependency expressed through field 'mappingContext'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'hibernateDatastore': Bean instantiation via constructor failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.grails.orm.hibernate.HibernateDatastore]: Constructor threw exception; nested exception is java.lang.NullPointerException
2017-11-09 09:19:19.622 ERROR --- [.au-startStop-1] o.s.boot.SpringApplication               : Application startup failed

org.springframework.context.ApplicationContextException: Unable to start embedded container; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'grailsCacheFilter': Cannot create inner bean '(inner bean)#abec3a' of type [grails.plugin.cache.web.filter.simple.MemoryPageFragmentCachingFilter] while setting bean property 'filter'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name '(inner bean)#abec3a': Unsatisfied dependency expressed through method 'setUrlMappingsHandlerMapping' parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'urlMappingsHandlerMapping': Unsatisfied dependency expressed through method 'setHandlerInterceptors' parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'grailsInterceptorMappedInterceptor': Cannot create inner bean '(inner bean)#4a1c1880' of type [org.grails.plugins.web.interceptors.GrailsInterceptorHandlerInterceptorAdapter] while setting constructor argument; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name '(inner bean)#4a1c1880': Unsatisfied dependency expressed through method 'setInterceptors' parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'alaSecuredInterceptor': Initialization of bean failed; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'grailsLinkGenerator': Unsatisfied dependency expressed through field 'mappingContext'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'hibernateDatastore': Bean instantiation via constructor failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.grails.orm.hibernate.HibernateDatastore]: Constructor threw exception; nested exception is java.lang.NullPointerException
	at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.onRefresh(EmbeddedWebApplicationContext.java:137)
	at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:537)
	at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.refresh(EmbeddedWebApplicationContext.java:122)
	at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:762)
	at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:372)
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:316)
	at grails.boot.GrailsApp.run(GrailsApp.groovy:83)
	at org.springframework.boot.web.support.SpringBootServletInitializer.run(SpringBootServletInitializer.java:151)
	at org.grails.boot.context.web.GrailsAppServletInitializer.createRootApplicationContext(GrailsAppServletInitializer.groovy:57)
	at org.springframework.boot.web.support.SpringBootServletInitializer.onStartup(SpringBootServletInitializer.java:86)
	at org.springframework.web.SpringServletContainerInitializer.onStartup(SpringServletContainerInitializer.java:169)
	at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5573)
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:147)
	at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:899)
	at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:875)
	at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652)
	at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:1091)
	at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1980)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'grailsCacheFilter': Cannot create inner bean '(inner bean)#abec3a' of type [grails.plugin.cache.web.filter.simple.MemoryPageFragmentCachingFilter] while setting bean property 'filter'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name '(inner bean)#abec3a': Unsatisfied dependency expressed through method 'setUrlMappingsHandlerMapping' parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'urlMappingsHandlerMapping': Unsatisfied dependency expressed through method 'setHandlerInterceptors' parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'grailsInterceptorMappedInterceptor': Cannot create inner bean '(inner bean)#4a1c1880' of type [org.grails.plugins.web.interceptors.GrailsInterceptorHandlerInterceptorAdapter] while setting constructor argument; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name '(inner bean)#4a1c1880': Unsatisfied dependency expressed through method 'setInterceptors' parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'alaSecuredInterceptor': Initialization of bean failed; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'grailsLinkGenerator': Unsatisfied dependency expressed through field 'mappingContext'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'hibernateDatastore': Bean instantiation via constructor failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.grails.orm.hibernate.HibernateDatastore]: Constructor threw exception; nested exception is java.lang.NullPointerException
	at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveInnerBean(BeanDefinitionValueResolver.java:313)
	at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:129)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyPropertyValues(AbstractAutowireCapableBeanFactory.java:1531)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1276)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:553)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483)
	at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
	at org.springframework.boot.web.servlet.ServletContextInitializerBeans.getOrderedBeansOfType(ServletContextInitializerBeans.java:234)
	at org.springframework.boot.web.servlet.ServletContextInitializerBeans.getOrderedBeansOfType(ServletContextInitializerBeans.java:215)
	at org.springframework.boot.web.servlet.ServletContextInitializerBeans.addServletContextInitializerBeans(ServletContextInitializerBeans.java:91)
	at org.springframework.boot.web.servlet.ServletContextInitializerBeans.<init>(ServletContextInitializerBeans.java:79)
	at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.getServletContextInitializerBeans(EmbeddedWebApplicationContext.java:241)
	at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.selfInitialize(EmbeddedWebApplicationContext.java:228)
	at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.access$000(EmbeddedWebApplicationContext.java:89)
	at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext$1.onStartup(EmbeddedWebApplicationContext.java:213)
	at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.createEmbeddedServletContainer(EmbeddedWebApplicationContext.java:168)
	at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.onRefresh(EmbeddedWebApplicationContext.java:134)
	... 22 common frames omitted
Caused by: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name '(inner bean)#abec3a': Unsatisfied dependency expressed through method 'setUrlMappingsHandlerMapping' parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'urlMappingsHandlerMapping': Unsatisfied dependency expressed through method 'setHandlerInterceptors' parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'grailsInterceptorMappedInterceptor': Cannot create inner bean '(inner bean)#4a1c1880' of type [org.grails.plugins.web.interceptors.GrailsInterceptorHandlerInterceptorAdapter] while setting constructor argument; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name '(inner bean)#4a1c1880': Unsatisfied dependency expressed through method 'setInterceptors' parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'alaSecuredInterceptor': Initialization of bean failed; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'grailsLinkGenerator': Unsatisfied dependency expressed through field 'mappingContext'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'hibernateDatastore': Bean instantiation via constructor failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.grails.orm.hibernate.HibernateDatastore]: Constructor threw exception; nested exception is java.lang.NullPointerException
	at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredMethodElement.inject(AutowiredAnnotationBeanPostProcessor.java:667)
	at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:88)
	at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessPropertyValues(AutowiredAnnotationBeanPostProcessor.java:366)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1264)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:553)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483)
	at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveInnerBean(BeanDefinitionValueResolver.java:299)
	... 41 common frames omitted
Caused by: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'urlMappingsHandlerMapping': Unsatisfied dependency expressed through method 'setHandlerInterceptors' parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'grailsInterceptorMappedInterceptor': Cannot create inner bean '(inner bean)#4a1c1880' of type [org.grails.plugins.web.interceptors.GrailsInterceptorHandlerInterceptorAdapter] while setting constructor argument; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name '(inner bean)#4a1c1880': Unsatisfied dependency expressed through method 'setInterceptors' parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'alaSecuredInterceptor': Initialization of bean failed; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'grailsLinkGenerator': Unsatisfied dependency expressed through field 'mappingContext'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'hibernateDatastore': Bean instantiation via constructor failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.grails.orm.hibernate.HibernateDatastore]: Constructor threw exception; nested exception is java.lang.NullPointerException
	at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredMethodElement.inject(AutowiredAnnotationBeanPostProcessor.java:667)
	at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:88)
	at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessPropertyValues(AutowiredAnnotationBeanPostProcessor.java:366)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1264)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:553)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483)
	at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
	at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:208)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1138)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1066)
	at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredMethodElement.inject(AutowiredAnnotationBeanPostProcessor.java:659)
	... 47 common frames omitted
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'grailsInterceptorMappedInterceptor': Cannot create inner bean '(inner bean)#4a1c1880' of type [org.grails.plugins.web.interceptors.GrailsInterceptorHandlerInterceptorAdapter] while setting constructor argument; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name '(inner bean)#4a1c1880': Unsatisfied dependency expressed through method 'setInterceptors' parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'alaSecuredInterceptor': Initialization of bean failed; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'grailsLinkGenerator': Unsatisfied dependency expressed through field 'mappingContext'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'hibernateDatastore': Bean instantiation via constructor failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.grails.orm.hibernate.HibernateDatastore]: Constructor threw exception; nested exception is java.lang.NullPointerException
	at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveInnerBean(BeanDefinitionValueResolver.java:313)
	at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:129)
	at org.springframework.beans.factory.support.ConstructorResolver.resolveConstructorArguments(ConstructorResolver.java:648)
	at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:145)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1193)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1095)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:513)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483)
	at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
	at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:208)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.addCandidateEntry(DefaultListableBeanFactory.java:1316)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.findAutowireCandidates(DefaultListableBeanFactory.java:1282)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveMultipleBeans(DefaultListableBeanFactory.java:1160)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1096)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1066)
	at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredMethodElement.inject(AutowiredAnnotationBeanPostProcessor.java:659)
	... 60 common frames omitted
Caused by: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name '(inner bean)#4a1c1880': Unsatisfied dependency expressed through method 'setInterceptors' parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'alaSecuredInterceptor': Initialization of bean failed; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'grailsLinkGenerator': Unsatisfied dependency expressed through field 'mappingContext'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'hibernateDatastore': Bean instantiation via constructor failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.grails.orm.hibernate.HibernateDatastore]: Constructor threw exception; nested exception is java.lang.NullPointerException
	at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredMethodElement.inject(AutowiredAnnotationBeanPostProcessor.java:667)
	at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:88)
	at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessPropertyValues(AutowiredAnnotationBeanPostProcessor.java:366)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1264)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:553)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483)
	at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveInnerBean(BeanDefinitionValueResolver.java:299)
	... 78 common frames omitted
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'alaSecuredInterceptor': Initialization of bean failed; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'grailsLinkGenerator': Unsatisfied dependency expressed through field 'mappingContext'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'hibernateDatastore': Bean instantiation via constructor failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.grails.orm.hibernate.HibernateDatastore]: Constructor threw exception; nested exception is java.lang.NullPointerException
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:564)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483)
	at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
	at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:208)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.addCandidateEntry(DefaultListableBeanFactory.java:1316)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.findAutowireCandidates(DefaultListableBeanFactory.java:1282)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveMultipleBeans(DefaultListableBeanFactory.java:1160)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1096)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1066)
	at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredMethodElement.inject(AutowiredAnnotationBeanPostProcessor.java:659)
	... 84 common frames omitted
Caused by: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'grailsLinkGenerator': Unsatisfied dependency expressed through field 'mappingContext'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'hibernateDatastore': Bean instantiation via constructor failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.grails.orm.hibernate.HibernateDatastore]: Constructor threw exception; nested exception is java.lang.NullPointerException
	at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:588)
	at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:88)
	at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessPropertyValues(AutowiredAnnotationBeanPostProcessor.java:366)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1264)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:553)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483)
	at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireByName(AbstractAutowireCapableBeanFactory.java:1294)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1244)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:553)
	... 96 common frames omitted
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'hibernateDatastore': Bean instantiation via constructor failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.grails.orm.hibernate.HibernateDatastore]: Constructor threw exception; nested exception is java.lang.NullPointerException
	at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:279)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1193)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1095)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:513)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483)
	at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
	at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:372)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1173)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1067)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:513)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483)
	at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
	at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:208)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1138)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1066)
	at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:585)
	... 108 common frames omitted
Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.grails.orm.hibernate.HibernateDatastore]: Constructor threw exception; nested exception is java.lang.NullPointerException
	at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:154)
	at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:122)
	at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:271)
	... 129 common frames omitted
Caused by: java.lang.NullPointerException: null
	at com.mysql.jdbc.StatementImpl.getWarnings(StatementImpl.java:2156)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.tomcat.jdbc.pool.StatementFacade$StatementProxy.invoke(StatementFacade.java:138)
	at com.sun.proxy.$Proxy92.getWarnings(Unknown Source)
	at org.hibernate.tool.schema.internal.exec.GenerationTargetToDatabase.accept(GenerationTargetToDatabase.java:57)
	at org.hibernate.tool.schema.internal.SchemaMigratorImpl.applySqlString(SchemaMigratorImpl.java:568)
	at org.hibernate.tool.schema.internal.SchemaMigratorImpl.applySqlStrings(SchemaMigratorImpl.java:552)
	at org.hibernate.tool.schema.internal.SchemaMigratorImpl.applyForeignKeys(SchemaMigratorImpl.java:511)
	at org.hibernate.tool.schema.internal.SchemaMigratorImpl.performMigration(SchemaMigratorImpl.java:310)
	at org.hibernate.tool.schema.internal.SchemaMigratorImpl.doMigration(SchemaMigratorImpl.java:132)
	at org.hibernate.tool.schema.internal.SchemaMigratorImpl.doMigration(SchemaMigratorImpl.java:105)
	at org.hibernate.tool.schema.spi.SchemaManagementToolCoordinator.performDatabaseAction(SchemaManagementToolCoordinator.java:176)
	at org.hibernate.tool.schema.spi.SchemaManagementToolCoordinator.process(SchemaManagementToolCoordinator.java:65)
	at org.hibernate.internal.SessionFactoryImpl.<init>(SessionFactoryImpl.java:459)
	at org.hibernate.boot.internal.SessionFactoryBuilderImpl.build(SessionFactoryBuilderImpl.java:465)
	at org.hibernate.cfg.Configuration.buildSessionFactory(Configuration.java:711)
	at org.grails.orm.hibernate.cfg.HibernateMappingContextConfiguration.buildSessionFactory(HibernateMappingContextConfiguration.java:259)
	at org.grails.orm.hibernate.connections.HibernateConnectionSourceFactory.create(HibernateConnectionSourceFactory.java:80)
	at org.grails.orm.hibernate.connections.AbstractHibernateConnectionSourceFactory.create(AbstractHibernateConnectionSourceFactory.java:38)
	at org.grails.orm.hibernate.connections.AbstractHibernateConnectionSourceFactory.create(AbstractHibernateConnectionSourceFactory.java:22)
	at org.grails.datastore.mapping.core.connections.AbstractConnectionSourceFactory.create(AbstractConnectionSourceFactory.java:64)
	at org.grails.datastore.mapping.core.connections.AbstractConnectionSourceFactory.create(AbstractConnectionSourceFactory.java:52)
	at org.grails.datastore.mapping.core.connections.ConnectionSourcesInitializer.create(ConnectionSourcesInitializer.groovy:24)
	at org.grails.orm.hibernate.HibernateDatastore.<init>(HibernateDatastore.java:206)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:142)
	... 131 common frames omitted
```
@ansell
Copy link
Contributor Author

ansell commented Nov 8, 2017

Sorry for the collapsed stacktraces, copied those from elk, which seems to have removed the line endings

@ansell
Copy link
Contributor Author

ansell commented Nov 9, 2017 •
edited

Setting useSSL=false in the connection string worked. Candidates for why useSSL made a difference are currently:

  • openssl/libssl unattended-upgrade 24 hours ago
  • AWS may have changed their configuration recently on the hosted MySQL instance we recently switched to

A factor that may be related to useSSL=false but not to the issues above (as it occurred after the log entries above) is:

  • openjdk/oraclejdk upgrade from 8u131 to 8u151

@ansell
Copy link
Contributor Author

ansell commented Nov 9, 2017

For avoidance of doubt, the pool exhaustion errors are still happening after "useSSL=false" was added, with the following from an hour ago:

2017-11-09 12:09:26.619 ERROR --- [io-8080-exec-15] o.g.web.errors.GrailsExceptionResolver   : PoolExhaustedException occurred when processing request: [GET] /speciesListItem/list/dr781
[http-bio-8080-exec-15] Timeout: Pool empty. Unable to fetch a connection in 30 seconds, none available[size:6; busy:5; idle:0; lastwait:30000].. Stacktrace follows:

org.hibernate.exception.GenericJDBCException: could not prepare statement
	at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:47)
	at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:111)
	at org.hibernate.engine.jdbc.internal.StatementPreparerImpl$StatementPreparationTemplate.prepareStatement(StatementPreparerImpl.java:182)
	at org.hibernate.engine.jdbc.internal.StatementPreparerImpl.prepareQueryStatement(StatementPreparerImpl.java:148)
	at org.hibernate.loader.Loader.prepareQueryStatement(Loader.java:1934)
	at org.hibernate.loader.Loader.executeQueryStatement(Loader.java:1903)
	at org.hibernate.loader.Loader.executeQueryStatement(Loader.java:1881)
	at org.hibernate.loader.Loader.doQuery(Loader.java:925)
	at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:342)
	at org.hibernate.loader.Loader.doList(Loader.java:2622)
	at org.hibernate.loader.Loader.doList(Loader.java:2605)
	at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2434)
	at org.hibernate.loader.Loader.list(Loader.java:2429)
	at org.hibernate.loader.criteria.CriteriaLoader.list(CriteriaLoader.java:109)
	at org.hibernate.internal.SessionImpl.list(SessionImpl.java:1787)
	at org.hibernate.internal.CriteriaImpl.list(CriteriaImpl.java:363)
	at org.grails.orm.hibernate.query.AbstractHibernateQuery.singleResultViaListCall(AbstractHibernateQuery.java:785)
	at org.grails.orm.hibernate.query.AbstractHibernateQuery.singleResult(AbstractHibernateQuery.java:775)
	at org.grails.datastore.gorm.finders.AbstractFindByFinder.invokeQuery(AbstractFindByFinder.java:35)
	at org.grails.datastore.gorm.finders.AbstractFindByFinder$1.doInSession(AbstractFindByFinder.java:29)
	at org.grails.datastore.mapping.core.DatastoreUtils.execute(DatastoreUtils.java:319)
	at org.grails.datastore.gorm.finders.AbstractFinder.execute(AbstractFinder.java:42)
	at org.grails.datastore.gorm.finders.AbstractFindByFinder.doInvokeInternal(AbstractFindByFinder.java:27)
	at org.grails.datastore.gorm.finders.DynamicFinder.invoke(DynamicFinder.java:174)
	at org.grails.datastore.gorm.finders.DynamicFinder.invoke(DynamicFinder.java:374)
	at org.grails.datastore.gorm.GormStaticApi$_methodMissing_closure2.doCall(GormStaticApi.groovy:182)
	at au.org.ala.specieslist.SecurityUtil.checkListAccess(SecurityUtil.groovy:14)
	at au.org.ala.specieslist.SecurityInterceptor.checkSecurity(SecurityInterceptor.groovy:27)
	at au.org.ala.specieslist.SecurityInterceptor.before(SecurityInterceptor.groovy:18)
	at org.grails.plugins.web.interceptors.GrailsInterceptorHandlerInterceptorAdapter.preHandle(GrailsInterceptorHandlerInterceptorAdapter.groovy:73)
	at org.springframework.web.servlet.HandlerExecutionChain.applyPreHandle(HandlerExecutionChain.java:133)
	at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:962)
	at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)
	at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
	at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:861)
	at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
	at org.springframework.boot.web.filter.ApplicationContextHeaderFilter.doFilterInternal(ApplicationContextHeaderFilter.java:55)
	at org.grails.web.servlet.mvc.GrailsWebRequestFilter.doFilterInternal(GrailsWebRequestFilter.java:77)
	at org.grails.web.filters.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:67)
	at au.org.ala.cas.client.UriFilter.doFilter(UriFilter.java:199)
	at au.org.ala.cas.client.UriFilter.doFilter(UriFilter.java:199)
	at au.org.ala.cas.client.UriFilter.doFilter(UriFilter.java:199)
	at org.jasig.cas.client.session.SingleSignOutFilter.doFilter(SingleSignOutFilter.java:97)
	at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:208)
	at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:177)
	at org.springframework.boot.web.support.ErrorPageFilter.doFilter(ErrorPageFilter.java:117)
	at org.springframework.boot.web.support.ErrorPageFilter.access$000(ErrorPageFilter.java:61)
	at org.springframework.boot.web.support.ErrorPageFilter$1.doFilterInternal(ErrorPageFilter.java:92)
	at org.springframework.boot.web.support.ErrorPageFilter.doFilter(ErrorPageFilter.java:110)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.tomcat.jdbc.pool.PoolExhaustedException: [http-bio-8080-exec-15] Timeout: Pool empty. Unable to fetch a connection in 30 seconds, none available[size:6; busy:5; idle:0; lastwait:30000].
	at org.springframework.jdbc.datasource.LazyConnectionDataSourceProxy$LazyConnectionInvocationHandler.getTargetConnection(LazyConnectionDataSourceProxy.java:403)
	at org.springframework.jdbc.datasource.LazyConnectionDataSourceProxy$LazyConnectionInvocationHandler.invoke(LazyConnectionDataSourceProxy.java:376)
	at org.springframework.jdbc.datasource.TransactionAwareDataSourceProxy$TransactionAwareInvocationHandler.invoke(TransactionAwareDataSourceProxy.java:240)
	at org.hibernate.engine.jdbc.internal.StatementPreparerImpl$5.doPrepare(StatementPreparerImpl.java:146)
	at org.hibernate.engine.jdbc.internal.StatementPreparerImpl$StatementPreparationTemplate.prepareStatement(StatementPreparerImpl.java:172)
	... 49 common frames omitted

@temi
Copy link
Contributor

temi commented Nov 9, 2017

@ansell raise timeout to 60 seconds?

@ansell
Copy link
Contributor Author

ansell commented Nov 9, 2017

I am not sure what steps to take, just collecting the debugging information to improve the lists server availability.

If it is a resource leak, it is possible that some of the 6 connections in the pool are permanently blocked, but difficult to say without taking a heap dump or attaching a debugger.

The log file above is from bie-b4-b if you want to log into now and do some forensics.

Forensics could help to distinguish between the long query case (where the increased timeout could help) and the connection pool size and resource leak possibilities, where and increased timeout wouldn't matter, and could possibly accentuate the issue by blocking some of the connections for longer.

@ansell
Copy link
Contributor Author

ansell commented Nov 14, 2017 •
edited

This is an example of a stacktrace taken while the lists server was having issues:

bie-b4-a-connectionpooldebug-stacktraces.txt

The process for generating one of those is to run ps to find the process id:

ps aux | grep java

Then use the process id in the following (replace "PID" with the process id you found above):

sudo jstack -F PID > stacktrace.txt

@ansell
Copy link
Contributor Author

ansell commented Nov 15, 2017

Tracked down the issue to SpeciesListItemController.generateFacetValues by opening up the number of connections to 50 and watching the actively used connections gradually fill up with threads under that path. Could be related to issue #78 but possibly a separate issue

@ansell
Copy link
Contributor Author

ansell commented Nov 15, 2017 •
edited

Executed one of the dr652 queries on the mysql command line tool and it took 9 seconds, so there is room for performance improvements:

select specieslis2_.the_key as col_0_0_, specieslis2_.value as col_1_0_, specieslis2_.vocab_value as col_2_0_, count(specieslis0_.id) as col_3_0_ from species_list_item specieslis0_ inner join species_list_item_species_listkvp kvpvalues1_ on specieslis0_.id=kvpvalues1_.species_list_item_kvp_values_id inner join species_listkvp specieslis2_ on kvpvalues1_.species_listkvp_id=specieslis2_.id where specieslis0_.data_resource_uid='dr652' group by specieslis2_.the_key , specieslis2_.value , specieslis2_.vocab_value order by specieslis2_.item_order, specieslis2_.the_key, col_3_0_ desc;

31826 rows in set (9.00 sec)

@ansell
Copy link
Contributor Author

ansell commented Nov 15, 2017

The query examining the most records is:

select this_.id as id1_2_2_, this_.version as version2_2_2_, this_.author as author3_2_2_, this_.common_name as common_n4_2_2_, this_.data_resource_uid as data_res5_2_2_, this_.date_created as date_cre6_2_2_, this_.family as family7_2_2_, this_.guid as guid8_2_2_, this_.image_url as image_ur9_2_2_, this_.is_published as is_publ10_2_2_, this_.item_order as item_or11_2_2_, this_.kingdom as kingdom12_2_2_, this_.last_updated as last_up13_2_2_, this_.matched_name as matched14_2_2_, this_.list_id as list_id15_2_2_, this_.raw_scientific_name as raw_sci16_2_2_, kvpvalues3_.species_list_item_kvp_values_id as species_1_3_4_, specieslis4_.id as species_2_3_4_, specieslis4_.id as id1_4_0_, specieslis4_.version as version2_4_0_, specieslis4_.data_resource_uid as data_res3_4_0_, specieslis4_.item_order as item_ord4_4_0_, specieslis4_.the_key as the_key5_4_0_, specieslis4_.value as value6_4_0_, specieslis4_.vocab_value as vocab_va7_4_0_, mylist_ali1_.id as id1_0_1_, mylist_ali1_.version as version2_0_1_, mylist_ali1_.authority as authorit3_0_1_, mylist_ali1_.category as category4_0_1_, mylist_ali1_.data_resource_uid as data_res5_0_1_, mylist_ali1_.date_created as date_cre6_0_1_, mylist_ali1_.description as descript7_0_1_, mylist_ali1_.first_name as first_na8_0_1_, mylist_ali1_.generalisation as generali9_0_1_, mylist_ali1_.is_authoritative as is_auth10_0_1_, mylist_ali1_.isbie as isbie11_0_1_, mylist_ali1_.is_invasive as is_inva12_0_1_, mylist_ali1_.is_private as is_priv13_0_1_, mylist_ali1_.issds as issds14_0_1_, mylist_ali1_.is_threatened as is_thre15_0_1_, mylist_ali1_.last_updated as last_up16_0_1_, mylist_ali1_.list_name as list_na17_0_1_, mylist_ali1_.list_type as list_ty18_0_1_, mylist_ali1_.region as region19_0_1_, mylist_ali1_.sds_type as sds_typ20_0_1_, mylist_ali1_.surname as surname21_0_1_, mylist_ali1_.url as url22_0_1_, mylist_ali1_.user_id as user_id23_0_1_, mylist_ali1_.username as usernam24_0_1_, mylist_ali1_.wkt as wkt25_0_1_, (select count(*) from species_list_item sli where sli.list_id = mylist_ali1_.id) as formula0_1_, concat(mylist_ali1_.first_name, ' ', mylist_ali1_.surname) as formula1_1_ from species_list_item this_ left outer join species_list_item_species_listkvp kvpvalues3_ on this_.id=kvpvalues3_.species_list_item_kvp_values_id left outer join species_listkvp specieslis4_ on kvpvalues3_.species_listkvp_id=specieslis4_.id inner join species_list mylist_ali1_ on this_.list_id=mylist_ali1_.id where ((mylist_ali1_.issds=1) and this_.guid is not null) order by lower(this_.guid) asc limit 2147483647

1:20.441532  seconds, 48143 rows, 121121578 rows examined

@ansell
Copy link
Contributor Author

ansell commented Nov 15, 2017

The following query has an embedded count and an embedded concat, and is passing a large number of records:

select this_.id as id1_2_2_, this_.version as version2_2_2_, this_.author as author3_2_2_, this_.common_name as common_n4_2_2_, this_.data_resource_uid as data_res5_2_2_, this_.date_created as date_cre6_2_2_, this_.family as family7_2_2_, this_.guid as guid8_2_2_, this_.image_url as image_ur9_2_2_, this_.is_published as is_publ10_2_2_, this_.item_order as item_or11_2_2_, this_.kingdom as kingdom12_2_2_, this_.last_updated as last_up13_2_2_, this_.matched_name as matched14_2_2_, this_.list_id as list_id15_2_2_, this_.raw_scientific_name as raw_sci16_2_2_, kvpvalues3_.species_list_item_kvp_values_id as species_1_3_4_, specieslis4_.id as species_2_3_4_, specieslis4_.id as id1_4_0_, specieslis4_.version as version2_4_0_, specieslis4_.data_resource_uid as data_res3_4_0_, specieslis4_.item_order as item_ord4_4_0_, specieslis4_.the_key as the_key5_4_0_, specieslis4_.value as value6_4_0_, specieslis4_.vocab_value as vocab_va7_4_0_, mylist_ali1_.id as id1_0_1_, mylist_ali1_.version as version2_0_1_, mylist_ali1_.authority as authorit3_0_1_, mylist_ali1_.category as category4_0_1_, mylist_ali1_.data_resource_uid as data_res5_0_1_, mylist_ali1_.date_created as date_cre6_0_1_, mylist_ali1_.description as descript7_0_1_, mylist_ali1_.first_name as first_na8_0_1_, mylist_ali1_.generalisation as generali9_0_1_, mylist_ali1_.is_authoritative as is_auth10_0_1_, mylist_ali1_.isbie as isbie11_0_1_, mylist_ali1_.is_invasive as is_inva12_0_1_, mylist_ali1_.is_private as is_priv13_0_1_, mylist_ali1_.issds as issds14_0_1_, mylist_ali1_.is_threatened as is_thre15_0_1_, mylist_ali1_.last_updated as last_up16_0_1_, mylist_ali1_.list_name as list_na17_0_1_, mylist_ali1_.list_type as list_ty18_0_1_, mylist_ali1_.region as region19_0_1_, mylist_ali1_.sds_type as sds_typ20_0_1_, mylist_ali1_.surname as surname21_0_1_, mylist_ali1_.url as url22_0_1_, mylist_ali1_.user_id as user_id23_0_1_, mylist_ali1_.username as usernam24_0_1_, mylist_ali1_.wkt as wkt25_0_1_, (select count(*) from species_list_item sli where sli.list_id = mylist_ali1_.id) as formula0_1_, concat(mylist_ali1_.first_name, ' ', mylist_ali1_.surname) as formula1_1_ from species_list_item this_ left outer join species_list_item_species_listkvp kvpvalues3_ on this_.id=kvpvalues3_.species_list_item_kvp_values_id left outer join species_listkvp specieslis4_ on kvpvalues3_.species_listkvp_id=specieslis4_.id inner join species_list mylist_ali1_ on this_.list_id=mylist_ali1_.id where (this_.guid='urn:lsid:biodiversity.org.au:afd.taxon:68d5fd52-71e7-4729-bf20-db7063f47ad5') limit 2147483647

29.399465 seconds 3839 rows, 25790980 rows examined

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@temi temi

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@ansell @temi