Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Flowable elasticsearch client crashing with "Request cannot be executed; I/O reactor status: STOPPED" error #3036

Closed
SympathyForTheDev opened this issue Aug 31, 2021 · 6 comments
Closed

Flowable elasticsearch client crashing with "Request cannot be executed; I/O reactor status: STOPPED" error #3036

SympathyForTheDev opened this issue Aug 31, 2021 · 6 comments

Comments

@SympathyForTheDev
Copy link

SympathyForTheDev commented Aug 31, 2021 •
edited
Loading

Describe the bug
Without any specific user action, my flowable is completely crash (all the backend request are failling with Error 500)
And I'm getting this error in the logs


java.lang.RuntimeException: Request cannot be executed; I/O reactor status: STOPPED
	at org.elasticsearch.client.RestClient.extractAndWrapCause(RestClient.java:887) ~[elasticsearch-rest-client-7.10.2.jar:7.10.2]
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:283) ~[elasticsearch-rest-client-7.10.2.jar:7.10.2]
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:270) ~[elasticsearch-rest-client-7.10.2.jar:7.10.2]
	at com.flowable.autoconfigure.indexing.RestClientHttpSender.send(RestClientHttpSender.java:50) ~[flowable-platform-autoconfigure-3.10.1.jar:3.10.1]
	at io.micrometer.core.ipc.http.HttpSender$Request$Builder.send(HttpSender.java:284) ~[micrometer-core-1.7.1.jar:1.7.1]
	at io.micrometer.elastic.ElasticMeterRegistry.publish(ElasticMeterRegistry.java:225) ~[micrometer-registry-elastic-1.7.1.jar:1.7.1]
	at io.micrometer.core.instrument.push.PushMeterRegistry.publishSafely(PushMeterRegistry.java:52) ~[micrometer-core-1.7.1.jar:1.7.1]
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[na:na]
	at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) ~[na:na]
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) ~[na:na]
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[na:na]
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[na:na]
	at java.base/java.lang.Thread.run(Unknown Source) ~[na:na]
Caused by: java.lang.IllegalStateException: Request cannot be executed; I/O reactor status: STOPPED
	at org.apache.http.util.Asserts.check(Asserts.java:46) ~[httpcore-4.4.14.jar:4.4.14]
	at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase.ensureRunning(CloseableHttpAsyncClientBase.java:90) ~[httpasyncclient-4.1.4.jar:4.1.4]
	at org.apache.http.impl.nio.client.InternalHttpAsyncClient.execute(InternalHttpAsyncClient.java:123) ~[httpasyncclient-4.1.4.jar:4.1.4]
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:279) ~[elasticsearch-rest-client-7.10.2.jar:7.10.2]
	... 11 common frames omitted

Expected behavior
Application working

Additional context

I'm using flowable-work 3.10.1, flowable-engine 6.6.2.13, Sprint Boot 2.5.3, elasticsearch 7.10.2

Anyone have already experienced this issue ?
@SympathyForTheDev
Copy link
Author

maybe it's related to elastic/elasticsearch#49124 ?

@SympathyForTheDev SympathyForTheDev changed the title Flowable elasticsearch client crashing with " Request cannot be executed; I/O reactor status: STOPPED" error Flowable elasticsearch client crashing with "Request cannot be executed; I/O reactor status: STOPPED" error Aug 31, 2021
@SympathyForTheDev
Copy link
Author

more investigation on my side, the first exception was a "OutOfMemoryError" and then the elasticsearchclient is crash and never recover then I get always the "IllegalStateException:Request cannot be executed; I/O reactor status: STOPPED"

I will increase my heap size from 2gb, to 8gb (recommended in the documentation)
But I think the flowable should recover if the elasticsearch client is crashing

here the full stacktrace

Exception in thread "I/O dispatcher 3" java.lang.OutOfMemoryError: Direct buffer memory
    at java.base/java.nio.Bits.reserveMemory(Unknown Source)
    at java.base/java.nio.DirectByteBuffer.<init>(Unknown Source)
    at java.base/java.nio.ByteBuffer.allocateDirect(Unknown Source)
    at java.base/sun.nio.ch.Util.getTemporaryDirectBuffer(Unknown Source)
    at java.base/sun.nio.ch.IOUtil.read(Unknown Source)
    at java.base/sun.nio.ch.IOUtil.read(Unknown Source)
    at java.base/sun.nio.ch.SocketChannelImpl.read(Unknown Source)
    at org.apache.http.impl.nio.codecs.AbstractContentDecoder.readFromChannel(AbstractContentDecoder.java:154)
    at org.apache.http.impl.nio.codecs.LengthDelimitedDecoder.read(LengthDelimitedDecoder.java:84)
    at org.apache.http.nio.util.SimpleInputBuffer.consumeContent(SimpleInputBuffer.java:66)
    at org.elasticsearch.client.HeapBufferedAsyncResponseConsumer.onContentReceived(HeapBufferedAsyncResponseConsumer.java:96)
    at org.apache.http.nio.protocol.AbstractAsyncResponseConsumer.consumeContent(AbstractAsyncResponseConsumer.java:147)
    at org.apache.http.impl.nio.client.MainClientExec.consumeContent(MainClientExec.java:329)
    at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.consumeContent(DefaultClientExchangeHandlerImpl.java:157)
    at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:336)
    at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:265)
    at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81)
    at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39)
    at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:114)
    at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)
    at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)
    at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
    at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
    at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
    at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)
    at java.base/java.lang.Thread.run(Unknown Source)
2021-08-31 10:22:27.978 ERROR 1 --- [io-9080-exec-34] o.f.c.r.e.BaseExceptionHandlerAdvice     : Unhandled exception
org.flowable.common.engine.api.FlowableException: Could not query endpoint tasks/_search
    at com.flowable.indexing.ElasticsearchClientImpl.getJsonNode(ElasticsearchClientImpl.java:468) ~[flowable-platform-indexing-3.10.1.jar:3.10.1]
    at com.flowable.indexing.ElasticsearchClientImpl.query(ElasticsearchClientImpl.java:451) ~[flowable-platform-indexing-3.10.1.jar:3.10.1]
    at com.flowable.indexing.SearchServiceImpl.query(SearchServiceImpl.java:53) ~[flowable-platform-indexing-3.10.1.jar:3.10.1]
    at com.flowable.platform.service.index.WorkIndexService.getPageFromQuery(WorkIndexService.java:550) ~[flowable-platform-service-3.10.1.jar:3.10.1]
    at com.flowable.platform.service.index.WorkIndexService.queryWithSafeQuery(WorkIndexService.java:379) ~[flowable-platform-service-3.10.1.jar:3.10.1]
    at com.flowable.platform.service.index.WorkIndexService.fetchData(WorkIndexService.java:334) ~[flowable-platform-service-3.10.1.jar:3.10.1]
    at com.flowable.platform.service.index.WorkIndexService.getInstanceRepresentationPage(WorkIndexService.java:281) ~[flowable-platform-service-3.10.1.jar:3.10.1]
    at com.flowable.platform.service.index.WorkIndexService.findTasksWithQuery(WorkIndexService.java:264) ~[flowable-platform-service-3.10.1.jar:3.10.1]
    at com.flowable.platform.service.index.WorkIndexService.findTasksWithQuery(WorkIndexService.java:259) ~[flowable-platform-service-3.10.1.jar:3.10.1]
    at com.flowable.platform.service.index.WorkIndexService$$FastClassBySpringCGLIB$$3c8b498b.invoke(<generated>) ~[flowable-platform-service-3.10.1.jar:3.10.1]
    at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) ~[spring-core-5.3.8.jar:5.3.8]
    at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:779) ~[spring-aop-5.3.8.jar:5.3.8]
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) ~[spring-aop-5.3.8.jar:5.3.8]
    at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:750) ~[spring-aop-5.3.8.jar:5.3.8]
    at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123) ~[spring-tx-5.3.8.jar:5.3.8]
    at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:388) ~[spring-tx-5.3.8.jar:5.3.8]
    at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119) ~[spring-tx-5.3.8.jar:5.3.8]
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.3.8.jar:5.3.8]
    at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:750) ~[spring-aop-5.3.8.jar:5.3.8]
    at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:692) ~[spring-aop-5.3.8.jar:5.3.8]
    at com.flowable.platform.service.index.WorkIndexService$$EnhancerBySpringCGLIB$$d6ee44db.findTasksWithQuery(<generated>) ~[flowable-platform-service-3.10.1.jar:3.10.1]
    at com.flowable.platform.rest.service.api.task.TasksResource.queryTasksInstances(TasksResource.java:75) ~[flowable-platform-rest-3.10.1.jar:3.10.1]
    at jdk.internal.reflect.GeneratedMethodAccessor2288.invoke(Unknown Source) ~[na:na]
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[na:na]
    at java.base/java.lang.reflect.Method.invoke(Unknown Source) ~[na:na]
    at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:197) ~[spring-web-5.3.8.jar:5.3.8]
    at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:141) ~[spring-web-5.3.8.jar:5.3.8]
    at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:106) ~[spring-webmvc-5.3.8.jar:5.3.8]
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:894) ~[spring-webmvc-5.3.8.jar:5.3.8]
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:808) ~[spring-webmvc-5.3.8.jar:5.3.8]
    at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) ~[spring-webmvc-5.3.8.jar:5.3.8]
    at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1063) ~[spring-webmvc-5.3.8.jar:5.3.8]
    at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:963) ~[spring-webmvc-5.3.8.jar:5.3.8]
    at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006) ~[spring-webmvc-5.3.8.jar:5.3.8]
    at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:898) ~[spring-webmvc-5.3.8.jar:5.3.8]
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:655) ~[tomcat-embed-core-9.0.48.jar:4.0.FR]
    at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883) ~[spring-webmvc-5.3.8.jar:5.3.8]
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:764) ~[tomcat-embed-core-9.0.48.jar:4.0.FR]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:228) ~[tomcat-embed-core-9.0.48.jar:9.0.48]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:163) ~[tomcat-embed-core-9.0.48.jar:9.0.48]
    at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53) ~[tomcat-embed-websocket-9.0.48.jar:9.0.48]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:190) ~[tomcat-embed-core-9.0.48.jar:9.0.48]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:163) ~[tomcat-embed-core-9.0.48.jar:9.0.48]
    at com.azure.spring.autoconfigure.aad.AADAuthenticationFilter.doFilterInternal(AADAuthenticationFilter.java:108) ~[azure-spring-boot-3.3.0.jar:na]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) ~[spring-web-5.3.8.jar:5.3.8]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:190) ~[tomcat-embed-core-9.0.48.jar:9.0.48]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:163) ~[tomcat-embed-core-9.0.48.jar:9.0.48]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:113) ~[spring-web-5.3.8.jar:5.3.8]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:190) ~[tomcat-embed-core-9.0.48.jar:9.0.48]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:163) ~[tomcat-embed-core-9.0.48.jar:9.0.48]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:113) ~[spring-web-5.3.8.jar:5.3.8]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:190) ~[tomcat-embed-core-9.0.48.jar:9.0.48]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:163) ~[tomcat-embed-core-9.0.48.jar:9.0.48]
    at org.springframework.web.servlet.resource.ResourceUrlEncodingFilter.doFilter(ResourceUrlEncodingFilter.java:67) ~[spring-webmvc-5.3.8.jar:5.3.8]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:190) ~[tomcat-embed-core-9.0.48.jar:9.0.48]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:163) ~[tomcat-embed-core-9.0.48.jar:9.0.48]
    at com.flowable.autoconfigure.license.UserRequestsLoggingFilter.doFilterInternal(UserRequestsLoggingFilter.java:37) ~[flowable-platform-autoconfigure-3.10.1.jar:3.10.1]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) ~[spring-web-5.3.8.jar:5.3.8]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:190) ~[tomcat-embed-core-9.0.48.jar:9.0.48]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:163) ~[tomcat-embed-core-9.0.48.jar:9.0.48]
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:327) ~[spring-security-web-5.5.1.jar:5.5.1]
    at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:115) ~[spring-security-web-5.5.1.jar:5.5.1]
    at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:81) ~[spring-security-web-5.5.1.jar:5.5.1]
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336) ~[spring-security-web-5.5.1.jar:5.5.1]
    at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:121) ~[spring-security-web-...

@jbarrez
Copy link
Contributor

jbarrez commented Aug 31, 2021

This is the Flowable open-source github project. This problem relates to Flowable Work, which is commercial (see the com.flowable. package names in the stacktrace) and not related to this repository.

Please open a forum post in the commercial forum (https://forum.flowable.com/) or through the Flowable Service Desk.

@jbarrez jbarrez closed this as completed Aug 31, 2021
@SympathyForTheDev
Copy link
Author

SympathyForTheDev commented Aug 31, 2021 •
edited
Loading

@jbarrez, just for my information, flowable-engine is not using the same implementation of the elasticsearch client ?

@jbarrez
Copy link
Contributor

jbarrez commented Aug 31, 2021

No, the Flowable open-source project doesn't use Elasticsearch nor the Elasticsearch client.

@SympathyForTheDev
Copy link
Author

@jbarrez ok thanks, it was not obvious for me

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@jbarrez @SympathyForTheDev