Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Low-level REST client status error:ava.lang.IllegalStateException: Request cannot be executed; I/O reactor status: STOPPED #42133

Open
weizijun opened this issue May 14, 2019 · 3 comments

Comments

Projects
None yet
4 participants
@weizijun
Copy link
Contributor

commented May 14, 2019

Describe the feature:

Elasticsearch version (bin/elasticsearch --version):
6.6.1

Plugins installed: []

JVM version (java -version):
1.8

OS version (uname -a if on a Unix-like system):
3.10.0-514.16.1.es01.x86_64 #1 SMP Wed Oct 17 23:14:35 CST 2018 x86_64 x86_64 x86_64 GNU/Linux

Description of the problem including expected versus actual behavior:
Low-level REST client accidentally casue exception:

java.lang.IllegalStateException: Request cannot be executed; I/O reactor status: STOPPED
	at org.apache.http.util.Asserts.check(Asserts.java:46)
	at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase.ensureRunning(CloseableHttpAsyncClientBase.java:90)
	at org.apache.http.impl.nio.client.InternalHttpAsyncClient.execute(InternalHttpAsyncClient.java:123)
	at org.elasticsearch.client.RestClient.performRequestAsync(RestClient.java:524)
	at org.elasticsearch.client.RestClient.performRequestAsyncNoCatch(RestClient.java:501)
	at org.elasticsearch.client.RestClient.performRequestAsync(RestClient.java:232)
	at com.didichuxing.datachannel.arius.elasticsearch.client.ESClient.doExecute(ESClient.java:133)

client status change to STOPPED from 2 case:
1、CloseableHttpAsyncClientBase.close()
2、CloseableHttpAsyncClientBase construction method

I sure that I don't call close() ,and one case is that :

org.apache.http.nio.reactor.IOReactorException: I/O dispatch worker terminated abnormally
	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:356)
	at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:221)
	at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.OutOfMemoryError: Direct buffer memory
	at java.nio.Bits.reserveMemory(Bits.java:658)
	at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)
	at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311)
	at sun.nio.ch.Util.getTemporaryDirectBuffer(Util.java:174)
	at sun.nio.ch.IOUtil.read(IOUtil.java:195)
	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
	at org.apache.http.impl.nio.codecs.AbstractContentDecoder.readFromChannel(AbstractContentDecoder.java:127)
	at org.apache.http.impl.nio.codecs.LengthDelimitedDecoder.read(LengthDelimitedDecoder.java:84)
	at org.apache.http.nio.util.SimpleInputBuffer.consumeContent(SimpleInputBuffer.java:66)
	at org.elasticsearch.client.HeapBufferedAsyncResponseConsumer.onContentReceived(HeapBufferedAsyncResponseConsumer.java:96)
	at org.apache.http.nio.protocol.AbstractAsyncResponseConsumer.consumeContent(AbstractAsyncResponseConsumer.java:147)
	at org.apache.http.impl.nio.client.MainClientExec.consumeContent(MainClientExec.java:329)
	at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.consumeContent(DefaultClientExchangeHandlerImpl.java:157)
	at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:330)

after remove XX:MaxDirectMemorySize , I face some other case that client status change to STOPPED.

as I issue in #42061 and other user issue in #39946.
Can RestClient add a method to check if client is running? or reset client internal.

@elasticmachine

This comment has been minimized.

Copy link

commented May 14, 2019

@jaymode

This comment has been minimized.

Copy link
Member

commented May 15, 2019

I face some other case that client status change to STOPPED.

What other cases exactly?

The one you've provided (OutOfMemoryError) is not an error that you can be sure to recover from. If you get an OutOfMemoryError then the best course of action is to restart the JVM.

@weizijun

This comment has been minimized.

Copy link
Contributor Author

commented May 16, 2019

I face some other case that client status change to STOPPED.

What other cases exactly?

The one you've provided (OutOfMemoryError) is not an error that you can be sure to recover from. If you get an OutOfMemoryError then the best course of action is to restart the JVM.

Other case is rare appear. AND not find the reason and phenomenon.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.