RestClient is not stack safe

[BBK-PiJ-2012-88]

Describe the feature:

Elasticsearch version (bin/elasticsearch --version): 5., 6.

Plugins installed: []

JVM version (java -version):

OS version (uname -a if on a Unix-like system):

Description of the problem including expected versus actual behavior:

The method performAsyncRequest is not stack safe. Specifically the retry logic can cause StackOverflowErrors.

The error occurs when the cluster is in a bad state and so the client receives many transient exceptions, if there are enough nodes in your cluster you can blow the stack.

Steps to reproduce:

Please include a minimal but complete recreation of the problem, including
(e.g.) index creation, mappings, settings, query etc. The easier you make for
us to reproduce it, the more likely that somebody will take the time to look at it.

Provide logs (if relevant):
java.lang.StackOverflowError: null
at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2333)
at java.io.ObjectInputStream$BlockDataInputStream.readUnsignedShort(ObjectInputStream.java:2816)
at java.io.ObjectInputStream$BlockDataInputStream.readUTF(ObjectInputStream.java:2874)
at java.io.ObjectInputStream.readUTF(ObjectInputStream.java:1073)
at java.io.ObjectStreamClass.readNonProxy(ObjectStreamClass.java:684)
at java.io.ObjectInputStream.readClassDescriptor(ObjectInputStream.java:831)
at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1602)
at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1518)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1774)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2000)
at java.io.ObjectInputStream.defaultReadObject(ObjectInputStream.java:501)
at org.apache.http.impl.auth.RFC2617Scheme.readObject(RFC2617Scheme.java:166)
at sun.reflect.GeneratedMethodAccessor130.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1058)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1900)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371)
at org.apache.http.impl.client.BasicAuthCache.get(BasicAuthCache.java:130)
at org.apache.http.client.protocol.RequestAuthCache.process(RequestAuthCache.java:107)
at org.apache.http.protocol.ImmutableHttpProcessor.process(ImmutableHttpProcessor.java:132)
at org.apache.http.impl.nio.client.MainClientExec.prepareRequest(MainClientExec.java:520)
at org.apache.http.impl.nio.client.MainClientExec.prepare(MainClientExec.java:146)
at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.start(DefaultClientExchangeHandlerImpl.java:128)
at org.apache.http.impl.nio.client.InternalHttpAsyncClient.execute(InternalHttpAsyncClient.java:141)
at org.elasticsearch.client.RestClient.performRequestAsync(RestClient.java:343)
at org.elasticsearch.client.RestClient.access$600(RestClient.java:89)
at org.elasticsearch.client.RestClient$1.retryIfPossible(RestClient.java:393)
at org.elasticsearch.client.RestClient$1.failed(RestClient.java:375)
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:134)
at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.executionFailed(DefaultClientExchangeHandlerImpl.java:101)
at org.apache.http.impl.nio.client.AbstractClientExchangeHandler.failed(AbstractClientExchangeHandler.java:426)
at org.apache.http.impl.nio.client.AbstractClientExchangeHandler.connectionRequestFailed(AbstractClientExchangeHandler.java:348)
at org.apache.http.impl.nio.client.AbstractClientExchangeHandler.access$100(AbstractClientExchangeHandler.java:62)
at org.apache.http.impl.nio.client.AbstractClientExchangeHandler$1.failed(AbstractClientExchangeHandler.java:392)
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:134)

[javanna]

Thanks for opening the issue @BBK-PiJ-2012-88 ! Now that I look at it, I think it may be the same as #25306 . Would you have more info on how to reproduce this?

[BBK-PiJ-2012-88]

Yeah it looks like the same issue. This has only happened to us once. We're not sure what the underlying cause was because the cluster had 100s of nodes in it and I think was in a reasonable state at the time

[javanna]

Thanks for the feedback, closing as duplicate of ##25306