Make elasticsearch client request socket timeout to be configurable #6428

buxingzhe · 2021-02-24T01:33:30Z

Please answer these questions before submitting your issue.

Why do you submit this issue?
[] Question or discussion
Bug
Requirement
Feature or performance improvement
Current, we often see the SocketTimeoutException at OAP server side when elasticsearch client executes bulk API.
I tried to changed the source code to set the socket timeout to 60 seconds instead of default 30 seconds, then the issued was resolved. So I hope the ES client request socket timeout to be configurable.

Question

What do you want to know?

Bug

Which version of SkyWalking, OS, and JRE?
v8.3.0, Linux, JDK11
Which company or project?
What happened?
If possible, provide a way to reproduce the error. e.g. demo application, component version.

Requirement or improvement

Please describe your requirements or improvement suggestions.
Make ES client request socket timeout to be configurable.

buxingzhe · 2021-02-24T01:34:33Z

Here is the log.

2021-01-11 06:06:30,964 - org.apache.skywalking.oap.server.telemetry.api.HealthCheckMetrics -17060395 [pool-10-thread-1] ERROR [] - Health check fails
java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-397 [ACTIVE]
at org.elasticsearch.client.RestClient.extractAndWrapCause(RestClient.java:808) ~[elasticsearch-rest-client-7.5.0.jar:7.5.0]
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:248) ~[elasticsearch-rest-client-7.5.0.jar:7.5.0]
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:235) ~[elasticsearch-rest-client-7.5.0.jar:7.5.0]
at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1514) ~[elasticsearch-rest-high-level-client-7.5.0.jar:7.5.0]
at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1484) ~[elasticsearch-rest-high-level-client-7.5.0.jar:7.5.0]
at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1454) ~[elasticsearch-rest-high-level-client-7.5.0.jar:7.5.0]
at org.elasticsearch.client.RestHighLevelClient.bulk(RestHighLevelClient.java:497) ~[elasticsearch-rest-high-level-client-7.5.0.jar:7.5.0]
at org.apache.skywalking.oap.server.storage.plugin.elasticsearch7.client.ElasticSearch7Client.synchronousBulk(ElasticSearch7Client.java:295) [storage-elasticsearch7-plugin-8.3.0.jar:8.3.0]
at org.apache.skywalking.oap.server.storage.plugin.elasticsearch.base.BatchProcessEsDAO.synchronous(BatchProcessEsDAO.java:78) [storage-elasticsearch-plugin-8.3.0.jar:8.3.0]
at org.apache.skywalking.oap.server.core.storage.PersistenceTimer.extractDataAndSave(PersistenceTimer.java:122) [server-core-8.3.0.jar:8.3.0]
at org.apache.skywalking.oap.server.core.storage.PersistenceTimer.lambda$start$0(PersistenceTimer.java:78) [server-core-8.3.0.jar:8.3.0]
at org.apache.skywalking.apm.util.RunnableWithExceptionProtection.run(RunnableWithExceptionProtection.java:33) [apm-util-8.3.0.jar:8.3.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) [?:?]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) [?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
at java.lang.Thread.run(Thread.java:832) [?:?]
Caused by: java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-397 [ACTIVE]
at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.timeout(HttpAsyncRequestExecutor.java:387) ~[httpcore-nio-4.4.12.jar:4.4.12]
at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:92) ~[httpasyncclient-4.1.4.jar:4.1.4]
at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:39) ~[httpasyncclient-4.1.4.jar:4.1.4]
at org.apache.http.impl.nio.reactor.AbstractIODispatch.timeout(AbstractIODispatch.java:175) ~[httpcore-nio-4.4.12.jar:4.4.12]
at org.apache.http.impl.nio.reactor.BaseIOReactor.sessionTimedOut(BaseIOReactor.java:261) ~[httpcore-nio-4.4.12.jar:4.4.12]
at org.apache.http.impl.nio.reactor.AbstractIOReactor.timeoutCheck(AbstractIOReactor.java:502) ~[httpcore-nio-4.4.12.jar:4.4.12]
at org.apache.http.impl.nio.reactor.BaseIOReactor.validate(BaseIOReactor.java:211) ~[httpcore-nio-4.4.12.jar:4.4.12]
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280) ~[httpcore-nio-4.4.12.jar:4.4.12]
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104) ~[httpcore-nio-4.4.12.jar:4.4.12]
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591) ~[httpcore-nio-4.4.12.jar:4.4.12]
... 1 more

wu-sheng · 2021-02-24T01:42:13Z

60s to do what? Seems a crazy latency.

buxingzhe · 2021-02-24T01:50:27Z

Refer to the log below, it should execute inserting metrics data in bulk synchronously. Actually, the max time cost is 32 seconds from logs, not 60 seconds.

at org.apache.skywalking.oap.server.storage.plugin.elasticsearch7.client.ElasticSearch7Client.synchronousBulk(ElasticSearch7Client.java:295) [storage-elasticsearch7-plugin-8.3.0.jar:8.3.0]
at org.apache.skywalking.oap.server.storage.plugin.elasticsearch.base.BatchProcessEsDAO.synchronous(BatchProcessEsDAO.java:78) [storage-elasticsearch-plugin-8.3.0.jar:8.3.0]

wu-sheng · 2021-02-24T02:07:27Z

Oh my, the period of persistent timer is 3s only, you took 32s to make that?

wu-sheng · 2021-02-24T02:08:23Z

I am wondering, is it better to make improvement for the ElasticSearch server?

buxingzhe · 2021-02-24T02:46:04Z

We tuned ES server, but the ES cluster looks good to us. I'm not sure why it runs so slowly.
By the way, if the exception raised, will the data be lost?

wu-sheng · 2021-02-24T03:01:47Z

It will not be lost because of this in one time, but as you are using 30s, it will be in 1min. We set session period aa 1 min.

nisiyong · 2021-02-24T11:13:52Z

Both of you were talking about the writing scenario.
When I just invoked queryBasicTraces with a 2-day duration via Graphql, and it responds as follow:

{
  "data": {},
  "errors": [
    {
      "message": "Exception while fetching data (/trace) : listener timeout after waiting for [30000] ms"
    }
  ]
}

java.io.IOException: listener timeout after waiting for [30000] ms
        at org.elasticsearch.client.RestClient$SyncResponseListener.get(RestClient.java:684) ~[elasticsearch-rest-client-6.3.2.jar:6.3.2]
        at org.elasticsearch.client.RestClient.performRequest(RestClient.java:235) ~[elasticsearch-rest-client-6.3.2.jar:6.3.2]
        at org.elasticsearch.client.RestClient.performRequest(RestClient.java:198) ~[elasticsearch-rest-client-6.3.2.jar:6.3.2]
        at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:522) ~[elasticsearch-rest-high-level-client-6.3.2.jar:6.3.2]
        at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:508) ~[elasticsearch-rest-high-level-client-6.3.2.jar:6.3.2]
        at org.elasticsearch.client.RestHighLevelClient.search(RestHighLevelClient.java:404) ~[elasticsearch-rest-high-level-client-6.3.2.jar:6.3.2]
        at org.apache.skywalking.oap.server.library.client.elasticsearch.ElasticSearchClient.doSearch(ElasticSearchClient.java:351) ~[library-client-8.2.0.jar:8.2.0]
        at org.apache.skywalking.oap.server.library.client.elasticsearch.ElasticSearchClient.search(ElasticSearchClient.java:336) ~[library-client-8.2.0.jar:8.2.0]
        at org.apache.skywalking.oap.server.storage.plugin.elasticsearch.query.TraceQueryEsDAO.queryBasicTraces(TraceQueryEsDAO.java:136) ~[storage-elasticsearch-plugin-8.2.0.jar:8.2.0]
        at org.apache.skywalking.oap.server.core.query.TraceQueryService.queryBasicTraces(TraceQueryService.java:91) ~[server-core-8.2.0.jar:8.2.0]
        at org.apache.skywalking.oap.query.graphql.resolver.TraceQuery.queryBasicTraces(TraceQuery.java:76) ~[query-graphql-plugin-8.2.0.jar:8.2.0]

How about the reading scenario? Could we make the fixed value 30s configurable?
Or just theat this like above said, only improve the ElasticSearch server?

wu-sheng · 2021-02-24T11:20:42Z

If you keep increasing the timeout for es query, you will end in webapp, gateway or browser timeout.😀
We know the performance impact, you need to power your ElasticSearch.
Also, which is why @hanahmily is leading the design of BanyanDB.

nisiyong · 2021-02-24T12:47:17Z

Okay, we focus es performance first. Wish BanyanDB coming.

wu-sheng · 2021-02-25T01:01:57Z

I am going to close this for now

DeadLion · 2021-03-17T03:44:24Z

@buxingzhe @wu-sheng hey guys , i met the same exception, i checked the bulk execution log, 99.99% in 10s completed, 100% in 20s completed , i didn't see over 30s record. any suggestions find out how this happened?
and every about 90s print this error , is this because of healcheck timer set to 90s?

full error log

2021-03-16 06:40:48,214 - org.apache.skywalking.oap.server.telemetry.api.HealthCheckMetrics - 42 [pool-10-thread-1] ERROR [] - Health check fails
java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-22991 [ACTIVE]
        at org.elasticsearch.client.RestClient.extractAndWrapCause(RestClient.java:808) ~[elasticsearch-rest-client-7.5.0.jar:7.5.0]
        at org.elasticsearch.client.RestClient.performRequest(RestClient.java:248) ~[elasticsearch-rest-client-7.5.0.jar:7.5.0]
        at org.elasticsearch.client.RestClient.performRequest(RestClient.java:251) ~[elasticsearch-rest-client-7.5.0.jar:7.5.0]
        at org.elasticsearch.client.RestClient.performRequest(RestClient.java:251) ~[elasticsearch-rest-client-7.5.0.jar:7.5.0]
        at org.elasticsearch.client.RestClient.performRequest(RestClient.java:235) ~[elasticsearch-rest-client-7.5.0.jar:7.5.0]
        at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1514) ~[elasticsearch-rest-high-level-client-7.5.0.jar:7.5.0]
        at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1484) ~[elasticsearch-rest-high-level-client-7.5.0.jar:7.5.0]
        at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1454) ~[elasticsearch-rest-high-level-client-7.5.0.jar:7.5.0]
        at org.elasticsearch.client.RestHighLevelClient.bulk(RestHighLevelClient.java:497) ~[elasticsearch-rest-high-level-client-7.5.0.jar:7.5.0]
        at org.apache.skywalking.oap.server.storage.plugin.elasticsearch7.client.ElasticSearch7Client.synchronousBulk(ElasticSearch7Client.java:295) ~[storage-elasticsearch7-plugin-8.4.0.jar:8.4.0]
        at org.apache.skywalking.oap.server.storage.plugin.elasticsearch.base.BatchProcessEsDAO.synchronous(BatchProcessEsDAO.java:78) ~[storage-elasticsearch-plugin-8.4.0.jar:8.4.0]
        at org.apache.skywalking.oap.server.core.storage.PersistenceTimer.extractDataAndSave(PersistenceTimer.java:122) ~[server-core-8.4.0.jar:8.4.0]
        at org.apache.skywalking.oap.server.core.storage.PersistenceTimer.lambda$start$0(PersistenceTimer.java:78) ~[server-core-8.4.0.jar:8.4.0]
        at org.apache.skywalking.apm.util.RunnableWithExceptionProtection.run(RunnableWithExceptionProtection.java:33) [apm-util-8.4.0.jar:8.4.0]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_242]
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_242]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_242]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_242]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_242]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_242]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_242]
        Suppressed: java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-22983 [ACTIVE]
                at org.elasticsearch.client.RestClient.extractAndWrapCause(RestClient.java:808) ~[elasticsearch-rest-client-7.5.0.jar:7.5.0]
                at org.elasticsearch.client.RestClient.performRequest(RestClient.java:248) ~[elasticsearch-rest-client-7.5.0.jar:7.5.0]
                at org.elasticsearch.client.RestClient.performRequest(RestClient.java:251) ~[elasticsearch-rest-client-7.5.0.jar:7.5.0]
                at org.elasticsearch.client.RestClient.performRequest(RestClient.java:235) ~[elasticsearch-rest-client-7.5.0.jar:7.5.0]
                at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1514) ~[elasticsearch-rest-high-level-client-7.5.0.jar:7.5.0]
                at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1484) ~[elasticsearch-rest-high-level-client-7.5.0.jar:7.5.0]
                at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1454) ~[elasticsearch-rest-high-level-client-7.5.0.jar:7.5.0]
                at org.elasticsearch.client.RestHighLevelClient.bulk(RestHighLevelClient.java:497) ~[elasticsearch-rest-high-level-client-7.5.0.jar:7.5.0]
                at org.apache.skywalking.oap.server.storage.plugin.elasticsearch7.client.ElasticSearch7Client.synchronousBulk(ElasticSearch7Client.java:295) ~[storage-elasticsearch7-plugin-8.4.0.jar:8.4.0]
                at org.apache.skywalking.oap.server.storage.plugin.elasticsearch.base.BatchProcessEsDAO.synchronous(BatchProcessEsDAO.java:78) ~[storage-elasticsearch-plugin-8.4.0.jar:8.4.0]
                at org.apache.skywalking.oap.server.core.storage.PersistenceTimer.extractDataAndSave(PersistenceTimer.java:122) ~[server-core-8.4.0.jar:8.4.0]
                at org.apache.skywalking.oap.server.core.storage.PersistenceTimer.lambda$start$0(PersistenceTimer.java:78) ~[server-core-8.4.0.jar:8.4.0]
                at org.apache.skywalking.apm.util.RunnableWithExceptionProtection.run(RunnableWithExceptionProtection.java:33) [apm-util-8.4.0.jar:8.4.0]
                at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_242]
                at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_242]
                at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_242]
                at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_242]
                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_242]
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_242]
                at java.lang.Thread.run(Thread.java:748) [?:1.8.0_242]
                Suppressed: java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-22989 [ACTIVE]
                        at org.elasticsearch.client.RestClient.extractAndWrapCause(RestClient.java:808) ~[elasticsearch-rest-client-7.5.0.jar:7.5.0]
                        at org.elasticsearch.client.RestClient.performRequest(RestClient.java:248) ~[elasticsearch-rest-client-7.5.0.jar:7.5.0]
                        at org.elasticsearch.client.RestClient.performRequest(RestClient.java:235) ~[elasticsearch-rest-client-7.5.0.jar:7.5.0]
                        at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1514) ~[elasticsearch-rest-high-level-client-7.5.0.jar:7.5.0]
                        at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1484) ~[elasticsearch-rest-high-level-client-7.5.0.jar:7.5.0]
                        at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1454) ~[elasticsearch-rest-high-level-client-7.5.0.jar:7.5.0]
                        at org.elasticsearch.client.RestHighLevelClient.bulk(RestHighLevelClient.java:497) ~[elasticsearch-rest-high-level-client-7.5.0.jar:7.5.0]
                        at org.apache.skywalking.oap.server.storage.plugin.elasticsearch7.client.ElasticSearch7Client.synchronousBulk(ElasticSearch7Client.java:295) ~[storage-elasticsearch7-plugin-8.4.0.jar:8.4.0]
                        at org.apache.skywalking.oap.server.storage.plugin.elasticsearch.base.BatchProcessEsDAO.synchronous(BatchProcessEsDAO.java:78) ~[storage-elasticsearch-plugin-8.4.0.jar:8.4.0]
                        at org.apache.skywalking.oap.server.core.storage.PersistenceTimer.extractDataAndSave(PersistenceTimer.java:122) ~[server-core-8.4.0.jar:8.4.0]
                        at org.apache.skywalking.oap.server.core.storage.PersistenceTimer.lambda$start$0(PersistenceTimer.java:78) ~[server-core-8.4.0.jar:8.4.0]
                        at org.apache.skywalking.apm.util.RunnableWithExceptionProtection.run(RunnableWithExceptionProtection.java:33) [apm-util-8.4.0.jar:8.4.0]
                        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_242]
                        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_242]
                        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_242]
                        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_242]
                        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_242]
                        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_242]
                        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_242]
                Caused by: java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-22989 [ACTIVE]
                        at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.timeout(HttpAsyncRequestExecutor.java:387) ~[httpcore-nio-4.4.12.jar:4.4.12]
                        at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:92) ~[httpasyncclient-4.1.4.jar:4.1.4]
                        at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:39) ~[httpasyncclient-4.1.4.jar:4.1.4]
                        at org.apache.http.impl.nio.reactor.AbstractIODispatch.timeout(AbstractIODispatch.java:175) ~[httpcore-nio-4.4.12.jar:4.4.12]
                        at org.apache.http.impl.nio.reactor.BaseIOReactor.sessionTimedOut(BaseIOReactor.java:261) ~[httpcore-nio-4.4.12.jar:4.4.12]
                        at org.apache.http.impl.nio.reactor.AbstractIOReactor.timeoutCheck(AbstractIOReactor.java:502) ~[httpcore-nio-4.4.12.jar:4.4.12]
                        at org.apache.http.impl.nio.reactor.BaseIOReactor.validate(BaseIOReactor.java:211) ~[httpcore-nio-4.4.12.jar:4.4.12]
                        at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280) ~[httpcore-nio-4.4.12.jar:4.4.12]
                        at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104) ~[httpcore-nio-4.4.12.jar:4.4.12]
                        at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591) ~[httpcore-nio-4.4.12.jar:4.4.12]
                        ... 1 more
        Caused by: java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-22983 [ACTIVE]
                at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.timeout(HttpAsyncRequestExecutor.java:387) ~[httpcore-nio-4.4.12.jar:4.4.12]
                at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:92) ~[httpasyncclient-4.1.4.jar:4.1.4]
                at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:39) ~[httpasyncclient-4.1.4.jar:4.1.4]
                at org.apache.http.impl.nio.reactor.AbstractIODispatch.timeout(AbstractIODispatch.java:175) ~[httpcore-nio-4.4.12.jar:4.4.12]
                at org.apache.http.impl.nio.reactor.BaseIOReactor.sessionTimedOut(BaseIOReactor.java:261) ~[httpcore-nio-4.4.12.jar:4.4.12]
                at org.apache.http.impl.nio.reactor.AbstractIOReactor.timeoutCheck(AbstractIOReactor.java:502) ~[httpcore-nio-4.4.12.jar:4.4.12]
                at org.apache.http.impl.nio.reactor.BaseIOReactor.validate(BaseIOReactor.java:211) ~[httpcore-nio-4.4.12.jar:4.4.12]
                at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280) ~[httpcore-nio-4.4.12.jar:4.4.12]
                at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104) ~[httpcore-nio-4.4.12.jar:4.4.12]
                at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591) ~[httpcore-nio-4.4.12.jar:4.4.12]
                ... 1 more
Caused by: java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-22991 [ACTIVE]
        at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.timeout(HttpAsyncRequestExecutor.java:387) ~[httpcore-nio-4.4.12.jar:4.4.12]
        at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:92) ~[httpasyncclient-4.1.4.jar:4.1.4]
        at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:39) ~[httpasyncclient-4.1.4.jar:4.1.4]
        at org.apache.http.impl.nio.reactor.AbstractIODispatch.timeout(AbstractIODispatch.java:175) ~[httpcore-nio-4.4.12.jar:4.4.12]
        at org.apache.http.impl.nio.reactor.BaseIOReactor.sessionTimedOut(BaseIOReactor.java:261) ~[httpcore-nio-4.4.12.jar:4.4.12]
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.timeoutCheck(AbstractIOReactor.java:502) ~[httpcore-nio-4.4.12.jar:4.4.12]
        at org.apache.http.impl.nio.reactor.BaseIOReactor.validate(BaseIOReactor.java:211) ~[httpcore-nio-4.4.12.jar:4.4.12]
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280) ~[httpcore-nio-4.4.12.jar:4.4.12]
        at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104) ~[httpcore-nio-4.4.12.jar:4.4.12]
        at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591) ~[httpcore-nio-4.4.12.jar:4.4.12]
        ... 1 more

wu-sheng added the question End user question and discussion. label Feb 24, 2021

wu-sheng closed this as completed Feb 25, 2021

wu-sheng added this to the 8.5.0 milestone Feb 25, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make elasticsearch client request socket timeout to be configurable #6428

Make elasticsearch client request socket timeout to be configurable #6428

buxingzhe commented Feb 24, 2021

buxingzhe commented Feb 24, 2021

wu-sheng commented Feb 24, 2021

buxingzhe commented Feb 24, 2021

wu-sheng commented Feb 24, 2021

wu-sheng commented Feb 24, 2021

buxingzhe commented Feb 24, 2021

wu-sheng commented Feb 24, 2021

nisiyong commented Feb 24, 2021

wu-sheng commented Feb 24, 2021

nisiyong commented Feb 24, 2021

wu-sheng commented Feb 25, 2021

DeadLion commented Mar 17, 2021 •

edited

Make elasticsearch client request socket timeout to be configurable #6428

Make elasticsearch client request socket timeout to be configurable #6428

Comments

buxingzhe commented Feb 24, 2021

Question

Bug

Requirement or improvement

buxingzhe commented Feb 24, 2021

wu-sheng commented Feb 24, 2021

buxingzhe commented Feb 24, 2021

wu-sheng commented Feb 24, 2021

wu-sheng commented Feb 24, 2021

buxingzhe commented Feb 24, 2021

wu-sheng commented Feb 24, 2021

nisiyong commented Feb 24, 2021

wu-sheng commented Feb 24, 2021

nisiyong commented Feb 24, 2021

wu-sheng commented Feb 25, 2021

DeadLion commented Mar 17, 2021 • edited

DeadLion commented Mar 17, 2021 •

edited