Caused by: org.apache.http.ContentTooLongException: entity content is too long [107962506] for the configured buffer limit [104857600]

[heimohe]

i meet this problem, but i don’t know how to modify this param... i am using 1.3 verison

[DarioBalinzo]

Hi, thanks for reporting this.

Are you able to share the full stack trace below the error?

Dario

[heimohe]

the error info full stack is ：
2022-11-03 08:43:05,026 ERROR error (com.github.dariobalinzo.task.ElasticSourceTask) [task-thread-elastic-index002-0] java.io.IOException: entity content is too long [107962506] for the configured buffer limit [104857600] at org.elasticsearch.client.RestClient.extractAndWrapCause(RestClient.java:885) at org.elasticsearch.client.RestClient.performRequest(RestClient.java:283) at org.elasticsearch.client.RestClient.performRequest(RestClient.java:270) at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1654) at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1624) at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1594) at org.elasticsearch.client.RestHighLevelClient.search(RestHighLevelClient.java:1110) at com.github.dariobalinzo.elastic.ElasticRepository.executeSearch(ElasticRepository.java:176) at com.github.dariobalinzo.elastic.ElasticRepository.searchAfter(ElasticRepository.java:90) at com.github.dariobalinzo.task.ElasticSourceTask.poll(ElasticSourceTask.java:205) at org.apache.kafka.connect.runtime.WorkerSourceTask.poll(WorkerSourceTask.java:289) at org.apache.kafka.connect.runtime.WorkerSourceTask.execute(WorkerSourceTask.java:256) at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:185) at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:234) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: org.apache.http.ContentTooLongException: entity content is too long [107962506] for the configured buffer limit [104857600] at org.elasticsearch.client.HeapBufferedAsyncResponseConsumer.onEntityEnclosed(HeapBufferedAsyncResponseConsumer.java:76) at org.apache.http.nio.protocol.AbstractAsyncResponseConsumer.responseReceived(AbstractAsyncResponseConsumer.java:137) at org.apache.http.impl.nio.client.MainClientExec.responseReceived(MainClientExec.java:315) at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.responseReceived(DefaultClientExchangeHandlerImpl.java:151) at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.responseReceived(HttpAsyncRequestExecutor.java:315) at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:255) at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81) at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39) at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:114) at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162) at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337) at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315) at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276) at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104) at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591) ... 1 more

[heimohe]

@DarioBalinzo hi , you know how to resolve this problem. i have changed elasticsearch-rest-client and elasticsearch-rest-high-level-client to different version

[heimohe]

i have resolve this problem by modifying source code....

[miltypig]

For anyone else facing this issue it can also be resolved by using the "batch.max.rows" config option for the connector to limit the number of documents received per request. defaults to 10000 and setting this to 5000 solved the issue for me (as seen in last row of config.

{
"name": "some_name",
"config": {
"connector.class":"com.github.dariobalinzo.ElasticSourceConnector",
"errors.retry.timeout":"-1",
"errors.log.include.messages":"true",
"es.host":"x.x.x.x",
"es.scheme": "https",
"tasks.max":"2",
"incrementing.field.name":"@timestamp",
"connection.attempts":"5",
"es.port":"9200",
"topic.prefix":"some",
"name":"some_name",
"value.converter":"org.apache.kafka.connect.json.JsonConverter",
"errors.log.enable":"true",
"index.prefix":"_prefix",
"es.tls.keystore.location": "xxxxx",
"es.tls.keystore.password": "xxxxx",
"es.tls.truststore.location": "xxxxxxx",
"es.tls.truststore.password": "xxxx",
"batch.max.rows": "5000"
}
}