LLRC direct memory consistent increase , finally program oom

[seantech2000]

Elasticsearch Version

Version 6.8.23

Installed Plugins

No response

Java Version

1.8

OS Version

xx

Problem Description

we use LLRC 6.5.4 version to search ,bulk, index our business data; we find our service will oom after run half month;
our service is use jdk1.8 ,setting -xmx:4096M; use default cms Garbage collection；
from our npm monitor graph,find that, dirct memory is increase consistent;

i use gdb touch one node after oom, and dump memory sgement ,find all is Es responses:

the last behavior is I/O dispatch worker terminated abnormally ,caused by oom

and new request will take error:

how to fix this behavior?

Steps to Reproduce

our service will reporduce after restart

Logs (if relevant)

No response

[elasticsearchmachine]

Pinging @elastic/clients-team (Team:Clients)

[pquentin]

Hello! I'm afraid the data you provided is not enough to characterize the issue. I suspect we're going to need a heap dump to find out what the leak is, but I don't know enough about Java myself to tell.

[swallez]

The IOReactor can terminate when an OOM happens in the callback of an asynchronous request. So the memory leak most certainly happens in your application, and not in LLRC.

Inspecting a heap dump with a profiler will tell you what is using this memory.