[BUG] Java off-heap memory overflow (Direct Buffer Memory OOM) error

[Doyle-net]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

Version used:1.7.3
Collector:4
Data collection tasks: 438
After running for a period of time, exceeding 8-9 hours, the following error occurred in the main program.

Java configuration for the main program,Memory allocated: 16GB
-XX:MaxDirectMemorySize=10g -Xms4g -Xmx4g -Dfile.encoding=UTF-8 -Doracle.jdbc.timezoneAsRegion=false --add-opens=java.base/java.nio=org.apache.arrow.memory.core,ALL-UNNAMED

Expected Behavior

No response

Steps To Reproduce

No response

Environment

HertzBeat version(s):1.7.3

Debug logs

2026-01-13 19:40:32.008 [595869159430144-system_data_anomaly-data_list-5347] ERROR org.apache.hertzbeat.collector.dispatch.WorkerPool Line:49 - Thread Name 595869159430144-system_data_anomaly-data_list-5347 : Cannot reserve 4194304 bytes of direct buffer memory (allocated: 10733664086, limit: 10737418240)
java.lang.OutOfMemoryError: Cannot reserve 4194304 bytes of direct buffer memory (allocated: 10733664086, limit: 10737418240)
at java.base/java.nio.Bits.reserveMemory(Bits.java:178)
at java.base/java.nio.DirectByteBuffer.(DirectByteBuffer.java:127)
at java.base/java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:360)
at io.netty.buffer.PoolArena$DirectArena.allocateDirect(PoolArena.java:718)
at io.netty.buffer.PoolArena$DirectArena.newChunk(PoolArena.java:693)
at io.netty.buffer.PoolArena.allocateNormal(PoolArena.java:213)
at io.netty.buffer.PoolArena.tcacheAllocateNormal(PoolArena.java:195)
at io.netty.buffer.PoolArena.allocate(PoolArena.java:137)
at io.netty.buffer.PoolArena.allocate(PoolArena.java:127)
at io.netty.buffer.PooledByteBufAllocatorL$InnerAllocator.newDirectBufferL(PooledByteBufAllocatorL.java:173)
at io.netty.buffer.PooledByteBufAllocatorL$InnerAllocator.directBuffer(PooledByteBufAllocatorL.java:206)
at io.netty.buffer.PooledByteBufAllocatorL.allocate(PooledByteBufAllocatorL.java:53)
at org.apache.arrow.memory.netty.NettyAllocationManager.(NettyAllocationManager.java:74)
at org.apache.arrow.memory.netty.NettyAllocationManager.(NettyAllocationManager.java:81)
at org.apache.arrow.memory.netty.NettyAllocationManager$1.create(NettyAllocationManager.java:38)
at org.apache.arrow.memory.BaseAllocator.newAllocationManager(BaseAllocator.java:376)
at org.apache.arrow.memory.BaseAllocator.newAllocationManager(BaseAllocator.java:372)
at org.apache.arrow.memory.BaseAllocator.bufferWithoutReservation(BaseAllocator.java:357)
at org.apache.arrow.memory.BaseAllocator.buffer(BaseAllocator.java:336)
at org.apache.arrow.memory.RootAllocator.buffer(RootAllocator.java:27)
at org.apache.arrow.memory.BaseAllocator.buffer(BaseAllocator.java:298)
at org.apache.arrow.memory.RootAllocator.buffer(RootAllocator.java:27)
at org.apache.arrow.vector.BaseVariableWidthVector.allocateBytes(BaseVariableWidthVector.java:498)
at org.apache.arrow.vector.BaseVariableWidthVector.allocateNew(BaseVariableWidthVector.java:446)
at org.apache.arrow.vector.BaseVariableWidthVector.allocateNew(BaseVariableWidthVector.java:407)
at org.apache.arrow.vector.VectorSchemaRoot.allocateNew(VectorSchemaRoot.java:150)
at org.apache.hertzbeat.common.entity.message.CollectRep$MetricsData$Builder.build(CollectRep.java:432)
at org.apache.hertzbeat.collector.dispatch.MetricsCollect.validateResponse(MetricsCollect.java:475)
at org.apache.hertzbeat.collector.dispatch.MetricsCollect.run(MetricsCollect.java:224)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at java.base/java.lang.Thread.run(Thread.java:1583)

Anything else?

No response

[Doyle-net]

Thanks for reporting this. Based on the codebase analysis, this appears to be a memory leak in Apache Arrow direct buffer management, likely caused by multiple resource management issues:

Evidence:

• CollectRep.MetricsData.Builder.build() creates a new RootAllocator per call (CollectRep.java:410) but only closes it on exception (line 463), not in the normal path
• MetricsData implements AutoCloseable but most data flow paths (Collector → Queue → Alerter → Storage) don't call close() (CommonDispatcher.java:310)
• RedisMetricsDataCodec creates its own RootAllocator per instance that is never closed (RedisMetricsDataCodec.java:47)

With 438 collection tasks running continuously, even a small per-task leak accumulates to ~10GB over 8-9 hours, matching your error state.

To confirm the diagnosis:

1. Which queue implementation are you using (Redis/Memory/Kafka)?
2. Can you share your complete JVM startup parameters?

Workaround while fixing: Increase -XX:MaxDirectMemorySize to 16g+ and reduce concurrent task count.

Currently using in-memory queues.
Increasing the MaxDirectMemorySize configuration still doesn't solve the problem; analysis of the source code reveals a serious memory leak bug in the queue implementation.

[zqr10159]

This issue appears to have been resolved (though we cannot be entirely certain). Would you be interested in testing our candidate release? If no similar issues arise, it indicates the problem has been fixed.
https://dist.apache.org/repos/dist/dev/hertzbeat/1.8.0-RC2/