java.lang.OutOfMemoryError - Direct buffer memory

[drcrallen]

If using -XX:MaxDirectMemorySize, it is possible to encounter out of memory exceptions in the following ways (there may be others not captured here):

Direct buffer memory
        at java.nio.Bits.reserveMemory(Bits.java:658)
        at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)
        at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311)
        at sun.nio.ch.Util.getTemporaryDirectBuffer(Util.java:174)
        at sun.nio.ch.IOUtil.write(IOUtil.java:58)
        at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471)
        at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:170)
        at org.eclipse.jetty.io.WriteFlusher.flush(WriteFlusher.java:441)
        at org.eclipse.jetty.io.WriteFlusher.completeWrite(WriteFlusher.java:399)
        at org.eclipse.jetty.io.SelectChannelEndPoint.onSelected(SelectChannelEndPoint.java:111)
        at org.eclipse.jetty.io.SelectorManager$ManagedSelector.processKey(SelectorManager.java:636)
        at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:607)
        at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:545)
        at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:620)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:540)
        at java.lang.Thread.run(Thread.java:745)

Drect buffer memory
        at java.nio.Bits.reserveMemory(Bits.java:658)
        at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)
        at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311)
        at sun.nio.ch.Util.getTemporaryDirectBuffer(Util.java:174)
        at sun.nio.ch.IOUtil.read(IOUtil.java:195)
        at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
        at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:68)
        at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:366)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)

Direct buffer memory
        at java.nio.Bits.reserveMemory(Bits.java:658)
        at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)
        at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311)
        at io.druid.segment.CompressedPools$4.get(CompressedPools.java:100)
        at io.druid.segment.CompressedPools$4.get(CompressedPools.java:93)
        at io.druid.collections.StupidPool.take(StupidPool.java:64)
        at io.druid.segment.CompressedPools.getByteBuf(CompressedPools.java:108)
        at io.druid.segment.data.CompressedObjectStrategy.fromByteBuffer(CompressedObjectStrategy.java:286)
        at io.druid.segment.data.CompressedObjectStrategy.fromByteBuffer(CompressedObjectStrategy.java:42)
        at io.druid.segment.data.GenericIndexed$BufferIndexed._get(GenericIndexed.java:217)
        at io.druid.segment.data.GenericIndexed$1.get(GenericIndexed.java:292)
        at io.druid.segment.data.CompressedLongsIndexedSupplier$CompressedIndexedLongs.loadBuffer(CompressedLongsIndexedSupplier.java:340)
        at io.druid.segment.data.CompressedLongsIndexedSupplier$1.get(CompressedLongsIndexedSupplier.java:88)
        at io.druid.segment.column.IndexedLongsGenericColumn.getLongSingleValueRow(IndexedLongsGenericColumn.java:85)
        at io.druid.segment.QueryableIndexStorageAdapter.getMinTime(QueryableIndexStorageAdapter.java:123)
        at io.druid.segment.QueryableIndexStorageAdapter.makeCursors(QueryableIndexStorageAdapter.java:175)
        at io.druid.query.QueryRunnerHelper.makeCursorBasedQuery(QueryRunnerHelper.java:71)
        at io.druid.query.timeseries.TimeseriesQueryEngine.process(TimeseriesQueryEngine.java:47)
        at io.druid.query.timeseries.TimeseriesQueryRunnerFactory$TimeseriesQueryRunner.run(TimeseriesQueryRunnerFactory.java:102)
        at io.druid.query.ReferenceCountingSegmentQueryRunner.run(ReferenceCountingSegmentQueryRunner.java:55)
        at io.druid.query.MetricsEmittingQueryRunner$1.accumulate(MetricsEmittingQueryRunner.java:118)
        at com.metamx.common.guava.MappedSequence.accumulate(MappedSequence.java:40)
        at com.metamx.common.guava.Sequences$1.accumulate(Sequences.java:90)
        at io.druid.query.MetricsEmittingQueryRunner$1.accumulate(MetricsEmittingQueryRunner.java:118)
        at io.druid.query.spec.SpecificSegmentQueryRunner$2$1.call(SpecificSegmentQueryRunner.java:87)
        at io.druid.query.spec.SpecificSegmentQueryRunner.doNamed(SpecificSegmentQueryRunner.java:171)
        at io.druid.query.spec.SpecificSegmentQueryRunner.access$400(SpecificSegmentQueryRunner.java:41)
        at io.druid.query.spec.SpecificSegmentQueryRunner$2.doItNamed(SpecificSegmentQueryRunner.java:162)
        at io.druid.query.spec.SpecificSegmentQueryRunner$2.accumulate(SpecificSegmentQueryRunner.java:80)
        at io.druid.query.CPUTimeMetricQueryRunner$1.accumulate(CPUTimeMetricQueryRunner.java:81)
        at com.metamx.common.guava.Sequences$1.accumulate(Sequences.java:90)
        at com.metamx.common.guava.Sequences.toList(Sequences.java:113)
        at io.druid.query.ChainedExecutionQueryRunner$1$1$1.call(ChainedExecutionQueryRunner.java:129)
        at io.druid.query.ChainedExecutionQueryRunner$1$1$1.call(ChainedExecutionQueryRunner.java:119)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at io.druid.query.PrioritizedListenableFutureTask.run(PrioritizedExecutorService.java:271)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

We have seen a flurry of these very recently, and it is unclear yet if this is a result of something in 0.9.0 or something in some configuration changes on our side (like using a socket appender in log4j2)

[himanshug]

it will happen when you end up allocating total direct BB size beyond what is specified in -XX:MaxDirectMemorySize .
we generally set that to 10 TB so that jvm continues to take up as much direct memory as available on the host. also we use a 64 bits jvm.

may be usage of socket appender increased your direct BB usage.

[drcrallen]

@himanshug that is my leading suspicion right now but I don't have the right kind of metrics to prove it yet.

[drcrallen]

This item is also here to help serve as a search target for anyone in the community who is hitting direct memory errors.

[iNikem]

I have encountered this error during batch ingestion task. And I am pretty sure I have not specified -XX:MaxDirectMemorySize on my middle manager. Does it mean my physical machine is too small to process this task? 16G of ram running manager and 3 peons.

[stale]

This issue has been marked as stale due to 280 days of inactivity. It will be closed in 2 weeks if no further activity occurs. If this issue is still relevant, please simply write any comment. Even if closed, you can still revive the issue at any time or discuss it on the dev@druid.apache.org list. Thank you for your contributions.

[stale]

This issue has been closed due to lack of activity. If you think that is incorrect, or the issue requires additional review, you can revive the issue at any time.