Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

io.netty.util.internal.OutOfDirectMemoryError #1787

Closed
madper opened this issue Jun 17, 2019 · 11 comments
Closed

io.netty.util.internal.OutOfDirectMemoryError #1787

madper opened this issue Jun 17, 2019 · 11 comments

Comments

@madper
Copy link

madper commented Jun 17, 2019 •
edited

Howdy,

We have about 2000 IoT devices connect to our thingsboard service via mqtt. Each post about 3000 length string every three minutes.
Also we have a cronjob which push shared-attributes to all the devices with a loop.
Additionally, we have RPC to device randomly. Sometimes I trigger the RPC to all the devices. Thus there could be 2000 rpcs running concurrently.

The following error happens every 4 or 5 days. And I have to restart thingsboard.service to workaround it.

2019-06-11 00:22:04,467 [nioEventLoopGroup-5-1] INFO o.t.s.t.mqtt.MqttTransportHandler - [mqtt423227] Processing connect msg for client: 32_IFHEQVKJJZHUET2ELEFA_20190116020078AB5DB- DISKSN! 2019-06-11 00:22:04,466 [nioEventLoopGroup-5-9] ERROR o.t.s.t.mqtt.MqttTransportHandler - [mqtt423393] Unexpected Exception io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 16777216 byte(s) of direct memory (used: 4177526784, max: 4189454336) at io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:640) at io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:594) at io.netty.buffer.PoolArena$DirectArena.allocateDirect(PoolArena.java:764) at io.netty.buffer.PoolArena$DirectArena.newChunk(PoolArena.java:740) at io.netty.buffer.PoolArena.allocateNormal(PoolArena.java:244) at io.netty.buffer.PoolArena.allocate(PoolArena.java:214) at io.netty.buffer.PoolArena.allocate(PoolArena.java:146) at io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:324) at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:185) at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:176) at io.netty.buffer.AbstractByteBufAllocator.ioBuffer(AbstractByteBufAllocator.java:137) at io.netty.channel.DefaultMaxMessagesRecvByteBufAllocator$MaxMessageHandle.allocate(DefaultMaxMessagesRecvByteBufAllocator.java:114) at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:130) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:886) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2019-06-11 00:22:04,467 [nioEventLoopGroup-5-1] WARN o.t.s.t.mqtt.MqttTransportHandler - peer not authenticated 2019-06-11 00:22:04,469 [nioEventLoopGroup-5-4] INFO o.t.s.t.mqtt.MqttTransportHandler - [mqtt423361] Processing connect msg for client: 32_ME2TQMRUGM2TCNI_SCRW18122702G248861FD1F-D ISKSN! 2019-06-11 00:22:04,469 [nioEventLoopGroup-5-4] WARN o.t.s.t.mqtt.MqttTransportHandler - peer not authenticated 2019-06-11 00:22:04,469 [nioEventLoopGroup-5-1] INFO o.t.s.t.mqtt.MqttTransportHandler - [mqtt423229] Processing connect msg for client: 18230088H1010F! 2019-06-11 00:22:04,469 [nioEventLoopGroup-5-1] WARN o.t.s.t.mqtt.MqttTransportHandler - peer not authenticated 2019-06-11 00:22:04,472 [nioEventLoopGroup-5-1] INFO o.t.s.t.mqtt.MqttTransportHandler - [mqtt423228] Processing connect msg for client: 32_IFHEQVKJJZHUET2ELEFA_I49031H005538A6E991- DISKSN!

Forget to mention that we are using v2.0.3 + postgresql(on another server B) + redis(on another server C). And the server has 16G memory with 2 cpu core.
And I just upgrade to the latest v2.3.1 but the memory usage still growing with time.
Please kindly do let me know if there is a fix / workaround for it.
@madper
Copy link
Author

madper commented Jun 19, 2019

I could still reproduce the issue with thingsboard v2.3.1
Screenshot from 2019-06-19 14-43-39

@madper
Copy link
Author

madper commented Jun 21, 2019

After enable netty leak detector I found following logs:

2019-06-21 00:09:58,902 [nioEventLoopGroup-6-4] ERROR io.netty.util.ResourceLeakDetector - LEAK: ByteBuf.release() was not called before it's garbage-collected. See http://netty.io/wiki/reference-counted-objects.html for more information.
Recent access records:
#1:
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:286)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:337)
io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1476)
io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1225)
io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1272)
io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:337)
io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:677)
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:612)
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:529)
io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:491)
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:748)
#2:
io.netty.buffer.AdvancedLeakAwareByteBuf.readUnsignedByte(AdvancedLeakAwareByteBuf.java:406)
io.netty.handler.codec.ReplayingDecoderByteBuf.readUnsignedByte(ReplayingDecoderByteBuf.java:523)
io.netty.handler.codec.mqtt.MqttDecoder.decodeMsbLsb(MqttDecoder.java:457)
io.netty.handler.codec.mqtt.MqttDecoder.decodeMsbLsb(MqttDecoder.java:453)
io.netty.handler.codec.mqtt.MqttDecoder.decodeMessageId(MqttDecoder.java:290)
io.netty.handler.codec.mqtt.MqttDecoder.decodePublishVariableHeader(MqttDecoder.java:280)
io.netty.handler.codec.mqtt.MqttDecoder.decodeVariableHeader(MqttDecoder.java:198)
io.netty.handler.codec.mqtt.MqttDecoder.decode(MqttDecoder.java:85)
io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
io.netty.handler.codec.ReplayingDecoder.callDecode(ReplayingDecoder.java:366)
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:337)
io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1476)
io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1225)
io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1272)
io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:337)
io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:677)
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:612)
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:529)
io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:491)
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:748)
#3:
io.netty.buffer.AdvancedLeakAwareByteBuf.skipBytes(AdvancedLeakAwareByteBuf.java:532)
io.netty.handler.codec.ReplayingDecoderByteBuf.skipBytes(ReplayingDecoderByteBuf.java:850)
io.netty.handler.codec.mqtt.MqttDecoder.decodeString(MqttDecoder.java:439)
io.netty.handler.codec.mqtt.MqttDecoder.decodeString(MqttDecoder.java:426)
io.netty.handler.codec.mqtt.MqttDecoder.decodePublishVariableHeader(MqttDecoder.java:272)
io.netty.handler.codec.mqtt.MqttDecoder.decodeVariableHeader(MqttDecoder.java:198)
io.netty.handler.codec.mqtt.MqttDecoder.decode(MqttDecoder.java:85)
io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
io.netty.handler.codec.ReplayingDecoder.callDecode(ReplayingDecoder.java:366)
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:337)
io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1476)
io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1225)
io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1272)
io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:337)
io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:677)
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:612)
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:529)
io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:491)
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:748)
#4:
io.netty.buffer.AdvancedLeakAwareByteBuf.readUnsignedByte(AdvancedLeakAwareByteBuf.java:406)
io.netty.handler.codec.ReplayingDecoderByteBuf.readUnsignedByte(ReplayingDecoderByteBuf.java:523)
io.netty.handler.codec.mqtt.MqttDecoder.decodeMsbLsb(MqttDecoder.java:458)
io.netty.handler.codec.mqtt.MqttDecoder.decodeMsbLsb(MqttDecoder.java:453)
io.netty.handler.codec.mqtt.MqttDecoder.decodeString(MqttDecoder.java:430)
io.netty.handler.codec.mqtt.MqttDecoder.decodeString(MqttDecoder.java:426)
io.netty.handler.codec.mqtt.MqttDecoder.decodePublishVariableHeader(MqttDecoder.java:272)
io.netty.handler.codec.mqtt.MqttDecoder.decodeVariableHeader(MqttDecoder.java:198)
io.netty.handler.codec.mqtt.MqttDecoder.decode(MqttDecoder.java:85)
io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
io.netty.handler.codec.ReplayingDecoder.callDecode(ReplayingDecoder.java:366)
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:337)
io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1476)
io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1225)
io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1272)
io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:337)
io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:677)
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:612)
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:529)
io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:491)
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:748)
#5:
io.netty.buffer.AdvancedLeakAwareByteBuf.readUnsignedByte(AdvancedLeakAwareByteBuf.java:406)
io.netty.handler.codec.ReplayingDecoderByteBuf.readUnsignedByte(ReplayingDecoderByteBuf.java:523)
io.netty.handler.codec.mqtt.MqttDecoder.decodeMsbLsb(MqttDecoder.java:457)
io.netty.handler.codec.mqtt.MqttDecoder.decodeMsbLsb(MqttDecoder.java:453)
io.netty.handler.codec.mqtt.MqttDecoder.decodeString(MqttDecoder.java:430)
io.netty.handler.codec.mqtt.MqttDecoder.decodeString(MqttDecoder.java:426)
io.netty.handler.codec.mqtt.MqttDecoder.decodePublishVariableHeader(MqttDecoder.java:272)
io.netty.handler.codec.mqtt.MqttDecoder.decodeVariableHeader(MqttDecoder.java:198)
io.netty.handler.codec.mqtt.MqttDecoder.decode(MqttDecoder.java:85)
io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
io.netty.handler.codec.ReplayingDecoder.callDecode(ReplayingDecoder.java:366)
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:337)
io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1476)
io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1225)
io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1272)
io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:337)
io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:677)
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:612)
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:529)
io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:491)
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:748)
#6:
Hint: 'decoder' will handle the message from this point.
io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:116)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:337)
io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1476)
io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1225)
io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1272)
io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:337)
io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:677)
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:612)
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:529)
io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:491)
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:748)
#7:
io.netty.buffer.AdvancedLeakAwareByteBuf.internalNioBuffer(AdvancedLeakAwareByteBuf.java:736)
io.netty.handler.ssl.SslHandler.toByteBuffer(SslHandler.java:1486)
io.netty.handler.ssl.SslHandler.access$300(SslHandler.java:166)
io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:296)
io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1330)
io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1225)
io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1272)
io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:337)
io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:677)
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:612)
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:529)
io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:491)
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:748)
#8:
io.netty.buffer.AdvancedLeakAwareByteBuf.nioBufferCount(AdvancedLeakAwareByteBuf.java:706)
io.netty.handler.ssl.SslHandler.toByteBuffer(SslHandler.java:1486)
io.netty.handler.ssl.SslHandler.access$300(SslHandler.java:166)
io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:296)
io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1330)
io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1225)
io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1272)
io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:337)
io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:677)
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:612)
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:529)
io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:491)
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:748)
Created at:
io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:334)
io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:187)
io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:178)
io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.java:115)
io.netty.handler.ssl.SslHandler.allocate(SslHandler.java:2122)
io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1325)
io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1225)
io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1272)
io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:337)
io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:677)
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:612)
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:529)
io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:491)
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:748)
: 5 leak records were discarded because the leak record count is targeted to 4. Use system property io.netty.leakDetection.targetRecords to increase the limit.

@madper madper mentioned this issue Jul 19, 2019
Closed
@madper
Copy link
Author

madper commented Jul 23, 2019

I noticed there is a upgrade of netty (63beb4c)

I just built the latest master (commit: 224a77f) And I can confirm the issus can be reproduced as well.

@ashvayka
Copy link
Member

Hi @madper please try to re-test, we have made an improvement.

@madper
Copy link
Author

madper commented Jul 24, 2019

Howdy @ashvayka ! Thanks tons for the update! However I could still found the memory leak. I will run two more days to track the memory usage.
Screenshot from 2019-07-24 18-17-34

@madper
Copy link
Author

madper commented Jul 24, 2019

Let me track the memory usage here:
currently we have 3359 connections:
Screenshot from 2019-07-24 18-23-51
and the memory usage is: 5.7%

Screenshot from 2019-07-24 18-23-37

@madper madper mentioned this issue Jul 24, 2019
Closed
ikulikov added a commit that referenced this issue Jul 24, 2019
@ikulikov
Fixed memory leaks in MqttTransportHandler (#1787)
4255bf4
@madper
Copy link
Author

madper commented Jul 25, 2019

Thanks for the fix, I'd test it now.
(P.S. compare to #1787 (comment), currently I have 3480 devices and 9.7% memory footprint. )

@madper
Copy link
Author

madper commented Jul 25, 2019

Hey, It's running!
With 3475 devices, it consumes 4.9% memory. And there is no memory leak in logs so far. Thanks tons!

@madper
Copy link
Author

madper commented Jul 26, 2019

Hey, The netty stopping complaining memory leaks!
With 3593 devices it consumes 9.7% memory now. Looks good to me. Let's keep the issue open for one week to check if I could still reproduce the out of direct memory issues.

And, Thank you for the fix.

@madper
Copy link
Author

madper commented Jul 30, 2019

I would like to close the issue as I can't reproduce the oom anymore. Thanks a lot!

@madper madper closed this as completed Jul 30, 2019
@madper
Copy link
Author

madper commented Aug 2, 2019

After 3 days of running, I could confirm the memory usage is super stable!
Thanks a lot for your fix, @ashvayka !

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@madper @ashvayka