Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

io.netty.util.internal.OutOfDirectMemoryError #7699

Closed
DonnyDarkoRabbit opened this issue Feb 8, 2018 · 17 comments
Closed

io.netty.util.internal.OutOfDirectMemoryError #7699

DonnyDarkoRabbit opened this issue Feb 8, 2018 · 17 comments

Comments

@DonnyDarkoRabbit
Copy link

@DonnyDarkoRabbit DonnyDarkoRabbit commented Feb 8, 2018

Actual behavior

io.netty.util.internal.OutOfDirectMemoryError - failed to allocate 16777216 byte(s) of direct memory (used: 3456106503, max: 3464495104) 200 B (0%) n/a

Netty version

4.1.16.Final

JVM version (e.g. java -version)

1.8.0_152

OS version (e.g. uname -a)

Linux

1
2
3

Four weeks after netty uptime, I'm experiencing CPU spikes that once happen they stay high. Then, netty server fails to accept an amount of 10-20% of incoming connections. I captured the threads that consume the most cpu and I found the exception they were throwing was io.netty.util.internal.OutOfDirectMemoryError.

Also before this I'm experiencing garbage collection issues once a week, system hangs unless I call manually System.gc() from a remote administration panel that uses another netty bootstrap instance for the admin client. Therefore the memory is heavily used but always goes down after full gc.

@DonnyDarkoRabbit DonnyDarkoRabbit changed the title out of memory io.netty.util.internal.OutOfDirectMemoryErro Feb 9, 2018
@DonnyDarkoRabbit DonnyDarkoRabbit changed the title io.netty.util.internal.OutOfDirectMemoryErro io.netty.util.internal.OutOfDirectMemoryError Feb 9, 2018
@normanmaurer

This comment has been minimized.

Copy link
Member

@normanmaurer normanmaurer commented Feb 9, 2018

@DonnyDarkoRabbit like the exception says you are using the maximum of configured direct memory. Why this happens is impossible to tell without more informations. It could be misconfiguration, writing to fast etc. You will need to provide more infos.

The thread dump you are showing just shows that there was an exception happening and you did not handle it in one of you ChannelInboundHandler so it hit the end of the pipeline where we log it.

@normanmaurer

This comment has been minimized.

Copy link
Member

@normanmaurer normanmaurer commented Feb 9, 2018

Also please upgrade to latest netty release (4.1.21.Final).

@DonnyDarkoRabbit

This comment has been minimized.

Copy link
Author

@DonnyDarkoRabbit DonnyDarkoRabbit commented Feb 9, 2018

Im using the following arguments
-server
-XX:MaxDirectMemorySize=3304m
-XX+UseConcMarkSweepGC
-XX:PermSize=128m
-XX:MaxPermSize=128m
-XX:+PrintGC
-XX:+PrintGCDetails
-XX:+PrintGCDateStamps
-Xloggc:gcdetails.log

and

-Dio.netty.recycler.maxCapacity=0
-Dio.netty.recycler.maxCapacity.default=0

initial Java Heap Size

wrapper.java.initmemory=3560

Maximum Java Heap Size (in MB)

wrapper.java.maxmemory=3560

@normanmaurer

This comment has been minimized.

Copy link
Member

@normanmaurer normanmaurer commented Feb 9, 2018

@DonnyDarkoRabbit seems like you used all your "allowed" direct memory. You will need to find out why... I can only guess without more infos, and my first guess would be either you not correctly releasing ByteBufs or you are writing to fast without taking care of back pressure.

@DonnyDarkoRabbit

This comment has been minimized.

Copy link
Author

@DonnyDarkoRabbit DonnyDarkoRabbit commented Feb 9, 2018

Firstly, I found in one of my handlers I use
ChannelInboundHandlerAdapter instead of SimpleChannelInboundHandler and I do not release the channelRead msg object.

Secondly, I use Unpooled.copiedBuffer & Unpooled.wrappedBuffer that I'm not releasing.
eg. ctx.writeAndFlush(Unpooled.copiedBuffer(XML, CharsetUtil.UTF_8));

Do these count as memory leaks?

@normanmaurer

This comment has been minimized.

Copy link
Member

@normanmaurer normanmaurer commented Feb 9, 2018

The first yes if its a ReferenceCounted object the second not if you pass it to write or writeAndFlush

@Scottmitch

This comment has been minimized.

Copy link
Member

@Scottmitch Scottmitch commented Feb 10, 2018

See http://netty.io/wiki/reference-counted-objects.html for more info on reference counting.

@DonnyDarkoRabbit

This comment has been minimized.

Copy link
Author

@DonnyDarkoRabbit DonnyDarkoRabbit commented Feb 10, 2018

I have a few concerns here

@Scottmitch

See http://netty.io/wiki/reference-counted-objects.html for more info on reference counting.

Since Netty version 4, the life cycle of certain objects are managed by their reference counts, so that Netty can return them (or their shared resources) to an object pool (or an object allocator) as soon as it is not used anymore...

So what is the point on reference counting Unpooled buffers?

@normanmaurer

The first yes if its a ReferenceCounted object

Does ChannelInboundHandlerAdapter automatically reference count channelRead's msg Object ? is this a direct memory buffer?

the second not if you pass it to write or writeAndFlush

Is this also if a byteBuf object is used in a Text/BinaryWebSocketFrame for writing?

byte[] bytesArray = ...
ByteBuf byteBuf = Unpooled.wrappedBuffer(bytesArray);
ch.writeAndFlush(new BinaryWebSocketFrame(byteBuf));

Also, I found that both Unpooled.copiedBuffer and Unpooled.wrappedBuffer in my case are reference counted once they constructed by their nature (in AbstractReferenceCountedByteBuf.java) but they are heap based and not direct.

1: Unpooled.copiedBuffer(XML, CharsetUtil.UTF_8)
in this case
Unpooled.java class public static ByteBuf copiedBuffer(CharSequence string, Charset charset) is called
which calls ByteBufUtil.encodeString0(ALLOC, true, buffer, charset, 0);
where true=enforceHeap

2: Unpooled.wrappedBuffer((byte[]) packet)
public static ByteBuf wrappedBuffer(byte[] array) { if (array.length == 0) { return EMPTY_BUFFER; } return new UnpooledHeapByteBuf(ALLOC, array, array.length); }

so both are heap based buffers.

Eventually, there must be 2 leaks

  1. A direct(?) memory leak because I use ChannelInboundHandlerAdapter channelRead instead of SimpleChannelInboundHandler channelRead0 and I do not release the msg object that is automatically ReferenceCounted on channelRead.

  2. A heap memory leak due to reference counting in Unpooled.wrappedBuffer(byte[]...) & Unpooled.copiedBuffer(CharSequence...), since I'm not always writing the created byteBufs to a channel.

However heap usage always goes down to bottom after full gc when all clients disconnect.
And I can't access the private static field PlatformDependent.DIRECT_MEMORY_COUNTER.get() that is used at the io.netty.util.internal.OutOfDirectMemoryError in order to do some tests with direct memory metrics.

@Scottmitch

This comment has been minimized.

Copy link
Member

@Scottmitch Scottmitch commented Feb 12, 2018

So what is the point on reference counting Unpooled buffers?

  1. We may not use the JDK's default allocation mechanisms for performance reasons. This means you may leak direct memory if you are not properly releasing.
  2. If you are conditionally releasing based upon what allocator you start with ... it will be difficult later to switch between pooled/unpooled allocators.

Also, I found that both Unpooled.copiedBuffer and Unpooled.wrappedBuffer in my case are reference counted...

All ByteBuf objects are reference counted. Anything that implements ReferenceCounted is reference counted.

@DonnyDarkoRabbit

This comment has been minimized.

Copy link
Author

@DonnyDarkoRabbit DonnyDarkoRabbit commented Feb 13, 2018

Very useful information, thank you both!

How about running 2 server bootstraps in the same process? How does this affect memory, buffers, pools etc?

because once a week (even in the first day) I get a very strange garbage collection behavior that stays between 80%-99% at heap usage, trying to do stop the world gc again and again... and java is freezing until I connect to the second netty bootstrap! then gc stops freezing immediately and heap goes down to 20-30% (without any further action from me! I was wrong at the initial comment, it's not the System.gc() call that corrects the issue, that was the initial thought, now we found that just by connecting to the 2nd bootstrap happens... )

Just by connecting to the second bootstrap gc finally clears the garbage and get's out from a "freezing the process loop".

Any suggestions ?

As this must be a different issue I post it at #7716

@jo-kin

This comment has been minimized.

Copy link

@jo-kin jo-kin commented Feb 15, 2018

@normanmaurer 6 days ago you mentioned here

or you are writing to fast without taking care of back pressure.

Already in issue #6221 the problem was caused by

writing to fast

Is is possible to explain your comment a little deeper?
If i'm right, Netty in NIO-mode tries to offer a "unlimited" layer above a actually limited layer (socket, network). Netty uses DirectMemory to buffer performance differences between both layers. Means "writing to fast" that the buffer capacity in DirectMemory is overcharged by to much Write&Flush on a Netty context?
How can I balance this? Is there an API, where i can request the current loading and use this information to slow down my writes on Netty?

@normanmaurer

This comment has been minimized.

Copy link
Member

@normanmaurer normanmaurer commented Feb 15, 2018

@jo-kin yes... you may try writing faster then the remote peer accepts.... You can check Channel.isWritable() to see if it is writable atm. ChannelInboundHandler.channelWritabilityChanged(...) will be triggered whenever the writability state of the Channel changes

@jo-kin

This comment has been minimized.

Copy link

@jo-kin jo-kin commented Feb 15, 2018

@normanmaurer Thanks a lot. I was caught exactly in that trap. A writing delay with isWritable() is the solution for my io.netty.util.internal.OutOfDirectMemoryError-Problem.

@normanmaurer

This comment has been minimized.

Copy link
Member

@normanmaurer normanmaurer commented Feb 15, 2018

@jo-kin thanks... please close the issue then :)

@DonnyDarkoRabbit

This comment has been minimized.

Copy link
Author

@DonnyDarkoRabbit DonnyDarkoRabbit commented Feb 15, 2018

I opened this issue :)) I 've done several changes to fix this error that I will be able to confirm by tomorrow.

@DonnyDarkoRabbit

This comment has been minimized.

Copy link
Author

@DonnyDarkoRabbit DonnyDarkoRabbit commented Feb 17, 2018

back pressure control seems like solved the issue for me too. thanks!!

@shufanhao

This comment has been minimized.

Copy link

@shufanhao shufanhao commented Jul 11, 2018

@DonnyDarkoRabbit which solution fix your issue, I also hit this issue.
`018-07-11 04:51:57.377 WARN 27 --- [ntLoopGroup-3-1] io.netty.channel.DefaultChannelPipeline : An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.

io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 16777216 byte(s) of direct memory (used: 520093696, max: 520093696)
at io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:615) ~[netty-common-4.1.12.Final.jar!/:4.1.12.Final]
at io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:569) ~[netty-common-4.1.12.Final.jar!/:4.1.12.Final]
at io.netty.buffer.PoolArena$DirectArena.allocateDirect(PoolArena.java:764) ~[netty-buffer-4.1.12.Final.jar!/:4.1.12.Final]
at io.netty.buffer.PoolArena$DirectArena.newChunk(PoolArena.java:740) ~[netty-buffer-4.1.12.Final.jar!/:4.1.12.Final]
at io.netty.buffer.PoolArena.allocateNormal(PoolArena.java:244) ~[netty-buffer-4.1.12.Final.jar!/:4.1.12.Final]
at io.netty.buffer.PoolArena.allocate(PoolArena.java:214) ~[netty-buffer-4.1.12.Final.jar!/:4.1.12.Final]
at io.netty.buffer.PoolArena.allocate(PoolArena.java:146) ~[netty-buffer-4.1.12.Final.jar!/:4.1.12.Final]
at io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:324) ~[netty-buffer-4.1.12.Final.jar!/:4.1.12.Final]
at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:181) ~[netty-buffer-4.1.12.Final.jar!/:4.1.12.Final]
at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:172) ~[netty-buffer-4.1.12.Final.jar!/:4.1.12.Final]
at io.netty.buffer.AbstractByteBufAllocator.ioBuffer(AbstractByteBufAllocator.java:133) ~[netty-buffer-4.1.12.Final.jar!/:4.1.12.Final]
at io.netty.channel.DefaultMaxMessagesRecvByteBufAllocator$MaxMessageHandle.allocate(DefaultMaxMessagesRecvByteBufAllocator.java:80)`

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
5 participants
You can’t perform that action at this time.