Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

BufferOverflowException with specific buffer sizes #9911

Closed
cilki opened this issue Dec 29, 2019 · 6 comments · Fixed by #9912
Closed

BufferOverflowException with specific buffer sizes #9911

cilki opened this issue Dec 29, 2019 · 6 comments · Fixed by #9912
Assignees
@njhill
Labels
defect
Milestone
4.1.45.Final

Comments

@cilki
Copy link

cilki commented Dec 29, 2019

When Netty reallocates pooled buffers that are approximately 256 to 1024 bytes, a BufferOverflowException can be thrown. Furthermore, the exception is swallowed and is not printed by LoggingHandler (I was only able to find it by capturing it from a debugger).

I'm on JDK 13 and therefore Unsafe isn't available.

Expected behavior

Buffers that are between 256 and 1024 bytes should be reallocated successfully. In my application. it always works when the buffer is outside of that range and always throws when the buffer is inside of the range.

Actual behavior

io.netty.handler.codec.EncoderException: java.nio.BufferOverflowException
        at io.netty.codec@4.1.44.Final/io.netty.handler.codec.MessageToByteEncoder.write(MessageToByteEncoder.java:125)
        at io.netty.transport@4.1.44.Final/io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:715)
        at io.netty.transport@4.1.44.Final/io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:707)
        at io.netty.transport@4.1.44.Final/io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:790)
        at io.netty.transport@4.1.44.Final/io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:700)
        at io.netty.codec@4.1.44.Final/io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:112)
        at io.netty.transport@4.1.44.Final/io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:715)
        at io.netty.transport@4.1.44.Final/io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:707)
        at io.netty.transport@4.1.44.Final/io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:790)
        at io.netty.transport@4.1.44.Final/io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:700)
        at io.netty.handler@4.1.44.Final/io.netty.handler.logging.LoggingHandler.write(LoggingHandler.java:235)
        at io.netty.transport@4.1.44.Final/io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:715)
        at io.netty.transport@4.1.44.Final/io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:762)
        at io.netty.transport@4.1.44.Final/io.netty.channel.AbstractChannelHandlerContext$WriteTask.run(AbstractChannelHandlerContext.java:1089)
        at io.netty.common@4.1.44.Final/io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
        at io.netty.common@4.1.44.Final/io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
        at io.netty.transport@4.1.44.Final/io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
        at io.netty.common@4.1.44.Final/io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
        at io.netty.common@4.1.44.Final/io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at io.netty.common@4.1.44.Final/io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.base/java.lang.Thread.run(Thread.java:830)
Caused by: java.nio.BufferOverflowException
        at java.base/java.nio.DirectByteBuffer.put(DirectByteBuffer.java:411)
        at io.netty.buffer@4.1.44.Final/io.netty.buffer.PoolArena$DirectArena.memoryCopy(PoolArena.java:795)
        at io.netty.buffer@4.1.44.Final/io.netty.buffer.PoolArena$DirectArena.memoryCopy(PoolArena.java:704)
        at io.netty.buffer@4.1.44.Final/io.netty.buffer.PoolArena.reallocate(PoolArena.java:405)
        at io.netty.buffer@4.1.44.Final/io.netty.buffer.PooledByteBuf.capacity(PooledByteBuf.java:118)
        at io.netty.buffer@4.1.44.Final/io.netty.buffer.AbstractByteBuf.ensureWritable0(AbstractByteBuf.java:306)
        at io.netty.buffer@4.1.44.Final/io.netty.buffer.AbstractByteBuf.ensureWritable(AbstractByteBuf.java:282)
        at io.netty.codec@4.1.44.Final/io.netty.handler.codec.protobuf.ProtobufVarint32LengthFieldPrepender.encode(ProtobufVarint32LengthFieldPrepender.java:48)
        at io.netty.codec@4.1.44.Final/io.netty.handler.codec.protobuf.ProtobufVarint32LengthFieldPrepender.encode(ProtobufVarint32LengthFieldPrepender.java:40)
        at io.netty.codec@4.1.44.Final/io.netty.handler.codec.MessageToByteEncoder.write(MessageToByteEncoder.java:107)
        ... 20 more

This leads to messages of certain sizes getting dropped out of the pipeline.

Minimal yet complete reproducer code (or URL to code)

I haven't been able to reproduce outside of my application (and I tried pretty hard). I suspect that the allocation sequence of pooled buffers in my application must be important. I'll try to come up with a way to test this easily in my application.

Netty version

4.1.44.Final

JVM version (e.g. java -version)

openjdk 13.0.1 2019-10-15
OpenJDK Runtime Environment (build 13.0.1+9)
OpenJDK 64-Bit Server VM (build 13.0.1+9, mixed mode)

OS version (e.g. uname -a)

Linux OCTOLOGY 5.4.6-arch1-1 #1 SMP PREEMPT Sat, 21 Dec 2019 16:34:41 +0000 x86_64 GNU/Linux
@cilki
Copy link
Author

cilki commented Dec 29, 2019

Is there any other information I can give that would be helpful?

@njhill
Copy link
Member

njhill commented Dec 29, 2019

@cilki thanks for reporting, is this something that started in 4.1.44.Final?

@cilki
Copy link
Author

cilki commented Dec 29, 2019

@njhill Yes. I just tested 4.1.40.Final through 4.1.43.Final and they all work as expected.

@njhill
Copy link
Member

njhill commented Dec 29, 2019

Thanks @cilki, sorry this is my bad, I should have time to submit a fix later today but you may need to stay on 4.1.43.Final until 4.1.45 is released.

The exception swallowing is likely unrelated but we should probably investigate that separately.

@njhill njhill added the defect label Dec 29, 2019
@cilki
Copy link
Author

cilki commented Dec 29, 2019

Thanks. I can confirm that reverting b0feb5a fixes the issue.

njhill added a commit to njhill/netty that referenced this issue Dec 30, 2019
@njhill
Fix BufferOverflowException during non-Unsafe PooledDirectByteBuf resize
e50fa05 
Motivation

Recent optimization netty#9765 introduced a bug where the native indices of
the internal reused duplicate nio buffer are not properly reset prior to
using it to copy data during a reallocation operation. This can result
in BufferOverflowExceptions thrown during ByteBuf capacity changes.

The code path in question applies only to pooled direct buffers when
Unsafe is disabled or not available.

Modification

Ensure ByteBuffer#clear() is always called on the reused internal nio
buffer prior to returning it from PooledByteBuf#internalNioBuffer()
(protected method); add unit test that exposes the bug.

Result

Fixes netty#9911
@njhill njhill mentioned this issue Dec 30, 2019
Merged
@njhill njhill self-assigned this Dec 30, 2019
@njhill
Copy link
Member

njhill commented Dec 30, 2019

I've opened #9912 to fix the main bug, but haven't looked at the exception swallowing yet.

normanmaurer pushed a commit that referenced this issue Jan 11, 2020
@njhill @normanmaurer
Fix BufferOverflowException during non-Unsafe PooledDirectByteBuf res…
607bc05 
…ize (#9912)


Motivation

Recent optimization #9765 introduced a bug where the native indices of
the internal reused duplicate nio buffer are not properly reset prior to
using it to copy data during a reallocation operation. This can result
in BufferOverflowExceptions thrown during ByteBuf capacity changes.

The code path in question applies only to pooled direct buffers when
Unsafe is disabled or not available.

Modification

Ensure ByteBuffer#clear() is always called on the reused internal nio
buffer prior to returning it from PooledByteBuf#internalNioBuffer()
(protected method); add unit test that exposes the bug.

Result

Fixes #9911
@normanmaurer normanmaurer added this to the 4.1.45.Final milestone Jan 11, 2020
normanmaurer pushed a commit that referenced this issue Jan 11, 2020
@njhill @normanmaurer
Fix BufferOverflowException during non-Unsafe PooledDirectByteBuf res…
727f037 
…ize (#9912)


Motivation

Recent optimization #9765 introduced a bug where the native indices of
the internal reused duplicate nio buffer are not properly reset prior to
using it to copy data during a reallocation operation. This can result
in BufferOverflowExceptions thrown during ByteBuf capacity changes.

The code path in question applies only to pooled direct buffers when
Unsafe is disabled or not available.

Modification

Ensure ByteBuffer#clear() is always called on the reused internal nio
buffer prior to returning it from PooledByteBuf#internalNioBuffer()
(protected method); add unit test that exposes the bug.

Result

Fixes #9911
@VIJAY-SUBRAMANIAM VIJAY-SUBRAMANIAM mentioned this issue Mar 13, 2020
Closed
ihanyong pushed a commit to ihanyong/netty that referenced this issue Jul 31, 2020
@njhill @ihanyong
Fix BufferOverflowException during non-Unsafe PooledDirectByteBuf res…
172337e 
…ize (netty#9912)


Motivation

Recent optimization netty#9765 introduced a bug where the native indices of
the internal reused duplicate nio buffer are not properly reset prior to
using it to copy data during a reallocation operation. This can result
in BufferOverflowExceptions thrown during ByteBuf capacity changes.

The code path in question applies only to pooled direct buffers when
Unsafe is disabled or not available.

Modification

Ensure ByteBuffer#clear() is always called on the reused internal nio
buffer prior to returning it from PooledByteBuf#internalNioBuffer()
(protected method); add unit test that exposes the bug.

Result

Fixes netty#9911
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@njhill njhill

Labels
defect
Projects
None yet
Milestone
4.1.45.Final
Development

Successfully merging a pull request may close this issue.

Fix BufferOverflowException during non-Unsafe PooledDirectByteBuf resize njhill/netty
3 participants
@normanmaurer @cilki @njhill