You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Calling .copy() mutates the internal state held in "internalNioBuffer", at least in the case of ReadOnlyByteBufferBuf.
When multiple threads call .copy() concurrently, this leads to various issues such as these stack traces:
java.lang.IllegalArgumentException: newPosition > limit: (2094 > 88)
at java.base/java.nio.Buffer.createPositionException(Buffer.java:341)
at java.base/java.nio.Buffer.position(Buffer.java:316)
at java.base/java.nio.ByteBuffer.position(ByteBuffer.java:1516)
at java.base/java.nio.HeapByteBuffer.get(HeapByteBuffer.java:185)
at io.netty.buffer.UnpooledHeapByteBuf.setBytes(UnpooledHeapByteBuf.java:268)
at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1113)
at io.netty.buffer.ReadOnlyByteBufferBuf.copy(ReadOnlyByteBufferBuf.java:431)
at io.netty.buffer.DuplicatedByteBuf.copy(DuplicatedByteBuf.java:210)
at io.netty.buffer.AbstractByteBuf.copy(AbstractByteBuf.java:1194)
at org.apache.pulsar.common.protocol.ByteBufPair$CopyingEncoder.write(ByteBufPair.java:149)
at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:893)
at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:875)
at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:984)
at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:868)
at org.apache.pulsar.broker.service.PulsarCommandSenderImpl.lambda$sendMessagesToConsumer$1(PulsarCommandSenderImpl.java:277)
at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:413)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:840)
java.nio.BufferUnderflowException
at java.base/java.nio.HeapByteBuffer.get(HeapByteBuffer.java:183)
at io.netty.buffer.UnpooledHeapByteBuf.setBytes(UnpooledHeapByteBuf.java:268)
at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1113)
at io.netty.buffer.ReadOnlyByteBufferBuf.copy(ReadOnlyByteBufferBuf.java:431)
at io.netty.buffer.DuplicatedByteBuf.copy(DuplicatedByteBuf.java:210)
at io.netty.buffer.AbstractByteBuf.copy(AbstractByteBuf.java:1194)
at org.apache.pulsar.common.protocol.ByteBufPair$CopyingEncoder.write(ByteBufPair.java:149)
at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:893)
at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:875)
at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:984)
at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:868)
at org.apache.pulsar.broker.service.PulsarCommandSenderImpl.lambda$sendMessagesToConsumer$1(PulsarCommandSenderImpl.java:277)
at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:413)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:840)
Steps to reproduce
Based on the stack traces, it should be possible to reproduce this by calling .copy() concurrently for a ReadOnlyByteBufferBuf instance.
Minimal yet complete reproducer code (or URL to code)
Expected behavior
Calling
.copy()
shouldn't mutate the internal state of the ByteBuf instance.An alternative would be to document this clearly in the Netty javadocs.
The current Javadoc gives the impression that it's safe to call it since it mentions "This method does not modify readerIndex or writerIndex of this buffer."
Actual behavior
Calling
.copy()
mutates the internal state held in "internalNioBuffer", at least in the case of ReadOnlyByteBufferBuf.When multiple threads call
.copy()
concurrently, this leads to various issues such as these stack traces:Steps to reproduce
.copy()
concurrently for a ReadOnlyByteBufferBuf instance.Minimal yet complete reproducer code (or URL to code)
The reason why Pulsar needs a copy is the fact that SslHandler contains a "feature" that it doesn't support shared ByteBuf instances since it uses the internalNioBuffer method which cause issues when ByteBuf instances are shared.
Netty version
4.1.108
JVM version (e.g.
java -version
)openjdk version "17.0.11" 2024-04-16
OpenJDK Runtime Environment Temurin-17.0.11+9 (build 17.0.11+9)
OpenJDK 64-Bit Server VM Temurin-17.0.11+9 (build 17.0.11+9, mixed mode, sharing)
OS version (e.g.
uname -a
)Linux pop-os 6.8.0-76060800daily20240311-generic #202403110203
171407766522.04~4c8e9a0 SMP PREEMPT_DYNAMIC Thu A x86_64 x86_64 x86_64 GNU/LinuxThe text was updated successfully, but these errors were encountered: