Calling ByteBuf.copy() concurrently from multiple threads will fail

[lhotari]

Expected behavior

Calling .copy() shouldn't mutate the internal state of the ByteBuf instance.
An alternative would be to document this clearly in the Netty javadocs.
The current Javadoc gives the impression that it's safe to call it since it mentions "This method does not modify readerIndex or writerIndex of this buffer."

Actual behavior

Calling .copy() mutates the internal state held in "internalNioBuffer", at least in the case of ReadOnlyByteBufferBuf.
When multiple threads call .copy() concurrently, this leads to various issues such as these stack traces:

java.lang.IllegalArgumentException: newPosition > limit: (2094 > 88)
    at java.base/java.nio.Buffer.createPositionException(Buffer.java:341)
    at java.base/java.nio.Buffer.position(Buffer.java:316)
    at java.base/java.nio.ByteBuffer.position(ByteBuffer.java:1516)
    at java.base/java.nio.HeapByteBuffer.get(HeapByteBuffer.java:185)
    at io.netty.buffer.UnpooledHeapByteBuf.setBytes(UnpooledHeapByteBuf.java:268)
    at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1113)
    at io.netty.buffer.ReadOnlyByteBufferBuf.copy(ReadOnlyByteBufferBuf.java:431)
    at io.netty.buffer.DuplicatedByteBuf.copy(DuplicatedByteBuf.java:210)
    at io.netty.buffer.AbstractByteBuf.copy(AbstractByteBuf.java:1194)
    at org.apache.pulsar.common.protocol.ByteBufPair$CopyingEncoder.write(ByteBufPair.java:149)
    at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:893)
    at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:875)
    at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:984)
    at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:868)
    at org.apache.pulsar.broker.service.PulsarCommandSenderImpl.lambda$sendMessagesToConsumer$1(PulsarCommandSenderImpl.java:277)
    at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173)
    at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166)
    at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
    at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:413)
    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.base/java.lang.Thread.run(Thread.java:840)

java.nio.BufferUnderflowException
    at java.base/java.nio.HeapByteBuffer.get(HeapByteBuffer.java:183)
    at io.netty.buffer.UnpooledHeapByteBuf.setBytes(UnpooledHeapByteBuf.java:268)
    at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1113)
    at io.netty.buffer.ReadOnlyByteBufferBuf.copy(ReadOnlyByteBufferBuf.java:431)
    at io.netty.buffer.DuplicatedByteBuf.copy(DuplicatedByteBuf.java:210)
    at io.netty.buffer.AbstractByteBuf.copy(AbstractByteBuf.java:1194)
    at org.apache.pulsar.common.protocol.ByteBufPair$CopyingEncoder.write(ByteBufPair.java:149)
    at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:893)
    at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:875)
    at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:984)
    at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:868)
    at org.apache.pulsar.broker.service.PulsarCommandSenderImpl.lambda$sendMessagesToConsumer$1(PulsarCommandSenderImpl.java:277)
    at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173)
    at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166)
    at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
    at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:413)
    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.base/java.lang.Thread.run(Thread.java:840)

Steps to reproduce

• Based on the stack traces, it should be possible to reproduce this by calling .copy() concurrently for a ReadOnlyByteBufferBuf instance.

Minimal yet complete reproducer code (or URL to code)

• Not minimal, but this came up in Pulsar bug [Bug] parseMessageMetadata error when broker entry metadata enable with high loading apache/pulsar#22601 . The comments contain a Pulsar system level reproducer. In Pulsar the workaround is [fix][broker] Fix multiple transfer corruption issues when TLS is enabled apache/pulsar#22760 . The workaround uses ByteBuf.nioBuffers()/.nioBuffer() method to make a shallow copy that doesn't face a similar problem as .copy().
  The reason why Pulsar needs a copy is the fact that SslHandler contains a "feature" that it doesn't support shared ByteBuf instances since it uses the internalNioBuffer method which cause issues when ByteBuf instances are shared.

Netty version

4.1.108

JVM version (e.g. java -version)

openjdk version "17.0.11" 2024-04-16
OpenJDK Runtime Environment Temurin-17.0.11+9 (build 17.0.11+9)
OpenJDK 64-Bit Server VM Temurin-17.0.11+9 (build 17.0.11+9, mixed mode, sharing)

OS version (e.g. uname -a)

Linux pop-os 6.8.0-76060800daily20240311-generic #202403110203171407766522.04~4c8e9a0 SMP PREEMPT_DYNAMIC Thu A x86_64 x86_64 x86_64 GNU/Linux

[lhotari]

#14070 is slightly related.

[normanmaurer]

Fixed by #14072