Fixed race condition between write operation and send timeout

[merlimat]

Motivation

There is a possible race condition between a send operation getting written on a socket channel and getting timed-out at the same time. The problem is reproducible by setting very low timeout (eg: 10ms) at sustained publish rate.

The reason of the race condition is that we pass the instance of the OpSendMsg to the io-thread to get the ByteBuf to write into the socket but the timeout is touching it as well. We should instead take a reference on the ByteBuf itself for which we have already a reference increased for the write operation.

This is the seen exception:

2018-01-24 14:24:08,738 - WARN  - [pulsar-client-io-2-2:Slf4JLogger@146] - A task raised an exception. Task: org.apache.pulsar.client.impl.ProducerImpl$WriteInEventLoopCallback@4b2480e1
java.lang.NullPointerException: msg
	at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:785)
	at org.apache.pulsar.client.impl.ProducerImpl$WriteInEventLoopCallback.run(ProducerImpl.java:462)
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463)
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:886)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.lang.Thread.run(Thread.java:745)

[jai1]

Don't we need
ReferenceCountUtil.safeRelease(cmd);

[merlimat]

The buffer gets release when the write operation completes

[rdhabalia]

👍 nice catch.