ClosedChannelException making RSocket request with invalid credentials

[ianbrandt]

With Spring Boot 2.6.2 through 2.6.11 and @EnableRSocketSecurity, if I make a RSocket message mapping request with invalid credentials, I get a RejectedSetupException with message "Invalid Credentials":

https://github.com/ianbrandt/spring-security-samples/blob/a56fcf2148440f2cdddc8e35f8536accc2c4474e/reactive/rsocket/hello-security/src/integTest/java/example/HelloRSocketApplicationITests.java#L79

When I upgrade to 2.6.12, or any later version through 2.7.10, I get a io.netty.util.IllegalReferenceCountException warning logged, and a ClosedChannelException with null message at the client. Output below. I was expecting the same RejectedSetupException as before 2.6.12.

Reproducer branch with before and after commits: https://github.com/ianbrandt/spring-security-samples/tree/invalid-credentials-error.

2023-04-04 15:03:55.563  WARN 38792 --- [ctor-http-nio-2] io.netty.util.ReferenceCountUtil         : Failed to release a message: CompositeByteBuf(freed, components=2)

io.netty.util.IllegalReferenceCountException: refCnt: 0, decrement: 1
	at io.netty.util.internal.ReferenceCountUpdater.toLiveRealRefCnt(ReferenceCountUpdater.java:83) ~[netty-common-4.1.82.Final.jar:4.1.82.Final]
	at io.netty.util.internal.ReferenceCountUpdater.release(ReferenceCountUpdater.java:147) ~[netty-common-4.1.82.Final.jar:4.1.82.Final]
	at io.netty.buffer.AbstractReferenceCountedByteBuf.release(AbstractReferenceCountedByteBuf.java:101) ~[netty-buffer-4.1.82.Final.jar:4.1.82.Final]
	at io.netty.util.ReferenceCountUtil.release(ReferenceCountUtil.java:90) ~[netty-common-4.1.82.Final.jar:4.1.82.Final]
	at io.netty.util.ReferenceCountUtil.safeRelease(ReferenceCountUtil.java:116) ~[netty-common-4.1.82.Final.jar:4.1.82.Final]
	at io.netty.channel.ChannelOutboundBuffer.remove0(ChannelOutboundBuffer.java:306) ~[netty-transport-4.1.82.Final.jar:4.1.82.Final]
	at io.netty.channel.ChannelOutboundBuffer.failFlushed(ChannelOutboundBuffer.java:660) ~[netty-transport-4.1.82.Final.jar:4.1.82.Final]
	at io.netty.channel.AbstractChannel$AbstractUnsafe.closeOutboundBufferForShutdown(AbstractChannel.java:672) ~[netty-transport-4.1.82.Final.jar:4.1.82.Final]
	at io.netty.channel.AbstractChannel$AbstractUnsafe.shutdownOutput(AbstractChannel.java:666) ~[netty-transport-4.1.82.Final.jar:4.1.82.Final]
	at io.netty.channel.AbstractChannel$AbstractUnsafe.handleWriteError(AbstractChannel.java:953) ~[netty-transport-4.1.82.Final.jar:4.1.82.Final]
	at io.netty.channel.AbstractChannel$AbstractUnsafe.flush0(AbstractChannel.java:933) ~[netty-transport-4.1.82.Final.jar:4.1.82.Final]
	at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.flush0(AbstractNioChannel.java:354) ~[netty-transport-4.1.82.Final.jar:4.1.82.Final]
	at io.netty.channel.AbstractChannel$AbstractUnsafe.flush(AbstractChannel.java:895) ~[netty-transport-4.1.82.Final.jar:4.1.82.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.flush(DefaultChannelPipeline.java:1372) ~[netty-transport-4.1.82.Final.jar:4.1.82.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:750) ~[netty-transport-4.1.82.Final.jar:4.1.82.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:765) ~[netty-transport-4.1.82.Final.jar:4.1.82.Final]
	at io.netty.channel.AbstractChannelHandlerContext$WriteTask.run(AbstractChannelHandlerContext.java:1071) ~[netty-transport-4.1.82.Final.jar:4.1.82.Final]
	at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174) ~[netty-common-4.1.82.Final.jar:4.1.82.Final]
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167) ~[netty-common-4.1.82.Final.jar:4.1.82.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470) ~[netty-common-4.1.82.Final.jar:4.1.82.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:569) ~[netty-transport-4.1.82.Final.jar:4.1.82.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[netty-common-4.1.82.Final.jar:4.1.82.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.82.Final.jar:4.1.82.Final]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.82.Final.jar:4.1.82.Final]
	at java.base/java.lang.Thread.run(Thread.java:833) ~[na:na]

Expecting actual throwable to be an instance of:
  io.rsocket.exceptions.RejectedSetupException
but was:
  reactor.core.Exceptions$ReactiveException: java.nio.channels.ClosedChannelException
	at reactor.core.Exceptions.propagate(Exceptions.java:396)
	at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:97)
	at reactor.core.publisher.Mono.block(Mono.java:1707)

[snicoll]

@ianbrandt thanks for the sample. Looking at the stacktrace and the problem you're experiencing, I don't think this should be reported against this project. I've reproduced the error above. Spring Boot 2.6.12 upgraded to a newer version of RSocket 1.1.3. Downgrading to 1.1.2 (only) makes the problem go away. Please report this against the RSocket project and feel free to link this issue so that we can follow if something is needed on our end.