TrustyAI Consumer fails for very large input + output payloads

[RobGeada]

For very large input + output payloads (on the order of 10s of MBs), TrustyAI fails to consume the payloads with message:

2023-09-28 14:43:23,364 INFO  [org.kie.tru.ser.end.con.ConsumerEndpoint] (executor-thread-0) Received partial output payload from model='yolo-model', id=58-1
2023-09-29 13:09:50,744 ERROR [org.jbo.res.rea.com.cor.AbstractResteasyReactiveContext] (executor-thread-0) Request failed: java.io.IOException: java.io.IOException: java.nio.channels.ClosedChannelException
	at org.jboss.resteasy.reactive.server.vertx.VertxInputStream$VertxBlockingInput.readBlocking(VertxInputStream.java:255)
	at org.jboss.resteasy.reactive.server.vertx.VertxInputStream.readIntoBuffer(VertxInputStream.java:122)
	at org.jboss.resteasy.reactive.server.vertx.VertxInputStream.read(VertxInputStream.java:84)
	at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._loadMore(UTF8StreamJsonParser.java:257)
	at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._loadMoreGuaranteed(UTF8StreamJsonParser.java:2444)
	at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._finishString2(UTF8StreamJsonParser.java:2527)
	at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._finishAndReturnString(UTF8StreamJsonParser.java:2507)
	at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.getText(UTF8StreamJsonParser.java:334)
	at com.fasterxml.jackson.databind.deser.std.StringDeserializer.deserialize(StringDeserializer.java:41)
	at com.fasterxml.jackson.databind.deser.std.StringDeserializer.deserialize(StringDeserializer.java:11)
	at com.fasterxml.jackson.databind.deser.impl.MethodProperty.deserializeAndSet(MethodProperty.java:129)
	at com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:314)
	at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:177)
	at com.fasterxml.jackson.databind.deser.DefaultDeserializationContext.readRootValue(DefaultDeserializationContext.java:323)
	at com.fasterxml.jackson.databind.ObjectReader._bindAndClose(ObjectReader.java:2051)
	at com.fasterxml.jackson.databind.ObjectReader.readValue(ObjectReader.java:1427)
	at io.quarkus.resteasy.reactive.jackson.runtime.serialisers.ServerJacksonMessageBodyReader.doReadFrom(ServerJacksonMessageBodyReader.java:64)
	at io.quarkus.resteasy.reactive.jackson.runtime.serialisers.ServerJacksonMessageBodyReader.readFrom(ServerJacksonMessageBodyReader.java:54)
	at org.jboss.resteasy.reactive.server.handlers.RequestDeserializeHandler.readFrom(RequestDeserializeHandler.java:122)
	at org.jboss.resteasy.reactive.server.handlers.RequestDeserializeHandler.handle(RequestDeserializeHandler.java:81)
	at io.quarkus.resteasy.reactive.server.runtime.QuarkusResteasyReactiveRequestContext.invokeHandler(QuarkusResteasyReactiveRequestContext.java:109)
	at org.jboss.resteasy.reactive.common.core.AbstractResteasyReactiveContext.run(AbstractResteasyReactiveContext.java:142)
	at io.quarkus.vertx.core.runtime.VertxCoreRecorder$14.runWith(VertxCoreRecorder.java:564)
	at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2449)
	at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1478)
	at org.jboss.threads.DelegatingRunnable.run(DelegatingRunnable.java:29)
	at org.jboss.threads.ThreadLocalResettingRunnable.run(ThreadLocalResettingRunnable.java:29)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:833)
	Suppressed: java.io.IOException: java.io.IOException: java.nio.channels.ClosedChannelException
		at org.jboss.resteasy.reactive.server.vertx.VertxInputStream$VertxBlockingInput.readBlocking(VertxInputStream.java:255)
		at org.jboss.resteasy.reactive.server.vertx.VertxInputStream.readIntoBuffer(VertxInputStream.java:122)
		at org.jboss.resteasy.reactive.server.vertx.VertxInputStream.close(VertxInputStream.java:150)
		at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._closeInput(UTF8StreamJsonParser.java:291)
		at com.fasterxml.jackson.core.base.ParserBase.close(ParserBase.java:374)
		at com.fasterxml.jackson.databind.ObjectReader._bindAndClose(ObjectReader.java:2037)
		... 14 more
	Caused by: java.io.IOException: java.nio.channels.ClosedChannelException
		at org.jboss.resteasy.reactive.server.vertx.VertxInputStream$VertxBlockingInput$2.handle(VertxInputStream.java:202)
		at org.jboss.resteasy.reactive.server.vertx.VertxInputStream$VertxBlockingInput$2.handle(VertxInputStream.java:198)
		at io.vertx.core.impl.ContextInternal.dispatch(ContextInternal.java:264)
		at io.vertx.core.http.impl.HttpEventHandler.handleException(HttpEventHandler.java:89)
		at io.vertx.core.http.impl.Http2ServerRequest.notifyException(Http2ServerRequest.java:152)
		at io.vertx.core.http.impl.Http2ServerRequest.handleClose(Http2ServerRequest.java:185)
		at io.vertx.core.impl.EventLoopContext.execute(EventLoopContext.java:76)
		at io.vertx.core.impl.DuplicatedContext.execute(DuplicatedContext.java:153)
		at io.vertx.core.http.impl.VertxHttp2Stream.onClose(VertxHttp2Stream.java:86)
		at io.vertx.core.http.impl.Http2ServerRequest.onClose(Http2ServerRequest.java:174)
		at io.vertx.core.http.impl.Http2ConnectionBase.onStreamClosed(Http2ConnectionBase.java:150)
		at io.vertx.core.http.impl.VertxHttp2ConnectionHandler.onStreamClosed(VertxHttp2ConnectionHandler.java:189)
		at io.netty.handler.codec.http2.DefaultHttp2Connection.notifyClosed(DefaultHttp2Connection.java:357)
		at io.netty.handler.codec.http2.DefaultHttp2Connection$ActiveStreams.removeFromActiveStreams(DefaultHttp2Connection.java:1007)
		at io.netty.handler.codec.http2.DefaultHttp2Connection$ActiveStreams.deactivate(DefaultHttp2Connection.java:963)
		at io.netty.handler.codec.http2.DefaultHttp2Connection$DefaultStream.close(DefaultHttp2Connection.java:515)
		at io.netty.handler.codec.http2.DefaultHttp2Connection.close(DefaultHttp2Connection.java:153)
		at io.netty.handler.codec.http2.Http2ConnectionHandler$BaseDecoder.channelInactive(Http2ConnectionHandler.java:217)
		at io.netty.handler.codec.http2.Http2ConnectionHandler$PrefaceDecoder.channelInactive(Http2ConnectionHandler.java:273)
		at io.netty.handler.codec.http2.Http2ConnectionHandler.channelInactive(Http2ConnectionHandler.java:432)
		at io.vertx.core.http.impl.VertxHttp2ConnectionHandler.channelInactive(VertxHttp2ConnectionHandler.java:150)
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262)
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248)
		at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241)
		at io.netty.channel.ChannelInboundHandlerAdapter.channelInactive(ChannelInboundHandlerAdapter.java:81)
		at io.netty.handler.timeout.IdleStateHandler.channelInactive(IdleStateHandler.java:277)
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262)
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248)
		at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241)
		at io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:401)
		at io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:366)
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262)
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248)
		at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241)
		at io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1405)
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262)
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248)
		at io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:901)
		at io.netty.channel.AbstractChannel$AbstractUnsafe$7.run(AbstractChannel.java:813)
		at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174)
		at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167)
		at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
		at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:566)
		at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
		at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
		... 2 more
	Caused by: java.nio.channels.ClosedChannelException
		... 42 more
	[CIRCULAR REFERENCE:java.io.IOException: java.nio.channels.ClosedChannelException]

Describe the bug
A clear and concise description of what the bug is and how often it occurs

Steps To Reproduce
Steps to reproduce the behavior:

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

Expected behavior
A clear and concise description of what you expected to happen.

Workaround (if any)
Any manual steps that allow you to resolve the issue

TrustyAI version
Please attach TrustyAI version or commit used.

OpenShift Version
(If applicable) Please attach OpenShift version used.
Version #:
Provider (Baremetal, OpenStack, RHV, AWS, OKD, CodeReady Containers, ...):

Additional context
Add any other context about the problem here.

[RobGeada]

Appears to be same issue as:

• Http2Exception: Flow control window exceeded for stream: 0 when sending a ~64 KiB attachment eclipse-vertx/vert.x#4814
• Netty Http2Exception: Flow control window exceeded for stream: 0 when sending a ~64 KiB attachment quarkiverse/quarkus-cxf#973
• Server fails receiving large data over http/2 quarkusio/quarkus#35180

Is fixed in eclipse-vertx/vert.x#4802

This can be verified this by upgrading all of TrustyAI to use Quarkus 3.4.2 (see #390) which contains this fix in the dependent Vertx version. However, we can only use a RH Quarkus version, meaning we cannot use Quarkus 3.4.2 just yet. Examing possibiliity for a Vertx/Quarkus backport.