Worker node org.glassfish.jersey.server.internal.process.MappableException: java.io.UncheckedIOException: java.io.IOException

[palaniappa]

Hi Folks,
In out prod Trino EMR clusters we see queries failing with timeout error. On looking further we saw that one of the worker node cpu is at 100% during that time. The server.log file has the below exception. this exception lasts for 15 to 30mins and then gets corrected automatically. During this time the queries do not complete and fail. Any pointers on why this is happening and how we can avoid it?

2021-10-21T00:09:35.100Z	ERROR	async-http-response-55436	org.glassfish.jersey.server.ServerRuntime$Responder	An I/O error has occurred while writing a response message entity to the container output stream.
org.glassfish.jersey.server.internal.process.MappableException: java.io.UncheckedIOException: java.io.IOException: Close org.eclipse.jetty.server.HttpConnection$SendCallback@5113346c[PROCESSING][i=null,cb=org.eclipse.jetty.server.HttpChannel$SendCallback@53e02146] in state PROCESSING
	at org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundWriteTo(MappableExceptionWrapperInterceptor.java:91)
	at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:163)
	at org.glassfish.jersey.message.internal.MessageBodyFactory.writeTo(MessageBodyFactory.java:1135)
	at org.glassfish.jersey.server.ServerRuntime$Responder.writeResponse(ServerRuntime.java:662)
	at org.glassfish.jersey.server.ServerRuntime$Responder.processResponse(ServerRuntime.java:395)
	at org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:385)
	at org.glassfish.jersey.server.ServerRuntime$AsyncResponder$3.run(ServerRuntime.java:884)
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272)
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:316)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:298)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:268)
	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289)
	at org.glassfish.jersey.server.ServerRuntime$AsyncResponder.resume(ServerRuntime.java:916)
	at org.glassfish.jersey.server.ServerRuntime$AsyncResponder.resume(ServerRuntime.java:872)
	at io.airlift.jaxrs.AsyncResponseHandler$1.onSuccess(AsyncResponseHandler.java:102)
	at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1089)
	at io.airlift.concurrent.BoundedExecutor.drainQueue(BoundedExecutor.java:80)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.io.UncheckedIOException: java.io.IOException: Close org.eclipse.jetty.server.HttpConnection$SendCallback@5113346c[PROCESSING][i=null,cb=org.eclipse.jetty.server.HttpChannel$SendCallback@53e02146] in state PROCESSING
	at io.airlift.slice.OutputStreamSliceOutput.writeToOutputStream(OutputStreamSliceOutput.java:363)
	at io.airlift.slice.OutputStreamSliceOutput.writeBytes(OutputStreamSliceOutput.java:186)
	at io.airlift.slice.OutputStreamSliceOutput.writeBytes(OutputStreamSliceOutput.java:177)
	at io.prestosql.execution.buffer.PagesSerdeUtil.writeSerializedPage(PagesSerdeUtil.java:71)
	at io.prestosql.execution.buffer.PagesSerdeUtil.writeSerializedPages(PagesSerdeUtil.java:100)
	at io.prestosql.server.PagesResponseWriter.writeTo(PagesResponseWriter.java:102)
	at io.prestosql.server.PagesResponseWriter.writeTo(PagesResponseWriter.java:45)
	at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.invokeWriteTo(WriterInterceptorExecutor.java:266)
	at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.aroundWriteTo(WriterInterceptorExecutor.java:251)
	at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:163)
	at org.glassfish.jersey.server.internal.JsonWithPaddingInterceptor.aroundWriteTo(JsonWithPaddingInterceptor.java:109)
	at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:163)
	at org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundWriteTo(MappableExceptionWrapperInterceptor.java:85)
	... 20 more
Caused by: java.io.IOException: Close org.eclipse.jetty.server.HttpConnection$SendCallback@5113346c[PROCESSING][i=null,cb=org.eclipse.jetty.server.HttpChannel$SendCallback@53e02146] in state PROCESSING
	at org.eclipse.jetty.util.IteratingCallback.close(IteratingCallback.java:428)
	at org.eclipse.jetty.server.HttpConnection.onClose(HttpConnection.java:512)
	at org.eclipse.jetty.io.ssl.SslConnection.onClose(SslConnection.java:366)
	at org.eclipse.jetty.io.SelectorManager.connectionClosed(SelectorManager.java:345)
	at org.eclipse.jetty.io.ManagedSelector$DestroyEndPoint.run(ManagedSelector.java:1062)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
	... 1 more