org.eclipse.jetty.http.BadMessageException: 500: No version

[csliu]

I am on Presto 0.143, from time to time, I see some (random) network errors like the following (on worker nodes), anyone has some insight on this issue?

2016-06-16T06:01:22.542Z ERROR async-http-response-55 org.glassfish.jersey.server.ServerRuntime$Responder An I/O error has occurred while writing a response message entity to the container output stream.
org.glassfish.jersey.server.internal.process.MappableException: io.airlift.slice.RuntimeIOException: java.io.IOException: org.eclipse.jetty.http.BadMessageException: 500: No version
at org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundWriteTo(MappableExceptionWrapperInterceptor.java:91)
at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:162)
at org.glassfish.jersey.message.internal.MessageBodyFactory.writeTo(MessageBodyFactory.java:1154)
at org.glassfish.jersey.server.ServerRuntime$Responder.writeResponse(ServerRuntime.java:621)
at org.glassfish.jersey.server.ServerRuntime$Responder.processResponse(ServerRuntime.java:377)
at org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:367)
at org.glassfish.jersey.server.ServerRuntime$AsyncResponder$3.run(ServerRuntime.java:828)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:297)
at org.glassfish.jersey.server.ServerRuntime$AsyncResponder.resume(ServerRuntime.java:858)
at org.glassfish.jersey.server.ServerRuntime$AsyncResponder.resume(ServerRuntime.java:820)
at io.airlift.http.server.AsyncResponseHandler.lambda$toCompletionCallback$3(AsyncResponseHandler.java:101)
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760)
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736)
at java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:442)
at io.airlift.concurrent.BoundedExecutor.executeOrMerge(BoundedExecutor.java:69)
at io.airlift.concurrent.BoundedExecutor.access$000(BoundedExecutor.java:28)
at io.airlift.concurrent.BoundedExecutor$1.run(BoundedExecutor.java:40)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: io.airlift.slice.RuntimeIOException: java.io.IOException: org.eclipse.jetty.http.BadMessageException: 500: No version
at io.airlift.slice.OutputStreamSliceOutput.writeBytes(OutputStreamSliceOutput.java:168)
at com.facebook.presto.spi.block.VariableWidthBlockEncoding.writeBlock(VariableWidthBlockEncoding.java:59)
at com.facebook.presto.block.BlockSerdeUtil.writeBlock(BlockSerdeUtil.java:50)
at com.facebook.presto.block.PagesSerde$PagesWriter.append(PagesSerde.java:83)
at com.facebook.presto.block.PagesSerde.writePages(PagesSerde.java:54)
at com.facebook.presto.block.PagesSerde.writePages(PagesSerde.java:47)
at com.facebook.presto.server.PagesResponseWriter.writeTo(PagesResponseWriter.java:91)
at com.facebook.presto.server.PagesResponseWriter.writeTo(PagesResponseWriter.java:41)
at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.invokeWriteTo(WriterInterceptorExecutor.java:265)
at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.aroundWriteTo(WriterInterceptorExecutor.java:250)
at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:162)
at org.glassfish.jersey.server.internal.JsonWithPaddingInterceptor.aroundWriteTo(JsonWithPaddingInterceptor.java:106)
at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:162)
at org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundWriteTo(MappableExceptionWrapperInterceptor.java:85)
... 24 more
Caused by: java.io.IOException: org.eclipse.jetty.http.BadMessageException: 500: No version
at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:173)
at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:415)
at io.airlift.http.server.TimingFilter$TimedServletOutputStream.write(TimingFilter.java:162)
at org.glassfish.jersey.servlet.internal.ResponseWriter$NonCloseableOutputStreamWrapper.write(ResponseWriter.java:298)
at org.glassfish.jersey.message.internal.CommittingOutputStream.write(CommittingOutputStream.java:233)
at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$UnCloseableOutputStream.write(WriterInterceptorExecutor.java:299)
at io.airlift.slice.CountingOutputStream.write(CountingOutputStream.java:56)
at java.io.DataOutputStream.write(DataOutputStream.java:107)
at io.airlift.slice.LittleEndianDataOutputStream.write(LittleEndianDataOutputStream.java:57)
at io.airlift.slice.Slice.getBytes(Slice.java:546)
at io.airlift.slice.OutputStreamSliceOutput.writeBytes(OutputStreamSliceOutput.java:165)
... 37 more
Caused by: org.eclipse.jetty.http.BadMessageException: 500: No version
at org.eclipse.jetty.http.HttpGenerator.generateResponse(HttpGenerator.java:348)
at org.eclipse.jetty.server.HttpConnection$SendCallback.process(HttpConnection.java:681)
at org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:241)
at org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:224)
at org.eclipse.jetty.server.HttpConnection.send(HttpConnection.java:512)
at org.eclipse.jetty.server.HttpChannel.sendResponse(HttpChannel.java:668)
at org.eclipse.jetty.server.HttpChannel.write(HttpChannel.java:722)
at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:179)
at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:163)
... 47 more

[csliu]

@martint @dain I reported this to Jetty community (jetty/jetty.project#650), they said it's a known issue to Presto/Jetty integration and there is not final fix yet. Does facebook have this issue and how this error is handled? We are running a 100+ node cluster and this issue happens randomly but pretty often, greatly affect query success ratio. Any clue to fix it is appreciated.

[csliu]

cc: @cberner thanks.

[cberner]

I believe we've tracked this down to a bug in Jetty which is fixed in the latest milestone release

[csliu]

@cberner great! so the Presto 0.149 release should have fixed this issue?

[electrum]

It will be in 0.150

[csliu]

Got it, thanks a lot!

[cberner]

I think this is fixed in 0.150. If not, please reopen