Jetty 9.3.1 throwing ClosedChannelException

[ohshhh]

Using:

• (mbp mid 2015, imac late 2015) with macos sierra (10.12.4)
• google chrome (57.0.2987)
• nginx(1.10.3 with minimal config; installed from homebrew)
• jetty(9.3.1)
• jersey(2.19)

Problem:
~3/4 requests (request type is GET) Jetty throwing ClosedChannelException on large json data return (aprox. 5-6MB).
If inspect response in chrome you can see it's truncated.
If open request url directly to jetty and avoiding nginx then you can see full response.

Can't understand where is the prob.

Tries with no success (same behaviour like on macos):

• deploy project and repeat ubuntu env. in vagrant on macos
• deploy project and repeat ubuntu env. in docker on macos
• add/change/remove nginx different config values (for macos; vagrant; docker nginx config)
• update jetty to latest version
• install nginx from sources

Success try:

• ubuntu 16.04 x64 - all ok with no probs (through nginx and avoiding it).

minimal nginx config:

gzip off;
proxy_buffering off; 
proxy_request_buffering off;

jetty debug log:

[DEBUG] doClose SocketChannelEndPoint@297659ff{/127.0.0.1:51515<->/127.0.0.1:45000,CLOSED,fill=-,flush=-,to=0/0}{io=0/0,kio=0,kro=4}->HttpConnection@3a41355[p=HttpParser{s=CLOSE,0 of 0},g=HttpGenerator@18060b7f{s=START}]=>HttpChannelOverHttp@2aa7efd6{r=1,c=false,a=IDLE,uri=null}
[DEBUG] ignored: WriteFlusher@2590c927{IDLE}->null {}
java.nio.channels.ClosedChannelException
	at org.eclipse.jetty.io.WriteFlusher.onClose(WriteFlusher.java:507)
	at org.eclipse.jetty.io.AbstractEndPoint.onClose(AbstractEndPoint.java:345)
	at org.eclipse.jetty.io.ChannelEndPoint.onClose(ChannelEndPoint.java:222)
	at org.eclipse.jetty.io.AbstractEndPoint.doOnClose(AbstractEndPoint.java:217)
	at org.eclipse.jetty.io.AbstractEndPoint.close(AbstractEndPoint.java:184)
	at org.eclipse.jetty.io.AbstractEndPoint.close(AbstractEndPoint.java:169)
	at org.eclipse.jetty.io.AbstractConnection.close(AbstractConnection.java:222)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:110)
	at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124)
	at org.eclipse.jetty.util.thread.Invocable.invokePreferred(Invocable.java:122)
	at org.eclipse.jetty.util.thread.strategy.ExecutingExecutionStrategy.invoke(ExecutingExecutionStrategy.java:58)
	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:201)
	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:133)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:672)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:590)
	at java.lang.Thread.run(Thread.java:745)
[DEBUG] FillInterest@d4a8ef6{false,null} onClose null
[DEBUG] Queued change org.eclipse.jetty.io.ManagedSelector$$Lambda$65/777465149@67f1ef33 on org.eclipse.jetty.io.ManagedSelector@486fcf46 id=1 keys=1 selected=0

[joakime]

So if I understand this correctly, you use nginx in front of Jetty and the responses are truncated.
But if you don't use nginx, and communicate with Jetty directly, then everything behaves normally.

Sounds like a problem in nginx.
If so, then this is an issue to file at nginx.

[joakime]

Another suggestion, try haproxy, we use it regularly without issue.

[gregw]

@tape88 The ClosedChannelException does suggest that it is nginx that closed the connection. Typically this can be because of some time or size limit.

Does it work with smaller files? What size does it start failing? Is it a size or a time?

Can you turn off timeouts in nginx?

[ohshhh]

@joakime @gregw thanks for your comments.
I started to reconfigure nginx and solved the issue with adding next properties:

keepalive_timeout 60;
proxy_http_version 1.1;
proxy_set_header Connection "";

and removed previous:

proxy_buffering off; 
proxy_request_buffering off;