Missing final empty chunk when closing connection

[cconroy]

When Jetty is doing chunked encoding and the client sets the request header Connection: close, Jetty writes out the response data and terminates the connection, but it fails to write the final empty chunk to signal the termination of the chunked response.

The missing data in hex is 30 0d 0a 0d 0a for the terminating 0\r\n\r\n

Some clients (e.g. curl) appear to be okay with this and simply assume that whatever content they have received until the closed connection is the complete response. However, a client should not accept such a response: it has no way of knowing if the response completed successfully or if the connection was prematurely terminated.

I discovered this issue on Jetty 9.2.15.v20160210

[gregw]

It is very strange that jetty is using a chunked response if there is a Connection:close. There is no need to chunk of the message is EOF terminated?

Are you somehow forcing chunking in the response?

[cconroy]

I am using a ServletOutputStream to write the response data which is probably forcing chunking.

I would say that chunking and connection keep alive are orthogonal issues.

[gregw]

ServletOutputStream does not force chunking. The issues are not orthogonal because jetty at the time it commits a response has to consider how it is going to frame the response body and it has several possibilities:

• EOF
• Content-Length
• Chunking

Jetty should not pick chunking if it knows that the response is Connection:close, because that means it is EOF framed and thus there is no need to go to the effort of chunking.

Can you send the headers of your request and response and I will see if I can reproduce.

[cconroy]

The request is setting the Connection: close header: it has nothing to do with the actual contents of the response. This is just a signal that the client doesn't want to keep the connection open when it is finished. Regardless, there are also cases where a server may want to close a connection but it doesn't know the length (e.g. it is currently gracefully draining connections in order to be taken out of rotation from a load balancer).

I am able to reproduce with a simple get request setting only the Connection: close (and implicitly, the Host) header using curl and a packet capture. On the server side, this is a simple static resource servlet that writes a StreamingOutput to the response ServletOutputStream.

[gregw]

A request with Connection:close will cause a response to be created also with Connection:close. A response with Connection:close should not be chunked.

I've created a simple server to demonstrate this by sending two requests, one is Connection:close and the other is not. It produces the output:

HTTP/1.1 200 OK
Date: Wed, 03 Aug 2016 22:09:07 GMT
Content-Type: text/plain; charset=UTF-8
Transfer-Encoding: chunked
Server: Jetty(9.2.z-SNAPSHOT)

13
How now brown cow

25
Now is the winter of our discontent

24
The moon is blue to a fish in love

0

HTTP/1.1 200 OK
Date: Wed, 03 Aug 2016 22:09:09 GMT
Content-Type: text/plain; charset=UTF-8
Connection: close
Server: Jetty(9.2.z-SNAPSHOT)

How now brown cow
Now is the winter of our discontent
The moon is blue to a fish in love

First response is correctly chunked (and terminated). The second response is not chunked and does not need termination.

So I cannot see how you are creating a response that is both closed and chunked.

[cconroy]

In an earlier comment, you noted that the request will be one of EOF, Content-Length, Chunking. However, this is not an accurate classification: there is no EOF in HTTP. The choice is only content-length vs chunking.

I can find no mention of any relationship between non-persistent connections and restrictions on chunking in the spec, and as I noted above, there are definitely valid use cases for chunking a non-persistent connection.

I'm not sure what's forcing the chunking. I'll dig a bit to see if I can find the responsible configuration/filter to assist in reproducing.

[gregw]

@cconroy let me assure you that EOF is and has always been a valid message framing in HTTP. One of the first issue I encountered writing Jetty 20 years ago was that java 0.9 did not have an explicit socket close API! So trust me on this one! or if not then at least check the RFCs:

• RFC1945
• RFC2616 point 5
• RFC 7230 point 7

But it is also true that it is not illegal to send a chunk response AND connection:close, it is just that left to its own devices, jetty will never do that. So unless I can reproduce, I can't see why we may be incorrectly leaving out the end-chunk.

Are you explicitly setting the transfer-encoding? If so, that's a bad thing to do as HTTP 0.9, 1.0 and 2.0 all do not support chunking, so your application should not be messing with those headers.

[cconroy]

If I read RFC 7230 correctly, the presence of a chunked transfer encoding (point 3) takes precedence over the possibility of the server closing the connection (point 7).

In general, it seems bad to use a server close of the connection to terminate the response: the client cannot distinguish between the server intentionally closing the connection vs a network or intermediate proxy terminating the connection. So, even if it were technically valid to close the connection in this case, it is strictly better to first send the final chunk (5 bytes) so that the client can be sure it has received the entire response.

FWIW, This application only supports HTTP 1.1 clients. I am not explicitly setting the transfer-encoding anywhere in the application. I'll ping this thread once I figure out what is triggering the chunking.

[gregw]

@cconroy I'm not disagreeing that chunking is a better framing mechanism than Connection:close. Nor am I disagreeing that if chunking and Connection:close are used together that the end chunk must be sent.

But in this case, the client has requested to use Connection:close, so Jetty validly chooses to not chunk, as was the norm for many years before HTTP/1.1 was available.

So there may well be a bug in jetty here, but you have to throw me a bone and tell me how you are creating the situation that causes it. From what you have to told me, jetty should not even be chunking, so you are not telling me something. How about taking my simple server pasted above and updating it so you can reproduce the problem.

[cconroy]

@gregw I tried to reproduce with your example but was unable to. So far I haven't been able to find the culprit in the stack of my application. The use of chunking does appear to be related to size: smaller responses are getting a Content-Length instead of getting chunked. However, I'm not able to trigger this with your SimpleServer.

[gregw]

So smaller responses fit within Jetty's aggregation buffer, so once the output stream is closed or the response completed, it can look at that buffer and set a content-length. If the response is committed because the aggregate buffer overflows (or from an explicit flush), then jetty has to commit the headers before it has seen all the content and thus cannot determine a content-length.

So that part sounds like normal jetty behaviour, but for some reason in both cases it is acting like there is no Connection:close. Can you capture the headers of your request and response that are causing the problem on your server.

[gregw]

@cconroy any progress on working out how to reproduce this? I've tried explicitly setting transfer-encoding or forcing Connection:close when the client does not ask for it, but I cannot get Jetty to send a chunked response with Connection:close... at least not with 9.3.x

So I'm closing for now, please reopen if you can reproduce.

[cconroy]

@gregw I was unable to get the simple server to reproduce, but unfortunately I also could not track down a root cause in my application. Thanks you for looking into this. I will try a few more things and hopefully can track down the culprit.

[joakime]

Is your jetty being fronted by something? A proxy? a load balancer? nginx? haproxy?

Just curious if something else is adding the Connection: close outside of Jetty's knowledge.

[minaguib]

@gregw Perhaps this is entering new-ticket territory, but I'd like to ask (since we were bitten by this)

As @cconroy mentioned, a client that cares about data integrity and wants to validate that it received a complete (non-truncated) response needs either a Content-Length header or proper HTTP chunking with the signature last-chunk-empty (logical end-of-transmission).

HTTP TCP connection closure alone, without one of the above two, does not offer completeness guarantees. This is independent of:

• The client sending Connection: close or not
• The server sending Connection: close or not
• The server honoring the above or not

Would it not make sense for jetty to adopt the safer behavior ? If a client declares it's HTTP/1.1 (supports chunking), then streamed responses (no known content-length) should chunk, irrespective of keepalive behavior

[joakime]

@minaguib you are misinterpreting the HTTP specs, both the older RFC2616 and the updates in RFC7230.

Some relevant sections in RFC2616:

https://tools.ietf.org/html/rfc2616#section-3.6

3.6 Transfer Codings

Whenever a transfer-coding is applied to a message-body, the set of
transfer-codings MUST include "chunked", unless the message is
terminated by closing the connection.

https://tools.ietf.org/html/rfc2616#section-4.4

4.4 Message Length

2.If a Transfer-Encoding header field (section 14.41) is present and
has any value other than "identity", then the transfer-length is
defined by use of the "chunked" transfer-coding (section 3.6),
unless the message is terminated by closing the connection.

https://tools.ietf.org/html/rfc2616#section-8.1

8.1.2 Overall Operation

Persistent connections provide a mechanism by which a client and a
server can signal the close of a TCP connection. This signaling takes
place using the Connection header field (section 14.10). Once a close
has been signaled, the client MUST NOT send any more requests on that
connection.

and

8.1.2.1 Negotiation

If either the client or the server sends the close token in the
Connection header, that request becomes the last one for the
connection.

Some relevant sections in RFC7230:

https://tools.ietf.org/html/rfc7230#section-3.3.1

3.3.1. Transfer-Encoding

If any transfer coding other than
chunked is applied to a response payload body, the sender MUST either
apply chunked as the final transfer coding or terminate the message
by closing the connection.

https://tools.ietf.org/html/rfc7230#section-3.4

3.4. Handling Incomplete Messages

A response that has neither chunked
transfer coding nor Content-Length is terminated by closure of the
connection and, thus, is considered complete regardless of the number
of message body octets received, provided that the header section was
received intact.

https://tools.ietf.org/html/rfc7230#section-9.6

9.6. Message Integrity

HTTP does not define a specific mechanism for ensuring message
integrity, instead relying on the error-detection ability of
underlying transport protocols and the use of length or
chunk-delimited framing to detect completeness. Additional integrity
mechanisms, such as hash functions or digital signatures applied to
the content, can be selectively added to messages via extensible
metadata header fields. Historically, the lack of a single integrity
mechanism has been justified by the informal nature of most HTTP
communication.

In short, if the client or the server indicates Connection: close, then there is no chunking for responses that have no Content-Length header.

This style of response, a raw body with connection close as the signal that the content is complete, isn't unique to Jetty, or HTTP/1.1.
This is the standard response format for HTTP/0.9, HTTP/1.0, HTTP/1.1, and HTTP/2 (except its the stream closing, not the connection)

Chunked Transfer-Encoding is a concept that only exists in HTTP/1.1, and only for persistent connections.

If you have a need to validate that the content is actually complete, you have to do it in your User-Agent, per the recommendations in the specs.

[fsaintjacques]

Whenever a transfer-coding is applied to a message-body, the set of
transfer-codings MUST include "chunked", unless the message is
terminated by closing the connection.

This only states that if Connection: close is present, then chunked length is not mandatory. You're implying that it's mutually exclusive, I don't agree.

What @minaguib is trying to say is that chunked encoding's mutually-exclusivity is dependent on the HTTP version and not the existence of the Connection: close header.

[frankgrimes97]

FWIW, Nginx will happily return a chunked response even when "Connection: close" is specified on the request.
i.e. it returns Transfer-Encoding: chunked and Connection: close

Others seem to be encountering this in the wild as well: hyperium/hyper#547

I agree that "Connection: close" shouldn't imply no chunking when content-length isn't available.

[joakime]

To address @minaguib and his statement:

As @cconroy mentioned, a client that cares about data integrity and wants to validate that it received a complete (non-truncated) response needs either a Content-Length header or proper HTTP chunking with the signature last-chunk-empty (logical end-of-transmission).

Just because you are using chunking has no meaning that the entire response content was received and/or sent.

The signals:

• Chunked and final 0 block
• Closed Connection

Mean the same thing, that the server will send no more response body content on this specific HTTP exchange.

If this is a persistent connection, then the User-Agent's HTTP parser will note that the exchange is complete and the next content it receives is for a different HTTP exchange.

It does not mean that the entire intended body content was received if the 0 chunk was received, and the User-Agent needs to make that determination based on the content it has actually received.

This is either done with knowledge of the content (like if the final closing element on XML, or final block close on JSON is present, or the image file can be fully decoded, etc), or with some information in the metadata (response headers) that helps the User-Agent validate that the entire response content was received.

If an error (servlet exception, threading exception, io exception, runtime exception, throwable, etc) occurs during the operations on the server side to write the response (blocking write, async I/O writelistener, AsyncContext, non-connection timeouts, etc) then the appropriate onError() methods are called, if they do nothing, then the write completes, the chunked 0 final block is sent, and your User-Agent thinks it got the "entire" content. This is normal behavior.

Using responses without a Content-Length means the response body content is essentially a stream. A stream that can just end, normally, or in error. A normal end is just a stream that stops, not that its "complete" or "finished" or any other meaning you are associating with message integrity.

[frankgrimes97]

@joakime What you're saying "Chunked and final 0 block and Closed Connection mean the same thing" seems to contradict the following section of RFC7230:

3.4.  Handling Incomplete Messages

...

   A client that receives an incomplete response message, which can
   occur when a connection is closed prematurely or when decoding a
   supposedly chunked transfer coding fails, MUST record the message as
   incomplete.

...

   A message body that uses the chunked transfer coding is incomplete if
   the zero-sized chunk that terminates the encoding has not been
   received.  A message that uses a valid Content-Length is incomplete
   if the size of the message body received (in octets) is less than the
   value given by Content-Length.  A response that has neither chunked
   transfer coding nor Content-Length is terminated by closure of the
   connection and, thus, is considered complete regardless of the number
   of message body octets received, provided that the header section was
   received intact.

Why would a server choose to send a "chunked 0 final block" if an error occurs?

[cconroy]

@joakime Let me reiterate a point I made above: even if it were technically valid to close the connection where we have a Connection:close header and are using chunked, it is strictly better to first send the final 0 byte chunk (5 bytes) so that the client can be sure it has received the entire response.

Using responses without a Content-Length means the response body content is essentially a stream. A stream that can just end, normally, or in error. A normal end is just a stream that stops, not that its "complete" or "finished" or any other meaning you are associating with message integrity.

Indeed: if a chunked stream encounters an unexpected error, then it should close the connection as that is the only way it can signal that something went wrong. However, if it finishes successfully it should send the terminating 0 chunk and then close so that the client can distinguish between these two cases.

[frankgrimes97]

The following is what we had been hoping would be possible using Jetty:

"If no Content-Length header field is present and Streaming Miss is in effect, Fastly will stream the content back to the client. However, if while streaming the response body Fastly determines that the object exceeds the maximum object size, it will terminate the client connection abruptly. The client will detect a protocol violation, because it will see its connection close without a properly terminating 0-length chunk."

https://docs.fastly.com/guides/performance-tuning/improving-caching-performance-with-large-files#failure-modes

It would be unfortunate to have to add an application-specific stream terminator when HTTP chunked encoding should support differentiating the two cases.

[tokenrove]

@joakime I think the most relevant portion of RFC7230 is 3.3.3:

Since there is no way to distinguish a successfully completed,
close-delimited message from a partially received message interrupted
by network failure, a server SHOULD generate encoding or
length-delimited messages whenever possible. The close-delimiting
feature exists primarily for backwards compatibility with HTTP/1.0.

I read this as stating that this behavior is acceptable but not encouraged.

[joakime]

@tokenrove that's for network failure, which is only a small fraction of error conditions that can cause a response stream to finish.

Application errors, for example, have to be handled by the application, and since there is no facility in the Servlet spec to close or terminate an active (committed) response, you are stuck with returning from the Servlet, which closes the stream normally.

The only situation where the servlet container can safely make the determination that the responses is invalid, is if the Content-Length was specified, and that amount of content was not sent, allowing the server to forcibly close the connection.

[joakime]

@cconroy the statement

Let me reiterate a point I made above: even if it were technically valid to close the connection where we have a Connection:close header and are using chunked, it is strictly better to first send the final 0 byte chunk (5 bytes) so that the client can be sure it has received the entire response.

This kind of mixing of Connection: close and chunked transfer encoding isn't valid for Jetty, as that is not possible under Jetty.

As greg pointed out, once you have Connection: close, its not possible to have a chunked Transfer-Encoding. The response is being sent in its raw form. Forcing the close chunk 0 isn't valid with Connection: close (currently)

[frankgrimes97]

@joakime I tried throwing a RuntimeException after writing a bunch of data on the ServletOuputStream and I don't see a "chunked 0 final block" written.
cURL correctly reports the following "* transfer closed with outstanding read data remaining" and exits with code 18.

CURLE_PARTIAL_FILE (18) A file transfer was shorter or larger than expected. This happens when the server first reports an expected transfer size, and then delivers data that doesn't match the previously given size.

[kirkwolf]

I have a related question: I want to abort a chunked output stream.

I have an application that is dynamic streaming and using Transfer-Encoding: chunked.
If I have an error condition (after the ServletOutputStream is committed), I want to force the socket to close without sending an EOF chunk.

• Is throwing a RuntimeException from the Servlet the proper way to do this?
• Is this a portable technique, or will different servlet containers behave differently? I can't find anything in the Servlet 3.1 spec related to this.
• Are there other better mechanisms for returning errors on chunked-output streams? I was considering trying to use a "Status-Code" header in the Trailer, but it isn't clear to me whether user-agents in the wild will handle this.

Thanks

[sbordet]

Throwing is probably the most portable way: the Servlet container should attempt to generate an error page, find out the response is already committed and hard close the connection.

[kirkwolf]

Hi Simone,
Thanks, I agree.

Is there anything in the servlet 3.1 spec that says that a container "should" hard-close the connection in this case? Jetty and Tomcat do, but I know of one vendor's Servlet 3.1 container that does not, and would really like to pin them to the spec and fix it.

[sbordet]

jakartaee/servlet#98