Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Clarify rules around half-closed TCP connections #22

Open
bradfitz opened this issue Jan 6, 2017 · 25 comments

Comments

Projects
None yet
@bradfitz
Copy link

commented Jan 6, 2017

The HTTP RFCs say nothing about the expectations around half-closed TCP connections.

In practice, I haven't seen any HTTP client in the wild send a request, and then send a FIN (shutdown) while still waiting for the server's response.

Because we haven't see any clients do that, as of Go 1.8, Go's HTTP server is starting to make assumptions that reading an EOF from the client means that the client is no longer interested in the response. (reading EOF being the closest portable approximation to "the client has gone away").

But in golang/go#18527, a user reports that they have an internal HTTP client which does indeed make a half-closed TCP request.

It would be nice if the HTTP RFCs provided guidance as to whether this is allowed or frowned upon.

I would recommend that the RFC suggest that clients SHOULD NOT half-close their TCP connections while awaiting responses. Because nobody else does, empirically, and relying on reading EOF is a useful signal for servers.

/cc @mnot @benburkert

@mcmanus

This comment has been minimized.

Copy link

commented Jan 6, 2017

I've maintained a server in the past that received the same bug report. As a matter of compliance I decided the client was right - EOF is a way to delimit a message (they were using it to stream request bodies without chunking). But as a matter of practicality I didn't fix the bug because, as you indicate, ignoring EOF in practice means a lot of useless buffering ending in RST or timeouts.

@bradfitz

This comment has been minimized.

Copy link
Author

commented Jan 6, 2017

@mcmanus, oh, interesting. And disgusting. I hadn't considered EOF being a means to end an HTTP/1.0 POST. Perhaps I can change Go to make an exception for HTTP/1.0 requests with request bodies.

@bradfitz

This comment has been minimized.

Copy link
Author

commented Jan 9, 2017

@mcmanus, but @badger points out that RFC 1945 (HTTP 1/.0) says:

The presence of an entity body in a
request is signaled by the inclusion of a Content-Length header field
in the request message headers. HTTP/1.0 requests containing an
entity body must include a valid Content-Length header field.

So using a half-closing a TCP connection was never a valid way to signal the end of an entity body.

@mnot mnot added the h1-messaging label Jun 20, 2017

@wenbozhu

This comment has been minimized.

Copy link

commented Jun 6, 2018

This is a pretty important question. Today, it's hard to spec out how cancellation works with REST.

I suspect the proposed spec change will break some users. I wonder if it's possible to measure the impact with real traffic, from the server-side, e.g. the success rate of response completion after a half-close is received. I am happy to run some experiments ...

@RataDP

This comment has been minimized.

Copy link

commented Oct 18, 2018

Here is a example of half-close connection in the wild, Livestatus. Livestatus is a broker for nagios. To use it via sockets you have to make the query, ex. GET hosts and the close the write channel. After this, the Livestatus returns by the half-closed socket the results.

Example in Python

#!/usr/bin/python
#
# Sample program for accessing the Livestatus Module
# from a python program
socket_path = ("localhost", 6557)

import socket
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect(socket_path)

# Write command to socket
s.send("GET hosts\n")

# Important: Close sending direction. That way
# the other side knows we are finished.
s.shutdown(socket.SHUT_WR)

# Now read the answer
answer = s.recv(100000000)

# Parse the answer into a table (a list of lists)
table = [ line.split(';') for line in answer.split('\n')[:-1] ]

print table 

The important snippet is this, when closing the Write channel of the socket:

# Important: Close sending direction. That way
# the other side knows we are finished.
s.shutdown(socket.SHUT_WR) 

Are there any workaround for golang?

Source: https://mathias-kettner.de/checkmk_livestatus.html

@kazuho

This comment has been minimized.

Copy link

commented Oct 19, 2018

@RataDP Interesting. But is that HTTP?

I see examples like the following on the web page. It does not even look like HTTP/0.9, though I am not sure what the proper syntax of 0.9 is.

root@linux# echo "COMMAND [$(date +%s)] START_EXECUTING_SVC_CHECKS" \
     | unixcat /var/lib/nagios/rw/live

@mnot mnot added the discuss label Oct 19, 2018

@RataDP

This comment has been minimized.

Copy link

commented Oct 21, 2018

@RataDP Interesting. But is that HTTP?

I see examples like the following on the web page. It does not even look like HTTP/0.9, though I am not sure what the proper syntax of 0.9 is.

root@linux# echo "COMMAND [$(date +%s)] START_EXECUTING_SVC_CHECKS" \
     | unixcat /var/lib/nagios/rw/live

It could be with unix socket or tcp socket.
My fail I only read TCP, i did not see the repo is http :s. My bad, sorry, I came from another issue that was speaking about TCP without the http specification..

@jhatala

This comment has been minimized.

Copy link

commented Nov 2, 2018

Hello, everyone! I would like to make a case for HTTP/1 disallowing half-closed TCP connections from the client.

What I mean by that is that when an HTTP/1 client shuts down their writing end of the connection (when they send a TCP frame with the FIN flag), that the server can safely assume that the client lost interest in the unsent remainder of the response.

On the other hand, if the server is required to support half-closed TCP connections from the client, in other words if the client is allowed to send a FIN after sending its request and it still expects to receive the response, then the server can only assume that the client lost interest when it receives an RST from the client.

To be able to treat an incoming FIN as an indication of the client's loss of interest has advantages for the server:

  • The server can free up resources earlier, upon the reception of the FIN, rather than having to wait for an RST a network round-trip time after sending a data packet.
  • The savings are most pronounced when dealing with responses that trickle information down slowly as a form of long polling.
  • Some stateful firewalls (NAT) between the client and the server drop packets for connections for which they no longer have state, rather than sending back RSTs. The RSTs thus never make it to the server, the server continues trickling down information until the lack of ACKs causes its window to fill and eventually a write to time out. Server resources held on behalf of a client who lost interest are wasted resources.

I believe that there would be little to no disadvantage to the client. Furthermore:

  • HTTP/1.1 encourages the reuse of a TCP connection for a subsequent request.
  • Request message bodies are framed using Content-Length or Transfer-Encoding: chunked: "The presence of a message body in a request is signaled by a Content-Length or Transfer-Encoding header field." ( https://tools.ietf.org/html/rfc7230#section-3.3 )
  • TLS 1.0 explicitly precludes half-closing at its layer: When one end receives a "close_notify" alert, it is required that it "respond with a close_notify alert of its own and close down the connection immediately, discarding any pending writes." ( https://tools.ietf.org/html/rfc2246#section-7.2.1 )

Thank you.

@royfielding

This comment has been minimized.

Copy link
Member

commented Nov 2, 2018

A half-close has never been an indication that the client isn't interested in the response. It only indicates that the client isn't going to send any future requests on that connection. Since HTTP/1.x is defined to be independent of the transport, I see no reason to specify half-close – what matters is that the request message be complete. A server is not obligated to send a response, regardless.

@bradfitz

This comment has been minimized.

Copy link
Author

commented Nov 2, 2018

Since HTTP/1.x is defined to be independent of the transport,

Yeah, but TCP is a pretty popular choice. I think it's worth clarifying what this part of TCP means for HTTP.

This bug exists because implementations are disagreeing on what it means. We're looking to a spec for guidance.

@wenbozhu

This comment has been minimized.

Copy link

commented Nov 5, 2018

It's pretty hopeless at this point. I propose we clarify that half-close means nothing to HTTP/1.x in this spec, i.e. it's not a cancellation. If any http/1->http/2+ proxies send a rst stream when receiving a half-close from the client, it's a bug.

@royfielding

This comment has been minimized.

Copy link
Member

commented Nov 5, 2018

The problem with "clarifying it for TCP" is that HTTP runs on any transport with connection qualities and finding a word that means half-close for TCP doesn't always translate into some other transport or session-layer's meaning of half-close, but …

I will try to find a way to do that when we get to the whole "what do we mean by a connection" rewrite in the semantics spec.

In any case, regarding the original question: Go should not interpret an EOF on read as implying an EOF on close. Even if we don't see that as common in standard practice, there are probably thousands of deployed C applications that distinguish the two states; they won't work when some poor soul tries to port them to Go, and you'll be reliving this discussion on a regular basis. For example, IIRC, the first conversation I ever had with @mnot (in 2000) was about using half-close to end requests in iCAP. It happens a lot more frequently in private hacks that are merely using the HTTP libraries for some other purpose.

@mcmanus

This comment has been minimized.

Copy link

commented Nov 6, 2018

in bkk: no strong consensus. mike bishop to see if he can identify clients that are using half close actively. agreed scope here is h1/tcp

@DavidSchinazi

This comment has been minimized.

Copy link

commented Nov 6, 2018

I don't think there are any clients currently using this, but I think it would make sense to add guidance stating that half-close does not have any semantics, for all versions of HTTP. Having servers not kill half-closed connections may allow innovation down the line, and I don't think giving this guidance is risky

@wtarreau

This comment has been minimized.

Copy link

commented Nov 6, 2018

I've seen that quite a bit in field with monitoring scripts involving netcat, as well as data transfer tools. It's a bit less common nowadays since wget and curl are everywhere, but the usual stuff used to be this :
$ echo -e "GET /status HTTP/1.0\r\n\r\n" | nc $host:$port | grep -q OK
$ echo -e "GET /new-acl-file.txt HTTP/1.0\r\n\r\n" | nc $host:$port | sed '1,/^$/d' > acl-file.txt

In haproxy we don't do anything specific with half-close, we only rely on the end of message, unless the admin specifies "option abortonclose" in which case a half-closed client connection will be aborted if the server has not started to respond.

Also there is no real value in killing half-closed connections. Often people who want to do this mix up the end of message and the end of connection and tend to believe that if the connection is not aborted, there will be no more opportunity to do it. But in fact if the client really aborts, the server will receive RST in response to some send() and will be able to detect the close. I would say that :

  • half-closed (shutdown(SHUT_WR)) is a graceful shutdown (i.e. "nothing more to say"), signaled on the wire with a TCP FIN and detected on the server as read()==0
  • abort (close()) is a real connection abort (i.e. "stop sending this to me"), signaled on the wire with a TCP RST and detected on the server as send()==-1

The only case where you don't know is until you've started to send(), which explains why haproxy uses its option to decide what to do before receiving the server's response.

I'm just realizing that we could even suggest to send a 100-continue in response to a half-closed connection to probe the connection!

@MikeBishop

This comment has been minimized.

Copy link
Contributor

commented Nov 6, 2018

What I've heard back from Microsoft folks with access to old emails is that we've seen half-close behavior from:

  • .NET 2.0's HTTP implementation
  • An IBM Java client, about which we have little additional detail

Obviously, these are old clients with negligible share. However, that also illustrates the risk: old clients will not be updated to comply with a new spec.

@bradfitz

This comment has been minimized.

Copy link
Author

commented Nov 6, 2018

Go should not interpret an EOF on read as implying an EOF on close.

Go's HTTP package uses EOF from clients to mean "the HTTP client is probably no longer interested in the server's response". We use it to notify callers of the HTTP server who've registered their interest in knowing when the HTTP client is gone (especially one reading a long-polled response, like Server-Sent Events). Notably, we want to know this immediately upon a FIN, without waiting for a write to fail. (We might not have anything to write for some time.)

While we might ideally use OS-specific TCP stats kernel interfaces to distinguish FIN from RST, we support a dozen+ OSes and the basic interface we can rely on is EOF on FIN. There often isn't a better userspace API available to know the TCP state.

I'd rather the HTTP spec say that clients should not half-close TCP connections, as some servers may interpret a half-closed connection as a client that's no longer interested in the response.

Even if we don't see that as common in standard practice, there are probably thousands of deployed C applications that distinguish the two states; they won't work when some poor soul tries to port them to Go, and you'll be reliving this discussion on a regular basis.

Go has a net package that lets you do low-level networking-y things. Anybody porting low-level C could port to that. The concern for this bug is about Go's net/http behavior, not its net package.

@wtarreau

This comment has been minimized.

Copy link

commented Nov 6, 2018

Hi Brad!

You must not resort to using the OS to distinguish between the two because you don't need to know that. As you say it's system-specific. And until you send anything you're not guaranteed to get an RST anyway.

The problem you're facing is that you're acting as a proxy between the client and the application. You need a way to let the application know that the client indicated that it stops sending, and only the application can decide if it's an indication of end of transfer or of client abort. Sometimes the application will decide that both are equivalent and that's fine. Of course when you get a notification of error via an RST, it's pretty clear and unambigous. If I may suggest, "just" pass a shut_read info in one case that applications are free to interpret as aborts if they want, and an abort or close info when you're certain it's closed. But really any network-based application must be able to tell the difference between half-closed and full-closed otherwise it's deemed to either abusively close on regular half-close or ignore real closes sometimes.

In practice people tend to see TCP as a bidirectional stream while it's in fact two independent unidirectional streams. The only link between the two are the ACK numbers passed along packets to confirm receipt. But FTP servers for example are well used to seing half-closed connections since the data channel can be half-closed during the whole transfer.

@bradfitz

This comment has been minimized.

Copy link
Author

commented Nov 6, 2018

@wtarreau, Go's HTTP package has supported its CloseNotifier API since 2013-05-13 and the Go compatibility promise means it's not going away. We can improve the implementation, but we can't pretend the problem doesn't exist and remove the feature. It came about because it was frequently requested. Authors of HTTP server handlers want to know when people close tabs in their browser, etc.

@wtarreau

This comment has been minimized.

Copy link

commented Nov 7, 2018

@bradfitz OK that's perfect. Then it's more a matter of clarifying what each event means and what shortcuts may be taken with what impacts. In practice it's fine most of the time, it's just that it's important to be clear about what this really means. For example it's fine to say "you can reasonably assume that a close notification indicates a closed tab and that you can abort an ongoing operation if your application is designed to work solely with a browser, but a more robust application should consider that it only indicates the client has nothing more to send and that the server must close just after sending the final response ; specifically, some scripts or API clients may induce a close event immediately after sending a request and while waiting for a response".

@mnot

This comment has been minimized.

Copy link
Member

commented Nov 12, 2018

Discussed in Bangkok; we need to collect data about behaviour.

@Lukasa

This comment has been minimized.

Copy link

commented Mar 25, 2019

As an extra data point, both SwiftNIO and Netty treat receiving EOF on read as an indication the client is no longer interested in the response. We're definitely not happy with this: from our perspective, clients should be able to send FIN without us giving up on them. I'd be in favour of wording to disincentivise servers from doing what we (currently) do.

@mcmanus

This comment has been minimized.

Copy link

commented Mar 25, 2019

ietf104: significant interest in documenting _something_here. at least "client should not do this" - hummed the question of discouraging the server from closing connection on rcpt of fin. a decent support for that pr.

@MikeBishop

This comment has been minimized.

Copy link
Contributor

commented Mar 25, 2019

There seems to be a really simple hack for differentiating FIN/RST. As @bradfitz noted, a write will fail if a RST was received. This issue only applies to HTTP/1.1. Therefore: upon read-EOF, immediately write the "HTTP/1.1 " of the response line. If that write fails, you can safely abort generation of the rest of the response, because it was a RST.

Or does this not work on certain platforms?

@Lukasa

This comment has been minimized.

Copy link

commented Mar 25, 2019

There are also various OS level options. Both epoll and kqueue can be used to distinguish FIN and RST, so it’s usually a matter of distinguishing the two cases in higher level APIs.

@mnot mnot removed the discuss label Apr 12, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.