Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TLS 1.3 0-RTT doesn't work with TCP Fast Open #4783

Closed
klzgrad opened this issue Nov 24, 2017 · 9 comments

Comments

Projects
None yet
3 participants
@klzgrad
Copy link

commented Nov 24, 2017

I tried to turn on TCP Fast Open upon TLS 1.3 0-RTT in an attempt to reduce connection startup latency even more. (Regular socket IO with TCP_FASTOPEN_CONNECT on kernel 4.11+.)

OpenSSL will flush the socket after ClientHello regardless of presence of early data. Linux's implementation of TCP Fast Open will only send the first write as the SYN packet. A second write (early data in this case) has to wait for the TCP 1-RTT handshake. Thus OpenSSL's 0-RTT and Linux's TCP Fast Open defeats each other.

1

(Selected is the early data.)

The setup: Nginx running on TLS 1.3 draft 21 reverse proxying to an HTTP server at localhost:10080. The loopback interface is set to have 1500 MTU and 100 ms delay on TLS ports. The client is ./openssl s_client -sess_in /tmp/s -alpn h2,http/1.1 -early_data /tmp/http.txt -connect 127.0.0.43:10443. The early data is GET / HTTP/1.0\r\n\r\n. Tested with e44480c.

The TLS draft doesn't seem to be concerned about the transport layer or require ClientHello and early data be sent as separate packets. I don't really know. Or is there some other reason for the flush after ClientHello?

4.1.2. Client Hello
[...] After sending the ClientHello message, the client waits for a ServerHello or HelloRetryRequest message. If early data is in use, the client may transmit early application data (Section 2.3) while waiting for the next handshake message.

I was able to start a connection with both Client Hello, Application Data in a single SYN packet by hacking the state machine a little bit to skip the flush.

2

@mattcaswell

This comment has been minimized.

Copy link
Member

commented Nov 24, 2017

Please see the NOTES section of this page:

https://www.openssl.org/docs/manmaster/man3/SSL_read_early_data.html#NOTES

Perhaps this is the reason?

@klzgrad

This comment has been minimized.

Copy link
Author

commented Nov 25, 2017

I should note that I've read #3906 before reporting this issue and the tests were done with TCP_NODELAY all along. In the scenario of TCP_FASTOPEN even enabling TCP_NODELAY would not avoid the 1 RTT before the early data packet is sent. LWN has noted this issue about TCP Fast Open:

This assumes that the client's initial request is small enough to fit inside a single TCP segment. This is true for most requests, but not all. Whether it might be technically possible to handle larger requests—for example, by transmitting multiple segments from the client before receiving the server's ACK—remains an open question. [1]

I guess what I'm asking for OpenSSL is that can changes be made to allow this usage of TCP Fast Open by optionally not flushing immediately after ClientHello (e.g. if TCP_NODELAY is detected on this socket).

@kroeckx

This comment has been minimized.

Copy link
Member

commented Nov 25, 2017

@klzgrad

This comment has been minimized.

Copy link
Author

commented Nov 25, 2017

@kroeckx This is not portable but Linux kernel provides TCP_FASTOPEN_CONNECT which allows regular sockets to use Fast Open without sendto() calls. See torvalds/linux@19f6d3f. The use case here is to minimize the latency of a TLS tunnel between two endpoints both controlled by the user (thus with updated Linux kernel). I haven't looked into how this can be implemented with sendto() on older Linux or other platforms.

diff --git a/apps/s_socket.c b/apps/s_socket.c
index 2f3e90bbf9..cf48fc9039 100644
--- a/apps/s_socket.c
+++ b/apps/s_socket.c
@@ -91,6 +91,23 @@ int init_client(int *sock, const char *host, const char *port,
             continue;
         }
 
+        if (type == SOCK_STREAM) {
+            ret = BIO_set_tcp_ndelay(*sock, 1);
+            if (ret < 0) {
+                BIO_closesocket(*sock);
+                ERR_print_errors(bio_err);
+                return 0;
+            }
+            int opt = 1;
+            ret = setsockopt(*sock, IPPROTO_TCP, /*TCP_FASTOPEN_CONNECT*/30, (char *)&opt, sizeof(opt));
+            if (ret < 0) {
+                BIO_closesocket(*sock);
+                ERR_print_errors(bio_err);
+                return 0;
+            }
+            ret = 0;
+        }
+
 #ifndef OPENSSL_NO_SCTP
         if (protocol == IPPROTO_SCTP) {
             /*
@kroeckx

This comment has been minimized.

Copy link
Member

commented Nov 25, 2017

@kroeckx

This comment has been minimized.

Copy link
Member

commented Nov 25, 2017

@klzgrad

This comment has been minimized.

Copy link
Author

commented Nov 25, 2017

The SYN packet is sent immediately on the first write(). The kernel TCP stack doesn't buffer for Fast Open. I just tested that both disabling TCP_NODELAY and turning on TCP_CORK suffered the 1-RTT TCP handshake.

https://github.com/torvalds/linux/blob/19f6d3f3c8422d65b5e3d2162e30ef07c6e21ea2/net/ipv4/tcp_fastopen.c#L350


Reading the commit

Yes.

@kroeckx

This comment has been minimized.

Copy link
Member

commented Nov 25, 2017

So I think it's best we try to avoid that flush after the client hello if we know we're going to send early data too.

mattcaswell added a commit to mattcaswell/openssl that referenced this issue Nov 27, 2017

Don't flush the ClientHello if we're going to send early data
We'd like the first bit of early_data and the ClientHello to go in the
same TCP packet if at all possible to enable things like TCP Fast Open.
Also, if you're only going to send one block of early data then you also
don't need to worry about TCP_NODELAY.

Fixes openssl#4783
@mattcaswell

This comment has been minimized.

Copy link
Member

commented Nov 27, 2017

Take a look at the patch in #4802. Does this solve it?

mattcaswell added a commit to mattcaswell/openssl that referenced this issue Dec 27, 2017

Don't flush the ClientHello if we're going to send early data
We'd like the first bit of early_data and the ClientHello to go in the
same TCP packet if at all possible to enable things like TCP Fast Open.
Also, if you're only going to send one block of early data then you also
don't need to worry about TCP_NODELAY.

Fixes openssl#4783

@levitte levitte closed this in 2a8db71 Dec 28, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.