Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

net: add support for TCP Fast Open #4842

Open
bradfitz opened this issue Feb 19, 2013 · 46 comments

Comments

@bradfitz
Copy link
Member

@bradfitz bradfitz commented Feb 19, 2013

TCP listeners and TCP client connections should both be able to conditionally enable
TCP_FASTOPEN.

See https://lwn.net/Articles/508865/

This is now available in Linux 3.7 (both client & server).

This would be nice to conditionally use in http.DefaultTransport, for GET requests.

See also: issue #3097 (adding Timeouts to DialTCP)
See also: issue #3610 (DialPlan)
@alexbrainman

This comment has been minimized.

Copy link
Member

@alexbrainman alexbrainman commented Feb 19, 2013

Comment 2:

Probably unrelated, but FYI. ConnectEx and AcceptEx we use to dial / accept tcp
connections allow to send / receive some data as part of the same api call.
Alex
@bradfitz

This comment has been minimized.

Copy link
Member Author

@bradfitz bradfitz commented Feb 27, 2013

Comment 3:

Re #2: IIUC, that's related, but against spec. TCP permits such a packet, but also says
that the recipient isn't supposed to act on it until the connection is verified, since
the initial SYN packet could've had its source IP/port spoofed. TCP_FASTOPEN is a
formalized version of that which isn't susceptible to spoofing.
I never knew that AcceptEx makes that optional. I suppose a Windows webserver could use
AcceptEx w/ data and verify first it's a GET request to a public resource and send the
response immediately.  I remember IE / Microsoft doing this years ago (to win HTTP
benchmarks?), but I didn't remember how. Interesting.
@bradfitz

This comment has been minimized.

Copy link
Member Author

@bradfitz bradfitz commented Feb 27, 2013

Comment 4:

Now that DialOpt is in (https://code.google.com/p/go/source/detail?r=54731addb542 /
docs: http://tip.golang.org/pkg/net/#DialOpt), this is easier.
There are two API possibilities.
a) lazy.
b) explicit.
In the lazy API, the connect would be like:
  conn, err := net.DialOpt("google.com:80", net.FastOpen())
... and a fake net.Conn is returned immediately.  No possibility for timeouts or errors.
 Not until the first Write does the actual sendto system call (on Linux) get called. 
This is what Chrome does.
With the explicit API alternative (my preference), the API will probably be like: 
  conn, err := net.DialOpt("google.com:80", net.InitialData("GET / HTTP/1.0\r\n\r\n"))
Where net.InitialData returns a DialOption (http://tip.golang.org/pkg/net/#DialOption)
that enables TFO if the client supports it (on Linux: checking the low bit of
/proc/sys/net/ipv6/tcp_fastopen) and otherwise just does a Write after connect.
Speaking with Yuchung Cheng, it seems there's enough momentum beh that something like it
will be formalized one way or another, it will be formalized one way or another.  And
Linux 3.8 supporting it (client & server) is promising (and Windows in an alternate
form).  Thus I'd like to keep "TFO" or "TCPFastOpen" out of the Go API name and make the
DialOption be just net.InitialData(...).  Up for debate whether that's a []byte, string,
or io.Reader.  Probably a []byte for consistency with Write.
@bradfitz

This comment has been minimized.

Copy link
Member Author

@bradfitz bradfitz commented Feb 27, 2013

Comment 5:

This issue was updated by revision abf5700.

R=golang-dev, r
CC=golang-dev
https://golang.org/cl/7369060
@rsc

This comment has been minimized.

Copy link
Contributor

@rsc rsc commented Mar 12, 2013

Comment 6:

[The time for maybe has passed.]

Labels changed: removed go1.1maybe.

@mikioh

This comment has been minimized.

Copy link
Contributor

@mikioh mikioh commented Sep 12, 2013

Comment 7:

Labels changed: added go1.3maybe.

@rsc

This comment has been minimized.

Copy link
Contributor

@rsc rsc commented Dec 4, 2013

Comment 9:

Labels changed: added release-none, removed go1.3maybe.

@rsc

This comment has been minimized.

Copy link
Contributor

@rsc rsc commented Dec 4, 2013

Comment 10:

Labels changed: added repo-main.

@mikioh

This comment has been minimized.

Copy link
Contributor

@mikioh mikioh commented Dec 4, 2013

Comment 11:

Status changed to Thinking.

@mikioh

This comment has been minimized.

Copy link
Contributor

@mikioh mikioh commented Aug 13, 2014

Comment 12:

Status changed to New.

@bradfitz bradfitz added new labels Aug 13, 2014
@bradfitz bradfitz removed the new label Dec 18, 2014
@mikioh mikioh changed the title net: support TCP Fast Open (TFO) net: add support for TCP Fast Open Dec 22, 2014
@mikioh

This comment has been minimized.

Copy link
Contributor

@mikioh mikioh commented Dec 22, 2014

@JunfengJia

This comment has been minimized.

Copy link

@JunfengJia JunfengJia commented Mar 3, 2015

how to make tfo work with http package? http's persist cache depend on established connections.

@bronze1man

This comment has been minimized.

Copy link
Contributor

@bronze1man bronze1man commented Mar 3, 2015

@JunfengJia you can use syscall to dial a new connection in http.Transport to work with http package.
Here is the tfo library I used:
http://godoc.org/github.com/bronze1man/kmg/kmgNet#TfoLazyDial

@mikioh

This comment has been minimized.

Copy link
Contributor

@mikioh mikioh commented Mar 3, 2015

@JunfengJia,

Still thinking. We need to handle the entire stack, TCP through HTTP 1.x, HTTP/2, or WebSocket w/ or w/o TLS, effectively. Once HTTP/2 support lands, I will revisit this issue.

@JunfengJia

This comment has been minimized.

Copy link

@JunfengJia JunfengJia commented Mar 3, 2015

@bronze1man,

http will make a read call first, then the connect will fall into 3whs.

@bronze1man

This comment has been minimized.

Copy link
Contributor

@bronze1man bronze1man commented Mar 4, 2015

@JunfengJia

Thanks.It looks like my library will not work with net/http library right now.
Read or write call first is a complex stuff.I have waited 1ms to make sure write call come first from client with my own defined protocol.

@mikioh

This comment has been minimized.

Copy link
Contributor

@mikioh mikioh commented Mar 8, 2015

FWIW, old changelist 27150044 has moved to https://go-review.googlesource.com/#/c/7121/.

@rsc rsc added this to the Unplanned milestone Apr 10, 2015
@rsc rsc removed release-none labels Apr 10, 2015
@bradfitz

This comment has been minimized.

Copy link
Member Author

@bradfitz bradfitz commented Jul 29, 2015

In addition to TFO, we need to figure out APIs for crypto/tls TLS 1.3 0-RTT stuff. And then the net/http client can use them both.

/cc @agl

@mikioh

This comment has been minimized.

Copy link
Contributor

@mikioh mikioh commented Mar 9, 2016

The change golang.org/cl/7121 is ready to land. However I'm not sure whether crypto/tls and net/http packages really need this feature. Please ping golang.org/cl/7121 when the time comes.

@bradfitz

This comment has been minimized.

Copy link
Member Author

@bradfitz bradfitz commented Mar 10, 2016

The change golang.org/cl/7121 is ready to land.

It says "DO NOT REVIEW" so I stopped reviewing.

@mikioh

This comment has been minimized.

Copy link
Contributor

@mikioh mikioh commented Mar 10, 2016

Thanks for the notice, just did s/DO NOT REVIEW/FOR DISCUSSION/

@mikioh mikioh modified the milestones: Go1.10, Go1.11 Nov 25, 2017
@ztheory

This comment has been minimized.

Copy link

@ztheory ztheory commented Jan 4, 2018

Hello @mikioh . Just curious on the likelihood if this being in Go 1.10 vs. Go 1.11. Thanks in advance.

@mikioh

This comment has been minimized.

Copy link
Contributor

@mikioh mikioh commented Jan 5, 2018

@ztheory,

No, the Go 1.10 release train departed a few months ago.

@mikioh

This comment has been minimized.

Copy link
Contributor

@mikioh mikioh commented Jan 5, 2018

Over the past 4 years, the dissemination of fancy newly TCP options including TCP Fast Open option progressed gradually. Linux kernel 3.x or above implements TCP Fast Open option on both active- and passive-open sides by default, and network intermediary manufacturers start to consider dealing with the newly TCP options. Unfortunately, the support of the other host platforms still requires struggling; Darwin kernel focuses on the active-open side only and requires the use of new system calls (though, I guess it'd be a great help when using both MPTCP and TCP Fast Open options), Windows requires the use of specific API such as ConnectEx, and FreeBSD requires to rebuild the kernel. [end of status update]

The good news is that Go 1.11 will probably fix #9661 and that will allow the use of TCP Fast Open option easily, without managing tedious states in between user-surface API and the protocol stack inside the kernel. You may achieve it by using the combo of the Control field of net.Dialer and TCP_FASTOPEN_CONNECT socket option, or the Control field of net.ListenConfig and TCP_FASTOPEN socket option on Linux kernel 4.11 or above.

@kimw kimw referenced this issue Mar 30, 2018
@bradfitz bradfitz changed the title proposal: net: add support for TCP Fast Open net: add support for TCP Fast Open May 18, 2018
@bradfitz bradfitz modified the milestones: Go1.11, Unplanned May 18, 2018
@riobard

This comment has been minimized.

Copy link

@riobard riobard commented Jun 28, 2018

Are we going to get this in GO 1.11?

@bradfitz

This comment has been minimized.

Copy link
Member Author

@bradfitz bradfitz commented Jun 28, 2018

@riobard, no. This bug is still open.

@AudriusButkevicius

This comment has been minimized.

Copy link
Contributor

@AudriusButkevicius AudriusButkevicius commented Jul 9, 2018

Isn't fastopen just a socket option that should be possible in go 1.11 atleast on linux?
For this to be closed, does it need to be solved on all platforms?

@bradfitz

This comment has been minimized.

Copy link
Member Author

@bradfitz bradfitz commented Jul 9, 2018

Yeah, this bug was about adding nice, portable API for it (ioctls aren't great API), and using it where it makes sense (the top comment mentions net/http for instance)

But it's true that in Go 1.11 you could use https://tip.golang.org/pkg/net/#ListenConfig

@rumpelsepp

This comment has been minimized.

Copy link

@rumpelsepp rumpelsepp commented Jul 10, 2018

@AudriusButkevicius

For the server yes. The client needs to send the initial data with sendto(2), continuing with read(2)/write(2). connect(2) triggers a three way handshake instead.

UPDATE: I missed this one: torvalds/linux@19f6d3f

@xqdoo00o

This comment has been minimized.

Copy link

@xqdoo00o xqdoo00o commented Sep 29, 2018

Are we going to get this in GO 2?

@bradfitz

This comment has been minimized.

Copy link
Member Author

@bradfitz bradfitz commented Sep 29, 2018

This has little if anything to do with Go 2.

@qJkee

This comment has been minimized.

Copy link

@qJkee qJkee commented Oct 19, 2018

Any progress?
Or it's freezed for a long time?

@acln0

This comment has been minimized.

Copy link
Contributor

@acln0 acln0 commented Jan 22, 2019

Hello. Is there still interest in this?

Given the flow described in torvalds/linux@19f6d3f, assuming support for TFO only for Linux 4.11+, the code could be simplified greatly, compared to what CL 7121 does. Adoption in other places in the standard library would also be easier, since caller code would not need to change.

@mikioh, @bradfitz: would it be okay if I tried this from scratch in the upcoming cycle? It looks like https://go-review.googlesource.com/c/go/+/7121/ would need a pretty complicated rebase to bring it into the present.

@bradfitz

This comment has been minimized.

Copy link
Member Author

@bradfitz bradfitz commented Jan 23, 2019

@acln0, oh, nice! I hadn't seen that. Given that new API, this is basically trivial. In fact, people can do it already today with Go 1.11 and its new:

https://tip.golang.org/pkg/net/#Dialer.Control

Then you can just do the setsockopt(s, IPPROTO_TCP, TCP_FASTOPEN_CONNECT ...) and give that net.Dialer to net/http, etc.

(Of course, if you do so, you should only use that http.Client for GET/HEAD/idempotent requests.)

But sure, you could work on it.

@mikioh

This comment has been minimized.

Copy link
Contributor

@mikioh mikioh commented Jan 23, 2019

@acln0,

If you are talking about stuff in the net package, probably still not the time. If net/http, crypto/tls, or net/http over crypto/tls, it should be another issue and probably it's fine unless you try to introduce a) a quick hack for message boundary handling; especially without considering byte-seq.+partial read/writes vs. datagram/chunks+full reads/writes, b) a quick hack for multipath transport functionality.

@acln0

This comment has been minimized.

Copy link
Contributor

@acln0 acln0 commented Jan 23, 2019

@bradfitz, I had the net/http use case for idempotent requests in mind. I don't know the intricacies of how crypto/tls might make use of TFO in the larger context of TLS 1.3 etc., so I cannot comment on that.

Yes, people can do it at the net.Dialer level themselves today, but I thought there might be some value in adding options to the net package as well. If the standard library is going to use TFO, my thought was the following: instead of making net/http and crypto/tls set socket options themselves, we could try to introduce FastOpen knobs to net.Dialer and net.Listener, to consolidate that code in one well defined place.

Once the new API existed in the net package, both the standard library and other clients could start adopting it, at whichever pace they like. I wrote my initial message in this bug because I was interested in giving the ground work in package net another try.

@mikioh, can you please clarify what you mean about "stuff in the net package, probably still not the time"? Do you mean the possibility of FastOpen knobs on net.Dialer and net.Listener? It seems to me like Linux and FreeBSD support both passive and active sides pretty easily. And OS X supports active side too. Do you think this level of support across operating systems is not wide enough to warrant an option in package net?

@mikioh

This comment has been minimized.

Copy link
Contributor

@mikioh mikioh commented Jan 23, 2019

@acln0,

Do you mean the possibility of FastOpen knobs on net.Dialer and net.Listener?

Yes.

FreeBSD support both passive and active sides pretty easily

As far as I can see, on FreeBSD, a few GENERIC conf files include TCP_RFC7413 option by default from 12-STABLE, but not all, also unlike passive-open side operation, enabling active-open operation still needs sysctl tweaking.

find freebsd/sys/conf -name GENERIC | xargs grep TCP_RFC7413
amd64/conf/GENERIC:options            TCP_RFC7413             # TCP Fast Open
powerpc/conf/GENERIC:options  TCP_RFC7413             # TCP Fast Open
arm64/conf/GENERIC:options            TCP_RFC7413             # TCP Fast Open

And OS X supports active side too.

As far as I can see, it still supports active-open operation only and requires connectx thru syscall.Syscall9.

this level of support across operating systems is not wide enough to warrant an option in package net?

Still a bit early to me. If it's pretty fine within the real world, it's better to be enabled by default (and we should avoid introducing a control knob each time we need to fix something user uncomfortable on the operation of TCP Fast Open option, including configuring cookie-related life-duration/invalidation/capacity control.)

@qJkee

This comment has been minimized.

Copy link

@qJkee qJkee commented Jan 23, 2019

@brain

@acln0, oh, nice! I hadn't seen that. Given that new API, this is basically trivial. In fact, people can do it already today with Go 1.11 and its new:

https://tip.golang.org/pkg/net/#Dialer.Control

Then you can just do the setsockopt(s, IPPROTO_TCP, TCP_FASTOPEN_CONNECT ...) and give that net.Dialer to net/http, etc.

(Of course, if you do so, you should only use that http.Client for GET/HEAD/idempotent requests.)

But sure, you could work on it.

Any more complete example?
all setsockoptByte(Inet|ICMP) require fd, but we got only syscall.RawConn, or it can be casted to fd(int)?

@acln0

This comment has been minimized.

Copy link
Contributor

@acln0 acln0 commented Jan 23, 2019

@qJkee, call the Control method on the syscall.RawConn. Its argument is a function of the integer file descriptor. You can call setsockopt in that function.

@qJkee

This comment has been minimized.

Copy link

@qJkee qJkee commented Jan 23, 2019

@acln0 thank you! I will try to implement it asap, and maybe post some bech.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
You can’t perform that action at this time.