Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

proposal: net: add support for MPTCP #56539

Open
matttbe opened this issue Nov 3, 2022 · 16 comments
Open

proposal: net: add support for MPTCP #56539

matttbe opened this issue Nov 3, 2022 · 16 comments
Labels
Milestone

Comments

@matttbe
Copy link

matttbe commented Nov 3, 2022

Hello,

I'm part of the Linux MPTCP Upstream community and some users reported difficulties to support MPTCP with Go.

Multipath TCP or MPTCP is an extension to TCP. The goal is to exchange data for a single connection over different paths, simultaneously (or not). Its support in the Linux kernel has started in Linux v5.6.

For the userspace, the usage is similar to TCP. The main difference is visible when creating the socket because the MPTCP protocol has to be set:

socket(AF_INET(6), SOCK_STREAM, IPPROTO_MPTCP);

After that, the application uses the socket like it would do if it was TCP. It can also get and set socket options from different levels: socket, IP(v6), TCP and also a new one: SOL_MPTCP.

(Note that it is also possible to create MPTCP connection on MacOS but the API is different)

An easy way to force an application to support MPTCP is to use mptcpize tool which uses LD_PRELOAD with a lib changing the behaviour of libc's socket API: if TCP is asked, force MPTCP instead. Of course, this is not possible with Go apps as this trick is specific to the libc.

I'm not a Go developer but from what I saw, it looks like the recommended way to communicate with a server with TCP is to use the net package. It would be great if the net package could support MPTCP, e.g.

conn, err := net.Dial("mptcp", "golang.org:80")

From what I saw, TCP sockets from the net package calls (1)(2)(3) internetSocket() function with 0 for the protocol ID. (This function will call: socket()sysSocket()socketFunc()syscall.Socket re-using the protocol ID given in argument.)

It should not be difficult to allow setting this protocol ID argument to IPPROTO_MPTCP (262) instead, no?

@gopherbot gopherbot added this to the Proposal milestone Nov 3, 2022
@apparentlymart
Copy link

apparentlymart commented Nov 4, 2022

Hi @matttbe!

Multipath TCP is new to me so I must admit that I've only read the introductory sections of RFC8684. I apologize if the following questions are naive.

As a developer of network applications, when I'm choosing between TCP and UDP (the two main choices available through the net package) I tend to do so based on fundamental differences between the two; stream-oriented vs. packet oriented being the key consideration.

The introductory material for Multipart TCP in the RFC describes it as being a backward-compatible extension to TCP, capable of interacting with existing TCP-only implementations and able to "progressive-enhance" (to borrow terminology from the web platform) a TCP connection into a MPTCP connection by negotiating subflows after an initial standard TCP handshake, if both parties set the MP_CAPABLE option on their handshake packets.

That leaves me a little unsure as to how I would decide between using MPTCP and TCP if presented in the way you've proposed here. If I take the RFC introductory content at face value, it appears that I should change all of my programs to unconditionally specify "mptcp" instead of "tcp" and never use "tcp" again moving forward. However, I see that the Linux kernel implementation is opt-in, so I have to assume the decision criteria are not as obvious as a naive reading of the RFC content would suggest.

I imagine there's a hypothetical alternative proposal here of making net.Dial("tcp", "golang.org:80") unconditionally set IPPROTO_MPTCP if it can somehow prove that it's running on a kernel which has support for that protocol, and then leave the kernel to negotiate subflows where possible or just use regular TCP if that's not possible or applicable. This is perhaps analogous to how Go's net/http package automatically selects HTTP 2 when possible, but uses HTTP 1.1 otherwise.

Can you comment on what the drawbacks might be of that alternative design?

Thanks!

@mengzhuo
Copy link
Contributor

mengzhuo commented Nov 4, 2022

cc @neild

@matttbe
Copy link
Author

matttbe commented Nov 7, 2022

Hi @apparentlymart

Thank you for your reply!

The introductory material for Multipart TCP in the RFC describes it as being a backward-compatible extension to TCP, capable of interacting with existing TCP-only implementations and able to "progressive-enhance" (to borrow terminology from the web platform) a TCP connection into a MPTCP connection by negotiating subflows after an initial standard TCP handshake, if both parties set the MP_CAPABLE option on their handshake packets.

That's correct but I prefer to add a few more precisions:

  • MPTCP is indeed an extension to TCP: technically, some bytes specific to MPTCP are added in the TCP options field.
  • After the 3-way handshake and with the current implementation, some MPTCP options will still be present in the following packets, even if you are only using one path: it means that compared to TCP, a few bytes are added in each packet (when you transfer a lot of data, the overhead is only around 1%).
  • Fallback to TCP is supported:
    • if the client doesn't support MPTCP, the server will reply with regular TCP packets
    • if the server doesn't support MPTCP, by design, it will ignore MPTCP options added in the first packet by the client and reply with regular TCP packets. The client will no longer add MPTCP options in the following packets attached to this connection
    • intermediate devices between the client and the server are supposed to forward the packets and not cause troubles but some middleboxes on the Internet might modify the connection and interfere with MPTCP, e.g. firewall can drop MPTCP options, transparent proxy can duplicate them but some can be problematic for MPTCP like injecting data without modifying MPTCP data sequence numbers (a NAT modifying packets from old protocols like FTP) and blocking the whole connection (very rare).
    • Most scenarios are covered but still, if a stupid annoying firewall decides to block the whole connection if it sees MPTCP option (I don't know why it would do that instead of removing the option but well), this will impact the user experience.

Please note that Apple has been using MPTCP for some apps since 2013. They started with Siri to better support handovers. Today they are still using it with more apps like Apple Music and Map. So it means MPTCP can survive on the wild Internet.

https://en.wikipedia.org/wiki/MPTCP
http://blog.multipath-tcp.org/blog/html/2018/12/15/apple_and_multipath_tcp.html
https://www.tessares.net/apples-mptcp-story-so-far/
https://www.tessares.net/technology/mptcp/

That leaves me a little unsure as to how I would decide between using MPTCP and TCP if presented in the way you've proposed here. If I take the RFC introductory content at face value, it appears that I should change all of my programs to unconditionally specify "mptcp" instead of "tcp" and never use "tcp" again moving forward. However, I see that the Linux kernel implementation is opt-in, so I have to assume the decision criteria are not as obvious as a naive reading of the RFC content would suggest.

On top of the possible (rare) network issues and the few additional bytes carried by each packet, there are a few additional points to raise here:

  • MPTCP in the Linux kernel implementation is indeed opt-in. It was a requirement to get MPTCP in the Upstream Linux kernel and there are multiple reasons:
    • The implementation had to be incremental: e.g. in Linux v5.6, only one subflow could be created and many socket options were not supported. New features were added progressively (changelog) but what I mean is that it would have broken many apps if from one version to another, all TCP connections were in fact MPTCP ones with limited support at that time.
    • TCP in the Linux kernel is highly optimised: on one hand, you don't want to introduce instabilities by adding new features turned on by default ; on the other hand, any extra layer somehow impacts performances, even if it is under 1%.
    • Each modification in the TCP code had to be motivated with a proof that was not impacting the performance for "plain" TCP sockets (at least in the "fast" path).
    • In other words, TCP will always perform better than MPTCP with a single flow: that's important for some use cases, not for all (most?). But of course, MPTCP generally compensates that by being able to use more paths.
    • One last thing is that because we add more features, there is always a risk of discovering new issues or new "misuses" including security issues. From a security point of view, the protocol has been deeply analysed. Even if fixes can come quickly, there can be some issues in the code and that's another reason why network maintainer didn't want to have MPTCP enabled by default.
  • Some (very) specific features available with TCP sockets are maybe missing when MPTCP is used: for example, there is a huge variety of socket option ([gs]etsockopt()), and some might be missing depending on the kernel you are using, e.g. TCP FASTOPEN is being implemented now (client part available in kernel 6.1), TCP ZeroCopy is not supported yet (quite specific), TCP MD5 cannot work with MPTCP (who cares? :) ), etc. → those are very specific to some use cases, not needed for more common use cases (e.g. HTTP) but still important to keep in mind.
  • That's not really an issue but you need to configure the kernel to use multiple endpoints (IP addresses): some tools are starting to configure that automatically (e.g. Network Manager) but if you do want to use multiple paths, it is important to remember this bit. And of course, like any configurations, it has to be done properly, e.g. the user should not tell the kernel it can use a costly path (e.g. cellular network) for all connections for example.

In short, it might still be needed for some people to ask for "plain" TCP sockets (without MPTCP).

I imagine there's a hypothetical alternative proposal here of making net.Dial("tcp", "golang.org:80") unconditionally set IPPROTO_MPTCP if it can somehow prove that it's running on a kernel which has support for that protocol, and then leave the kernel to negotiate subflows where possible or just use regular TCP if that's not possible or applicable. This is perhaps analogous to how Go's net/http package automatically selects HTTP 2 when possible, but uses HTTP 1.1 otherwise.

It would be really great to have MPTCP more widely used by setting it by default and I think it would help to improve handover use cases or to get more bandwidth for example. But from my point of view, I think we should have the possibility to use MPTCP or not when we ask to create "a basic socket" (net.Dial("mptcp", "...")).

On the other hand, libraries built on top of it could decide to use MPTCP by default (on Linux) and retry with plain TCP if the connection is rejected for example or if MPTCP is not supported by the kernel (errno set to ENOPROTOOPT (Protocol not available) or EPROTONOSUPPORT (Protocol not supported)). Some useful functions could be available in net, e.g. to create a TCP socket if it is not possible to create an MPTCP one, to check if the created socket is a TCP/MPTCP one, to check if later on, the connection fell back to TCP, etc.

Please also note that in the Linux kernel, MPTCP "listening" sockets will create "plain" "accepted" TCP sockets if the connection request (initial TCP SYN) didn't contain any MPTCP options. In other words, from a performance/impact point of view, it makes sense to enable MPTCP support for the server side "by default": it will only be used if the client asks for it.

One last thing, Apple proved the protocol works on the Internet, it should probably be used by default by more libs/apps and net/http is probably a good candidate! Before, net API should of course allow users to create MPTCP sockets.

I hope this answers your questions! Do not hesitate to ask more if you have any others.

@apparentlymart
Copy link

apparentlymart commented Nov 8, 2022

Thanks for those details, @matttbe!

Based on what you've shared -- particularly that MPTCP seems compatible enough with TCP that it's been successfully deployed for production systems like Apple's -- I think I'd personally prefer a design which by default lets the system choose what it deems to be the best available protocol, but offers a way to constrain more tightly as an alternative for those who need more control. This would then echo the design of net/http with regard to HTTP 2 and other successor protocols which are mechanically different than but conceptually compatible with HTTP, so that Go programs would immediately gain support for MPTCP when talking with an MPTCP-supporting peer.

I suppose I can see the argument that if someone is programming at the level of sockets (net) rather than using an application-layer client (like net/http) then it is justified to expose these low-level details. But I also note this common pattern of just forcing arbitrary existing libc-based software to use MPTCP using LD_PRELOAD tricks, which puts that decision outside of the direct control of the developer of that software, suggesting to me that it's typically fine to just swap out TCP for MPTCP in existing software without causing any significant problems.

I must admit though that I'm coming at this from the perspective of someone who only occasionally deals with sockets directly in my Go programs, and so my position might not be shared by those who more frequently do low-level protocol implementation in terms of sockets. With that in mind, I'll bow out of the discussion at this point to make space for others to share different perspectives.

Thanks again for the detailed response!

@rsc
Copy link
Contributor

rsc commented Nov 9, 2022

This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.
— rsc for the proposal review group

@rsc
Copy link
Contributor

rsc commented Nov 9, 2022

This does feel a lot like adding support for HTTP/2 or TLS 1.3, both of which did not require users to modify their programs to get the newer, better protocols. It seems like the right end state is for Go programs to "do the right thing" and use MPTCP whenever that makes sense in place of plain TCP for dialed "tcp" connections. Programs shouldn't have to update to take advantage of the new protocol. Of course we don't want to break people either, if MPTCP would break them. See my recent Compatibility talk for more of my thinking about these kinds of changes.

I am inclined to say that we add MPTCP as an opt-in for a release or two, and then make it opt-out. There should be a GODEBUG setting and possibly also a source-code mechanism, probably for both listening and dialing. I'm not sure what the source code mechanism would be. Maybe a field in net.Dialer and net.ListenConfig, but it seems a little out-of-place, and the field would need to be more than just a single bool, because you need three values: force on, force off, default.

@Jorropo
Copy link
Contributor

Jorropo commented Nov 9, 2022

@rsc given that this would be implemented in kernel and might be disabled, it would be more like copy_file_range for os.(*File).WriteTo where the runtime would first parse kernel fields (/sys/... afait) to check of MPTCP is enabled and then use it if available ?

@ianlancetaylor
Copy link
Contributor

ianlancetaylor commented Nov 9, 2022

@Jorropo More likely we would first try to use IPPROTO_MPTCP and if that returns ENOSYS or EINVAL fall back to using IPPROTO_TCP.

@rsc
Copy link
Contributor

rsc commented Nov 10, 2022

@Jorropo, for the case where we are holding an fd in a net.Conn and don't know whether it's TCP or MPTCP and want to do some optimized code path like sendfile(2) that only supports TCP, it seems like we would just call sendfile and fall back to generic code if sendfile returns an error. We already do that fallback today in code that uses sendfile and copy_file_range. I don't think we'd have to update anything, but if we do it wouldn't be much.

@matttbe
Copy link
Author

matttbe commented Nov 10, 2022

@rsc Thank you for having added this ticket to the proposals project!

I am inclined to say that we add MPTCP as an opt-in for a release or two, and then make it opt-out. There should be a GODEBUG setting and possibly also a source-code mechanism, probably for both listening and dialing. I'm not sure what the source code mechanism would be. Maybe a field in net.Dialer and net.ListenConfig, but it seems a little out-of-place, and the field would need to be more than just a single bool, because you need three values: force on, force off, default.

That looks like a good plan!

@Jorropo More likely we would first try to use IPPROTO_MPTCP and if that returns ENOSYS or EINVAL fall back to using IPPROTO_TCP.

@ianlancetaylor Yes, that's probably the easiest solution.

Please note that if I'm not mistaken, there can be 3 different errors:

  • ENOPROTOOPT 92 Protocol not available: if MPTCP has been disabled with sysctl net.mptcp.enabled=0 for example
  • EPROTONOSUPPORT 93 Protocol not supported: if MPTCP option has not been is not compiled on kernel >=v5.6
  • EINVAL 22 Invalid argument: if MPTCP is not available on kernels <5.6
  • maybe others if MPTCP is blocked by custom "rules" from SELinux, eBPF, etc.

If you get EPROTONOSUPPORT or EINVAL the first time, it means MPTCP will never be possible with the current kernel. It is then possible to have an optimisation there and directly use IPPROTO_TCP for the next sockets.

@neild
Copy link
Contributor

neild commented Nov 10, 2022

Maybe a field in net.Dialer and net.ListenConfig, but it seems a little out-of-place, and the field would need to be more than just a single bool, because you need three values: force on, force off, default.

I don't see any place other than net.Dialer and net.ListenConfig to put the option. These structs already have the Control callback func field; changing the socket protocol seems analogous.

Are we willing to add unexported fields to these types? If so, we could add methods to enable/disable MPTCP. That might be simpler to use than a tristate field.

func (d *Dialer) EnableMultipathTCP(enabled bool)
func (lc *ListenConfig) EnableMultipathTCP(enabled bool)

@rsc
Copy link
Contributor

rsc commented Nov 16, 2022

EnableMPTCP(false) reads funny; it should probably be SetMPTCP.
Otherwise this sounds fine.
The method approach is nice.

@neild
Copy link
Contributor

neild commented Nov 16, 2022

Accessors as well?

func (*Dialer) MPTCP() bool
func (*ListenConfig) MPTCP() bool

I've got a mild preference for spelling out MultipathTCP; https://www.multipath-tcp.org/ mostly uses "Multipath TCP" in prose and only occasionally abbreviates to MPTCP, but the abbreviation is fine as well.

@matttbe
Copy link
Author

matttbe commented Nov 17, 2022

I've got a mild preference for spelling out MultipathTCP; https://www.multipath-tcp.org/ mostly uses "Multipath TCP" in prose and only occasionally abbreviates to MPTCP, but the abbreviation is fine as well.

@neild : I don't have any preferences nor recommendations. All I can say is that in the Linux kernel, we are usually referring to MPTCP (IPPROTO_MPTCP, TCPOPT_MPTCP, SOL_MPTCP, /proc/sys/net/mptcp/, etc.) but that's mainly to have a short name. In technical documents and paper, we can find both, e.g. Apple's description, Annotated bibliography (pdf), etc.

Note that Apple is also using applemultipath in some programs, e.g. in their OpenSSH's fork. But I would recommend not to just use "multipath" without "TCP" because it is too vague, there are other multipath protocols. On the other hand, I think some people quickly short "Multipath TCP" to "Multipath". When "MPTCP" is used, it is less tempting to short it even more :)

@apparentlymart
Copy link

apparentlymart commented Nov 17, 2022

I see that there is some discussion about optional additional API surface area specifically for Multipath TCP. For example, I see An enhanced socket API for Multipath TCP although I do not have access to the content of the paper, only to this abstract. It seems plausible to me that it's related to IETF draft-hesmans-mptcp-socket-00, given the overlap of authorship, so perhaps that draft is sufficient to understand the shape of the proposed API.

This might be ignorable for now to keep initial scope small, but I raise it in case it might be desirable to reserve API surface area for some future way to dynamically obtain something implementing that new API given an active socket that is already using the Multipart TCP protocol. (I assume that Go would want to expose an idiomatic API equivalent to the lower-level socket API proposed there, rather than requiring direct use of setsockopt and getsockopt, but that may be presumptuous in itself given how early this stuff seems to be.)

@matttbe
Copy link
Author

matttbe commented Nov 18, 2022

I see that there is some discussion about optional additional API surface area specifically for Multipath TCP. For example, I see An enhanced socket API for Multipath TCP although I do not have access to the content of the paper, only to this abstract. It seems plausible to me that it's related to IETF draft-hesmans-mptcp-socket-00, given the overlap of authorship, so perhaps that draft is sufficient to understand the shape of the proposed API.

@apparentlymart these drafts have been written by two colleagues and they were designed for a previous out of tree implementation (kernels <= v5.4, custom kernel), not the one in the "official" upstream Linux kernel (kernels >= v5.6).

In the "upstream" project, we decided to be as transparent and as generic as possible: it means that we try to have a behaviour similar to TCP if that makes sense. In other words, userspace app can continue to set and get socket options as it was doing with TCP. Typically, options are propagated to all subflows, even the new ones later. Of course, this needs to be checked case by case: some options don't make sense for MPTCP (specific to other protocols), some cannot work with MPTCP (e.g. TCP MD5 but that's very specific, typically for routers), some only affect the MPTCP sockets and not the subflows (e.g. poll related ones), some are only for the first subflow (e.g. TFO), etc. There are also new options specific to MPTCP in a dedicated space (SOL_MPTCP), e.g. MPTCP_INFO, similar to TCP_INFO but adapted to MPTCP.

The idea is not to have a complex socket API exposed to userspace but keep it simple and have dedicated daemons handling specific use cases. For example,

  • mptcpd is initially dedicated to act as a userspace path manager (but not limited to).
  • If an application needs something specific to handle socket options per path, it can use eBPF to do that.
  • eBPF will "soon" be used to customise the packet scheduler behaviour (WIP).

If you are interested by the subject, I recommend you to check a recent presentation about that: abstract, slides, video

One last thing: if userspace devs noticed something is missing, it is never too late to introduce new features in the kernel ;-)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
Status: Active
Development

No branches or pull requests

8 participants