MPTCP Linux Kernel specifics

[ddimopou]

Hello,

I have some fundamental questions, in case somebody could provide any hints.

I'm not experienced with the Linux kernel code base, but am trying to understand how the MPTCP schedulers and Path Manager are implemented. I may understand the algorithmic part (i.e. how the MPTCP default minRTT scheduler operates), but when it comes to the implementation in the Linux kernel, it is a totally different story.

I need to understand what all the different structures are used for and get a notion of them, i.e. the mpcb, the tcp_sock vs. tcp_sk and so on. I have started with the sk_buff basics and read some of the fundamentals of the Networking sub-module in the Linux Kernel from the book: "The Linux Networking Architecture: Design and Implementation of Network Protocols in the Linux Kernel".

Q1: Is there any source of high-level information, i.e. any architecture diagram or anything else, depicting in a visual or even written format, the different structures used within MPTCP? I am interested in customizing the schedulers and also get a knowledge of how the path manager is implemented, in order to perform my own customizations.

Q2: I am looking at a paper which makes some modifications in the path manager to perform a delayed subflow establishment, and suspend or resume a subflow under specific conditions. I'm trying to find out how such a strategy could be implemented in the PM and which is the appropriate file the PM is implemented in (if it is only one or split into more). Is the PM solely implemented within "mptcp_pm.c"?

Q3: Another question is how the PM gets knowledge of the interfaces available underneath (i.e. in case of mobile devices, how does the PM differentiate the subflow establishment between the LTE and WiFi interfaces? -- does the PM see interfaces or is only aware of IP addresses/ports corresponding to the interfaces and is agnostic of the interface per se?)

I understand this may not be an appropriate place to ask for this kind of guidance, but could not find more specific info on the MPTCP implementation specifics in the Linux kernel during months of study.

Any assistance/guidance is welcome.
Thanks in advance.

Dimitris

p.s.: I have deployed MPTCP v0.96 (linux kernel 5.4.230.mptcp) in a VBox VM running Ubuntu 22.04, which is functional and am able to perform some basic aggregation tests using Mininet.

[matttbe]

Hello,

Q1: Is there any source of high-level information, i.e. any architecture diagram or anything else, depicting in a visual or even written format, the different structures used within MPTCP? I am interested in customizing the schedulers and also get a knowledge of how the path manager is implemented, in order to perform my own customizations.

I don't think such doc exists. The architecture has been presented in some papers (e.g. here), but at the end, this implementation is deprecated, we do recommend looking at the one in the official Linux kernel.

Q2: I am looking at a paper which makes some modifications in the path manager to perform a delayed subflow establishment, and suspend or resume a subflow under specific conditions. I'm trying to find out how such a strategy could be implemented in the PM and which is the appropriate file the PM is implemented in (if it is only one or split into more). Is the PM solely implemented within "mptcp_pm.c"?

Probably easier to implement such path-manager in userspace, e.g. by using mptcpd. mptcpd still supports both kernels the old deprecated one (here), and the new one depending on how it is compiled. In the old implementation, there are different PMs: fullmesh, ndiffport, netlink (userspace PM), etc. each in a dedicated .c file.

Q3: Another question is how the PM gets knowledge of the interfaces available underneath (i.e. in case of mobile devices, how does the PM differentiate the subflow establishment between the LTE and WiFi interfaces? -- does the PM see interfaces or is only aware of IP addresses/ports corresponding to the interfaces and is agnostic of the interface per se?)

It doesn't. The user has to provide such info to the path-manager if it supports it. e.g. the fullmesh PM can have addresses marked as "backup". For the userspace PM, you do whatever you want.

---

In short, if you are interested by MPTCP today, please look at the implementations from a recent Linux version, e.g. from a >= v6.6 kernel (or maybe v6.1, but not older)

[ddimopou]

Thank you for the prompt response and the hints.

To be honest, I was not sure to what extent the new upstream MPTCP version was mature in terms of performance evaluation compared to the out-of-tree version. I had read some older posts on performance issues which might be still open in the upstream version, that's why opted for the out-of-tree.

In general, I'm interested in experiments with the different schedulers, like BLEST, ECF, redundant (that the older version offers), and customizing the schedulers and the PM. If I properly understand, they are not yet in the new version (here), and I'm not sure how difficult it would be for somebody to port them in the upstream version for experimental purposes.

If you have any view on this, it would help me decide whether it is better to move on with the new version.
Thanks once again for you time on this.

Dimitris

[matttbe]

To be honest, I was not sure to what extent the new upstream MPTCP version was mature in terms of performance evaluation compared to the out-of-tree version. I had read some older posts on performance issues which might be still open in the upstream version, that's why opted for the out-of-tree.

The new upstream implementation is maturing, there are improvements in each Linux kernel version: https://github.com/multipath-tcp/mptcp_net-next/wiki#changelog

It is then important to use a recent version, and report what is not working properly.

In general, I'm interested in experiments with the different schedulers, like BLEST, ECF, redundant (that the older version offers), and customizing the schedulers and the PM. If I properly understand, they are not yet in the new version (here), and I'm not sure how difficult it would be for somebody to port them in the upstream version for experimental purposes.

In the upstream version, there are two PMs: the in-kernel one to support most use-cases and the userspace PM for the rest. There is currently one packet scheduler, that can be tweaked a bit via sysctl options, but that's it for the moment. The idea is to be able to modify the behaviour using eBPF. The work is in progress, with some PoC, but currently blocked by an architectural change.

If you have any view on this, it would help me decide whether it is better to move on with the new version.

For some experimentations, sure you can use this out-of-tree implementation. If you plan to use it in the long term, or you want to contribute by improving something, probably best to look at the upstream version.

[ddimopou]

Thank you for your guidance and your work on improving MPTCP.

I'm planning to try the new version at some point and come back in case of questions.
Closing the ticket for now.