Number of subflows is equal to zero when expecting 1.

[vanyingenzi]

Hello,

I'm currently working a MPTCP C extension library for python for MPTCP, mptcplib. For this module, I'm trying to add a function to get the number of subflows used by a socket.

In the current version of the lib the user does do by doing :
[Client.py]

import socket
import mptcplib

def client_program():
    host = socket.gethostname()  # as both code is running on same pc
    port = 5000  # socket server port number

    client_socket = mptcplib.create_mptcp_socket(socket.AF_INET, socket.SOCK_STREAM)
    client_socket.connect((host, port))  # connect to the server

    message = input(" -> ")  # take input

    while message.lower().strip() != 'bye':
        client_socket.send(message.encode())  # send message
        data = client_socket.recv(1024).decode()  # receive response

        print('Received from server: ' + data)  # show in terminal
        message = input(" -> ")  # again take input
    
    print(mptcplib.used_subflows(client_socket.fileno()))
    client_socket.close()  # close the connection


if __name__ == '__main__':
    client_program()

[Server.py]

import socket
import mptcplib

def server_program():
    # get the hostname
    host = socket.gethostname()
    port = 5000

    server_socket = mptcplib.create_mptcp_socket(socket.AF_INET, socket.SOCK_STREAM)
    server_socket.bind((host, port))  

    server_socket.listen(2)
    conn, address = server_socket.accept()  # accept new connection
    print("Connection from: " + str(address))
    while True:
        data = conn.recv(1024).decode()
        if not data:
            break
        print("from connected user: " + str(data))
        data = input(' -> ')
        conn.sendall(data.encode())  # send data to the client

    print(mptcplib.used_subflows(server_socket.fileno()))
    conn.close()  # close the connection

if __name__ == '__main__':
    server_program()

However, when I run these two peers at the end of the execution I get that the number of subflows used is zero. The ip mptcp monitor shows the following logs :

root@ubuntu-nptcp-server:~# ip mptcp monitor
[CREATED] token=a35d7cd1 remid=0 locid=0 saddr4=10.18.0.5 daddr4=10.18.0.5 sport=55866 dport=5000
[CREATED] token=9979ccfe remid=0 locid=0 saddr4=10.18.0.5 daddr4=10.18.0.5 sport=5000 dport=55866
[ESTABLISHED] token=9979ccfe remid=0 locid=0 saddr4=10.18.0.5 daddr4=10.18.0.5 sport=5000 dport=55866
[ESTABLISHED] token=a35d7cd1 remid=0 locid=0 saddr4-10.18.0.5 daddr4=10.18.0.5 sport-55866 dport-5000
[CLOSED] token=a35d7cd1
[CLOSED] token=9979ccfe

The host, hostnamectl :

 Static hostname: ubuntu-mptcp-server
       Icon name: computer-vm
         Chassis: vm
      Machine ID: db5c8284be7730ef5b609c996331e187
         Boot ID: 24990ea64b52474fbe00002069f07d84
  Virtualization: kvm
Operating System: Ubuntu 22.04.1 LTS              
          Kernel: Linux 6.0.0-060000-generic
    Architecture: x86-64

The C code that checks the number of subflows is :

int
Cused_subflows(int sockfd)
{
	int socket_is_mptcp_returned = Csocket_is_mptcp(sockfd);
	if ( socket_is_mptcp_returned == 0 ){
		struct mptcp_info inf;
		socklen_t optlen;
		optlen = sizeof(inf);
		if (!getsockopt(sockfd, SOL_MPTCP, MPTCP_INFO, &inf, &optlen)) {
			return inf.mptcpi_subflows;
		} else {
			return MPTCPLIB_ERROR_FLAG;
		}
	}
	return MPTCPLIB_SOCKET_FALLBACK_TCP;
}

Thank you all for your time. I'd love to give more information if what I gave above is not enough.

[matttbe]

Hello,

Good idea to work on a lib for MPTCP!

Here, you are looking at mptcpi_subflows. This field indicates the number of additional subflow a client has established (or being established). In your case, you don't create any additional subflows, so it is 0. I agree, this is confusing...

A reliable way to get the number of subflows is to use MPTCP_TCPINFO (or MPTCP_SUBFLOW_ADDRS). The struct mptcp_subflow_data structure will be filled by the kernel with the number of subflows. (if you are not interested by the rest, I think you can set 0 for the size_user and just pass the mptcp_subflow_data structure pointer)
Examples:

• MPTCP_TCPINFO:

  mptcp_net-next/tools/testing/selftests/net/mptcp/mptcp_sockopt.c

  Lines 331 to 349 in 8bb20ac

  	static void do_getsockopt_tcp_info(struct so_state *s, int fd, size_t r, size_t w)
  	{
  	struct my_tcp_info {
  	struct mptcp_subflow_data d;
  	struct tcp_info ti[2];
  	} ti;
  	int ret, tries = 5;
  	socklen_t olen;
  	do {
  	memset(&ti, 0, sizeof(ti));
  	ti.d.size_subflow_data = sizeof(struct mptcp_subflow_data);
  	ti.d.size_user = sizeof(struct tcp_info);
  	olen = sizeof(ti);
  	ret = getsockopt(fd, SOL_MPTCP, MPTCP_TCPINFO, &ti, &olen);
  	if (ret < 0)
  	xerror("getsockopt MPTCP_TCPINFO (tries %d, %m)");

• MPTCP_SUBFLOW_ADDRS:

  mptcp_net-next/tools/testing/selftests/net/mptcp/mptcp_sockopt.c

  Lines 381 to 401 in 8bb20ac

  	static void do_getsockopt_subflow_addrs(int fd)
  	{
  	struct sockaddr_storage remote, local;
  	socklen_t olen, rlen, llen;
  	int ret;
  	struct my_addrs {
  	struct mptcp_subflow_data d;
  	struct mptcp_subflow_addrs addr[2];
  	} addrs;
  	memset(&addrs, 0, sizeof(addrs));
  	memset(&local, 0, sizeof(local));
  	memset(&remote, 0, sizeof(remote));
  	addrs.d.size_subflow_data = sizeof(struct mptcp_subflow_data);
  	addrs.d.size_user = sizeof(struct mptcp_subflow_addrs);
  	olen = sizeof(addrs);
  	ret = getsockopt(fd, SOL_MPTCP, MPTCP_SUBFLOW_ADDRS, &addrs, &olen);
  	if (ret < 0)
  	die_perror("getsockopt MPTCP_SUBFLOW_ADDRS");

Out of curiosity, why do you do the getsockopt() calls in C and the rest in Python? Why not doing everything in Python (using struct I guess to parse the buffer written by getsockopt()'s syscall)?

PS: you should avoid doing if (!getsockopt(...)): it is correct but very confusing to use the negation here: it looks like you are handling the error case. If the function returns an int, it is usually recommended to compare it to another int, e.g. == 0. Please note that typically, we more regularly see something like:

if (getsockopt(...) < 0) {
    /* log error */
    return -1;
}

return inf.mptcpi_subflows;

(Also, it is strange to return a NULL "pointer" to return the integer 0)

If this replies to your questions, please close the ticket.

[vanyingenzi]

Hello @matttbe

Thank you very much for your quick reply. Indeed, you do answer my question.

Out of curiosity, why do you do the getsockopt() calls in C and the rest in Python? Why not doing everything in Python (using struct I guess to parse the buffer written by getsockopt()'s syscall)?

=> The reason I do so it's because I thought it would be better to handle most of the calls at the C level since we have more freedom of operations on the socket. In addition, if I'm not mistaken the Linux upstream implementation is not the same on MacOs, so I thought it would be better to handle it at the C level. The goal of the lib is to even propose path management in future releases, and I think it would be harder (even impossible) to achieve only at the Python level. Please correct me if I'm wrong.

Thank you for your P.S it was indeed a mistake from me the function had to return an error flag not null.

[matttbe]

The reason I do so it's because I thought it would be better to handle most of the calls at the C level since we have more freedom of operations on the socket.

Mmh, yes, it depends on the point of view. You can find many libs in Python where they proudly say it is a "pure" Python implementation: it is often easier to maintain (nothing to compile, no external dependences, etc.).

I understand it is easier to do the getsockopt(MPTCP_INFO) in C because you can use the kernel header to get the structures you need. But on the other hand, you will need to maintain the interface between the C API and Python. If you want to access other fields, it might start to get more complex to maintain if you need to add a different function for each field (also it might mean doing a new getsockopt() call each time).

I think it might be more interesting for a longer term to do all of that in Python directly. It will probably be a bit painful to define the struct to match the structure in C but once it is done, it will be easier to extract the info you need and an application in Python will be able to easily access any fields, not just the ones where you added explicit getters (you can still have the getters as helpers of course).

In addition, if I'm not mistaken the Linux upstream implementation is not the same on MacOs, so I thought it would be better to handle it at the C level.

Yes, the API is different. I don't know a lot about the implementation on MacOSx, I don't even know if you can use their API with C (AF_MULTIPATH was not in the SDK from what I remembered): you might need to use NSURLSession in Swift or Objective-C but I don't know more about that.

The goal of the lib is to even propose path management in future releases, and I think it would be harder (even impossible) to achieve only at the Python level.

It would not be difficult, you need to talk to the kernel using Netlink and some specific structure. If you want to manage the "in-kernel" path-manager, doing that in Python is perfectly fine to me (I guess there are Netlink libs in Python to ease that part) and there is not a lot to discuss with the kernel, just the configuration bit you can already do with ip mptcp (but having an interface in Python might still be useful).

If you want to manage the "userspace" path-manager with hundreds/thousands of connections, it might not be a good idea to do that in Python. (mptcpd might be better for that ; also you need extra permissions to do that ; mptcpd can also be extended to be use as a lib for C programs → and why not using this extension with your lib :) )

But please note that managing the userspace PM is quite a different/bigger job than providing info to the application: the userspace PM is controlled with Netlink while what you are doing is around the socket API. I might prefer to say: if something is missing in mptcpd, first check if it would not be better to extend it than rewriting it :)

[vanyingenzi]

Hello @matttbe,

I can't thank you enough for your insight. Your feedback will be added to the lib, and it is indeed a better route to take.