Problem with IPROTO

[AlejandraOliver]

Hi, I am trying to get my Linux machine to use MPTCP socket instead of TCP (I'm using a kernel from the export branch). So far I have been running mptcpize run iperf3 to test for MPTCP. Now I want to run netperf but I think this tool doesn't work with mptcize. Does anyone have the .stap application I need to create to have MPTCP sockets and don't need to use mptcpize?
I have used

#include <linux/ip.h>
%}

/* according to [1], RSI contains 'type' and RDX
 * contains 'protocol'.
 * [1] https://github.com/torvalds/linux/blob/master/arch/x86/entry/entry_64.S#L79
 */

function mptcpify () %{
    if (CONTEXT->kregs->si == SOCK_STREAM &&
        (CONTEXT->kregs->dx == IPPROTO_TCP ||
         CONTEXT->kregs->dx == 0)) {
                CONTEXT->kregs->dx = IPPROTO_MPTCP;
                STAP_RETVALUE = 1;
    } else {
           STAP_RETVALUE = 0;
    }
%}

probe kernel.function("__sys_socket") {
        if (mptcpify() == 1) {
                printf("command %16s mptcpified\n", execname());
        }
}

But I get an error:

Pass 1: parsed user script and 480 library scripts using 146580virt/113408res/11520shr/101264data kb, in 260usr/80sys/344real ms.
Pass 2: analyzed script: 1 probe, 2 functions, 1 embed, 0 globals using 243180virt/210936res/12732shr/197864data kb, in 1480usr/320sys/2498real ms.
Pass 3: translated to C into "/tmp/staprkn0Ms/stap_f28dfdbfec690cd4ed5aa23c2825ddac_1493_src.c" using 243180virt/211060res/12856shr/197864data kb, in 20usr/0sys/15real ms.
In file included from /usr/share/systemtap/runtime/linux/task_finder.c:17,
                 from /usr/share/systemtap/runtime/linux/runtime.h:275,
                 from /usr/share/systemtap/runtime/runtime.h:26,
                 from /tmp/staprkn0Ms/stap_f28dfdbfec690cd4ed5aa23c2825ddac_1493_src.c:21:
/usr/share/systemtap/runtime/linux/task_finder2.c: In function ‘__stp_call_mmap_callbacks_for_task’:
/usr/share/systemtap/runtime/linux/task_finder2.c:1289:17: error: ‘struct mm_struct’ has no member named ‘mmap’
 1289 |         vma = mm->mmap;
      |                 ^~
/usr/share/systemtap/runtime/linux/task_finder2.c:1293:26: error: ‘struct vm_area_struct’ has no member named ‘vm_next’
 1293 |                 vma = vma->vm_next;
      |                          ^~
/usr/share/systemtap/runtime/linux/task_finder2.c:1302:25: error: ‘struct mm_struct’ has no member named ‘mmap’
 1302 |                 vma = mm->mmap;
      |                         ^~
/usr/share/systemtap/runtime/linux/task_finder2.c:1330:34: error: ‘struct vm_area_struct’ has no member named ‘vm_next’
 1330 |                         vma = vma->vm_next;
      |                                  ^~
In file included from /usr/share/systemtap/runtime/transport/procfs.c:13,
                 from /usr/share/systemtap/runtime/transport/transport.c:72,
                 from /usr/share/systemtap/runtime/linux/print.c:17,
                 from /usr/share/systemtap/runtime/print.c:17,
                 from /usr/share/systemtap/runtime/runtime_context.h:22,
                 from /tmp/staprkn0Ms/stap_f28dfdbfec690cd4ed5aa23c2825ddac_1493_src.c:68:
/usr/share/systemtap/runtime/proc_fs_compatibility.h: At top level:
/usr/share/systemtap/runtime/proc_fs_compatibility.h:29:13: error: static declaration of ‘proc_set_user’ follows non-static declaration
   29 | static void proc_set_user(struct proc_dir_entry *de, kuid_t uid, kgid_t gid)
      |             ^~~~~~~~~~~~~
In file included from /usr/share/systemtap/runtime/linux/runtime.h:22,
                 from /usr/share/systemtap/runtime/runtime.h:26,
                 from /tmp/staprkn0Ms/stap_f28dfdbfec690cd4ed5aa23c2825ddac_1493_src.c:21:
./include/linux/proc_fs.h:112:13: note: previous declaration of ‘proc_set_user’ with type ‘void(struct proc_dir_entry *, kuid_t,  kgid_t)’
  112 | extern void proc_set_user(struct proc_dir_entry *, kuid_t, kgid_t);
      |             ^~~~~~~~~~~~~
In file included from /usr/share/systemtap/runtime/transport/procfs.c:13,
                 from /usr/share/systemtap/runtime/transport/transport.c:72,
                 from /usr/share/systemtap/runtime/linux/print.c:17,
                 from /usr/share/systemtap/runtime/print.c:17,
                 from /usr/share/systemtap/runtime/runtime_context.h:22,
                 from /tmp/staprkn0Ms/stap_f28dfdbfec690cd4ed5aa23c2825ddac_1493_src.c:68:
/usr/share/systemtap/runtime/proc_fs_compatibility.h: In function ‘proc_set_user’:
/usr/share/systemtap/runtime/proc_fs_compatibility.h:31:11: error: invalid use of undefined type ‘struct proc_dir_entry’
   31 |         de->uid = __kuid_val(uid);
      |           ^~
/usr/share/systemtap/runtime/proc_fs_compatibility.h:32:11: error: invalid use of undefined type ‘struct proc_dir_entry’
   32 |         de->gid = __kgid_val(gid);
      |           ^~
/usr/share/systemtap/runtime/proc_fs_compatibility.h: At top level:
/usr/share/systemtap/runtime/proc_fs_compatibility.h:47:13: error: static declaration of ‘proc_remove’ follows non-static declaration
   47 | static void proc_remove(struct proc_dir_entry *de)
      |             ^~~~~~~~~~~
In file included from /usr/share/systemtap/runtime/linux/runtime.h:22,
                 from /usr/share/systemtap/runtime/runtime.h:26,
                 from /tmp/staprkn0Ms/stap_f28dfdbfec690cd4ed5aa23c2825ddac_1493_src.c:21:
./include/linux/proc_fs.h:124:13: note: previous declaration of ‘proc_remove’ with type ‘void(struct proc_dir_entry *)’
  124 | extern void proc_remove(struct proc_dir_entry *);
      |             ^~~~~~~~~~~
In file included from /usr/share/systemtap/runtime/transport/procfs.c:13,
                 from /usr/share/systemtap/runtime/transport/transport.c:72,
                 from /usr/share/systemtap/runtime/linux/print.c:17,
                 from /usr/share/systemtap/runtime/print.c:17,
                 from /usr/share/systemtap/runtime/runtime_context.h:22,
                 from /tmp/staprkn0Ms/stap_f28dfdbfec690cd4ed5aa23c2825ddac_1493_src.c:68:
/usr/share/systemtap/runtime/proc_fs_compatibility.h: In function ‘proc_remove’:
/usr/share/systemtap/runtime/proc_fs_compatibility.h:49:21: error: invalid use of undefined type ‘struct proc_dir_entry’
   49 |         if (de && de->subdir == NULL) {
      |                     ^~
/usr/share/systemtap/runtime/proc_fs_compatibility.h:50:36: error: invalid use of undefined type ‘struct proc_dir_entry’
   50 |                 if (atomic_read(&de->count) != LAST_ENTRY_COUNT)
      |                                    ^~
In file included from ./include/linux/kernel.h:29,
                 from ./arch/x86/include/asm/percpu.h:27,
                 from ./arch/x86/include/asm/preempt.h:6,
                 from ./include/linux/preempt.h:78,
                 from ./include/linux/spinlock.h:56,
                 from ./include/linux/mmzone.h:8,
                 from ./include/linux/gfp.h:7,
                 from /usr/share/systemtap/runtime/linux/runtime_defines.h:20,
                 from /usr/share/systemtap/runtime/runtime_defines.h:8,
                 from /tmp/staprkn0Ms/stap_f28dfdbfec690cd4ed5aa23c2825ddac_1493_src.c:12:
/usr/share/systemtap/runtime/proc_fs_compatibility.h:54:34: error: invalid use of undefined type ‘struct proc_dir_entry’
   54 |                                de->name);
      |                                  ^~
./include/linux/printk.h:427:33: note: in definition of macro ‘printk_index_wrap’
  427 |                 _p_func(_fmt, ##__VA_ARGS__);                           \
      |                                 ^~~~~~~~~~~
/usr/share/systemtap/runtime/proc_fs_compatibility.h:51:25: note: in expansion of macro ‘printk’
   51 |                         printk(KERN_ERR "Removal of %s from /proc"
      |                         ^~~~~~
In file included from /usr/share/systemtap/runtime/transport/procfs.c:13,
                 from /usr/share/systemtap/runtime/transport/transport.c:72,
                 from /usr/share/systemtap/runtime/linux/print.c:17,
                 from /usr/share/systemtap/runtime/print.c:17,
                 from /usr/share/systemtap/runtime/runtime_context.h:22,
                 from /tmp/staprkn0Ms/stap_f28dfdbfec690cd4ed5aa23c2825ddac_1493_src.c:68:
/usr/share/systemtap/runtime/proc_fs_compatibility.h:55:37: error: invalid use of undefined type ‘struct proc_dir_entry’
   55 |                 remove_proc_entry(de->name, de->parent);
      |                                     ^~
/usr/share/systemtap/runtime/proc_fs_compatibility.h:55:47: error: invalid use of undefined type ‘struct proc_dir_entry’
   55 |                 remove_proc_entry(de->name, de->parent);
      |                                               ^~
/tmp/staprkn0Ms/stap_f28dfdbfec690cd4ed5aa23c2825ddac_1493_src.c: In function ‘function___global_mptcpify__overload_0’:
/tmp/staprkn0Ms/stap_f28dfdbfec690cd4ed5aa23c2825ddac_1493_src.c:301:31: error: ‘SOCK_STREAM’ undeclared (first use in this function); did you mean ‘LOCK_READ’?
  301 |     if (CONTEXT->kregs->si == SOCK_STREAM &&
      |                               ^~~~~~~~~~~
      |                               LOCK_READ
/tmp/staprkn0Ms/stap_f28dfdbfec690cd4ed5aa23c2825ddac_1493_src.c:301:31: note: each undeclared identifier is reported only once for each function it appears in
In file included from /usr/share/systemtap/runtime/linux/../regs.c:16,
                 from /usr/share/systemtap/runtime/linux/runtime.h:270,
                 from /usr/share/systemtap/runtime/runtime.h:26,
                 from /tmp/staprkn0Ms/stap_f28dfdbfec690cd4ed5aa23c2825ddac_1493_src.c:21:
/tmp/staprkn0Ms/stap_f28dfdbfec690cd4ed5aa23c2825ddac_1493_src.c: In function ‘enter_kretprobe_common’:
/tmp/staprkn0Ms/stap_f28dfdbfec690cd4ed5aa23c2825ddac_1493_src.c:574:45: error: ‘struct kretprobe_instance’ has no member named ‘ret_addr’
  574 |         SET_REG_IP(regs, (unsigned long)inst->ret_addr);
      |                                             ^~
/usr/share/systemtap/runtime/linux/../regs.h:85:44: note: in definition of macro ‘SET_REG_IP’
   85 | #define SET_REG_IP(regs, x) REG_IP(regs) = x
      |                                            ^
make[1]: *** [scripts/Makefile.build:252: /tmp/staprkn0Ms/stap_f28dfdbfec690cd4ed5aa23c2825ddac_1493_src.o] Error 1
make: *** [Makefile:2025: /tmp/staprkn0Ms] Error 2
WARNING: kbuild exited with status: 2
Pass 4: compiled C into "stap_f28dfdbfec690cd4ed5aa23c2825ddac_1493.ko" in 19340usr/4440sys/5264real ms.
Pass 4: compilation failed.  [man error::pass4]
Tip: /usr/share/doc/systemtap/README.Debian should help you get started.

[AlejandraOliver]

I have seen on this comment that the use of IPROTO_MPTCP via systemstap has been stopped (as wontfix) as there is an easier tool which is mptcpize, isn't it?
But here you use a .sh that could be with netperf and force MPTCP. I don't find this .sh.

So, I should continue using mptcpize, right?

[pabeni]

Hi,

This issue mixes a few different topics, let's try to separate them.

1. you state that "this tool [netperf] doesn't work with mptcpize". Why? here I see:

# echo 1 > /proc/sys/net/mptcp/enabled
# mptcpize run netserver
Starting netserver with host 'IN(6)ADDR_ANY' port '12865' and family AF_UNSPEC
# mptcpize run  netperf -H 127.0.0.1 -l 1
MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 127.0.0.1 () p
ort 0 AF_INET
get_transport_info: getsockopt: errno 95
Recv   Send    Send                          
Socket Socket  Message  Elapsed              
Size   Size    Size     Time     Throughput  
bytes  bytes   bytes    secs.    10^6bits/sec  

131072  16384  16384    1.00     14531.77

so the tool works, even if it's unable to fetch some info (get_transport_info: getsockopt: errno 95), but that is a warning not a failure. Note that such warning is caused by the mptcp protocol not supporting the TCP_MAXSEG socket option. We could add such support, but first we should clarify the meaning of such option in case of multiple active subflows. Should we pick min, max, or first subfloow maxseg? I'm wild guessing 'min' should be a good pick.

In anycase netperf is expected to work, please report your setup details and the exact error message if not.

2. Using .stap should produce exact the same results as using mptcpize, because basically they take action in the same way. The error you see why using the stap don't seam related to mptcp but to systemtap itself. Please ensure that stap is functional in your environment, running a simple/dummy example.

3. the mentioned closes issue was about a different method to force existing TCP application to use MPTCP instead. We chose to not implement such method as it was too invasive for the kernel and yealding the same results as mptcpize

4. the shell script you mention is an old tool very alike mptcpize. Since the latter has been packaged and is available out-of-the box in relevant distros, we dropped the first.

TL;DR: netperf should work with mptcpize, please provide more info if you see failures.

[AlejandraOliver]

But, if I use mptcpize with netperf, it doesn't take multiple subflows (with default scheduler), while mptcpize with iperf3 yes.

---

I have checked that my systemtap is working fine and still I keep getting the same error.

[pabeni]

Hi,

But, if I use mptcpize with netperf, it doesn't take multiple subflows (with default scheduler), while mptcpize with iperf3 yes.

I guess you really mean that "mptcpized" netperf does not create multiples subflows, while iperf3 does. That is quite unexpected, as the mptcpize tool only forces the specified application to use the mptcp protocol instead of TCP. Additional subflow creation is always in charge of the Path Manager (thus even the scheduler is irrelevant).

The Path Manager can be configured e.g. via the iproute2 tool, see:

• man ip-mptcp
• https://access.redhat.com/documentation/it-it/red_hat_enterprise_linux/9/html/configuring_and_managing_networking/getting-started-with-multipath-tcp_configuring-and-managing-networking#using-iproute2-to-temporarily-configure-and-enable-multiple-paths-for-mptcp-applications_getting-started-with-multipath-tcp

If the number of additional subflows created by iperf3 and netperf are different, probably something changed in your environment in between the 2 tests. As already asked, please report more information, alike:

• ip mptcp endpoint output before each test
• ip mptcp limits output before each test
• ss -MeinmO output during each tests
• nstat MPTcp* output just after each test
  on both sides (client and server).

Additionally a pcap capture on the relevant interface(s) limited to TCP syn,fin,rst packets could help.

I have checked that my systemtap is working fine and still I keep getting the same error.

Still the error you report above looks more related to systemtop env than the specific mptcp stap. In any case systemtap will provide the same results as mptcpize, and is an obsolete method. Please instead provide the info requested above, thanks!

[AlejandraOliver]

In theory, if the default scheduler is used, the same data should be sent through all subflows as the redundant? Or different data for each one? Using mptcpize with netperf?

[matttbe]

In theory, if the default scheduler is used, the same data should be sent through all subflows as the redundant? Or different data for each one?

The default scheduler should use all available paths in a way to reach the best throughput.
It should use the backup paths only if there is no more non-backup paths (or if these non-backup paths are considered as stale / unusable).

Using mptcpize with netperf?

mptcpize and systemstap will only force an application to create MPTCP connections instead of "plain" TCP ones. It doesn't influence the path-manager or the packets scheduler which is configured on the system via sysctl.

EDIT: you can have a situation where multiple paths are created (or it tries to create multiple ones) but it only uses one for some reasons. It is always important to check the counters, limits and endpoints to know what's happening as mentioned by Paolo.

[AlejandraOliver]

The default scheduler should use all available paths in a way to reach the best throughput

So, It's like redundant?

[matttbe]

So, It's like redundant?

No, redundant is supposed to duplicate the data over the different paths in order to minimise delay in case of issue with one link. (and probably not a good idea to use this "redundant" technique with MPTCP except maybe for very specific use-cases and coupled with a dedicated path-manager recreating bad paths very quickly)