Skip to content

feat(output): display MPTCP options#152

Merged
mozillazg merged 5 commits into
masterfrom
mptcp
Oct 3, 2024
Merged

feat(output): display MPTCP options#152
mozillazg merged 5 commits into
masterfrom
mptcp

Conversation

@mozillazg
Copy link
Copy Markdown
Owner

@mozillazg mozillazg commented Sep 28, 2024

Closes #148

02:55:35.698856 IP (...).34204 > (...).12345: Flags [S], seq 412382786, win 65495, options [mss 65495,sackOK,TS val 642407978 ecr 0,nop,wscale 7,mptcp 4 capable v1 flags [H]], length 0
02:55:35.698936 IP (...).12345 > (...).34204: Flags [S.], seq 2707981557, ack 412382787, win 65483, options [mss 65495,sackOK,TS val 642407978 ecr 642407978,nop,wscale 7,mptcp 12 capable v1 flags [H] {0xe356a8f252e41976}], length 0
02:55:35.699025 IP (...).34204 > (...).12345: Flags [.], seq 412382787, ack 2707981558, win 512, options [nop,nop,TS val 642407978 ecr 642407978,mptcp 20 capable v1 flags [H] {0xfe7b93dc894dcb73,0xe356a8f252e41976}], length 0
02:55:38.509583 IP (...).34204 > (...).12345: Flags [P.], seq 412382787:412382792, ack 2707981558, win 512, options [nop,nop,TS val 642410789 ecr 642407978,mptcp 22 capable v1 flags [H] {0xfe7b93dc894dcb73,0xe356a8f252e41976,data_len=5},nop,nop], length 5

@mozillazg
Copy link
Copy Markdown
Owner Author

mozillazg commented Sep 28, 2024

cc @matttbe

@matttbe
Copy link
Copy Markdown

matttbe commented Sep 28, 2024

@mozillazg wow, thank you very much for this big work! I didn't realise there was so much work to do on the pktdump side. Congrats!

It looks good to me. I will try to run it locally with various MPTCP traffic.

One last thing: I noticed you added a commit in the gopacket repo: I sent two questions there.

@matttbe
Copy link
Copy Markdown

matttbe commented Sep 28, 2024

I just tried, using the binary from: https://github.com/mozillazg/ptcpdump/actions/runs/11082221375

I had this error:

2024-09-28 16:36:30 WARN skip Docker Engine integration due to Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
2024-09-28 16:36:32 WARN skip containerd integration due to failed to dial "/run/containerd/containerd.sock": context deadline exceeded
2024-09-28 16:36:32 WARN skip kubernetes integration due to [connect using endpoint /var/run/dockershim.sock: no such file or directory, connect using endpoint /var/run/cri-dockerd.sock: no such file or directory, connect using endpoint /run/crio/crio.sock: no such file or directory, connect using endpoint /run/containerd/containerd.sock: no such file or directory]
2024-09-28 16:36:32 WARN skip attach cgroup due to get cgroup v2 root dir failed: cgroupv2 is not mounted
2024-09-28 16:36:32 WARN ptcpdump: verbose output suppressed, use -v[v]... for verbose output
2024-09-28 16:36:32 WARN capturing on any, link-type EN10MB (Ethernet), snapshot length 262144 bytes
16:36:32.588920 ns1eth3 In IP6 fe80::444e:7eff:fec7:8073 > ff02::16: ICMP6, length 56
16:36:32.731637 ns1eth2 Out IP6 fe80::94ce:82ff:fea1:5cad > ff02::16: ICMP6, length 56
16:36:33.115589 ns1eth4 In IP6 fe80::642c:8cff:fe14:2a5e > ff02::16: ICMP6, length 56
16:36:33.118219 ns1eth3 Out IP6 fe80::1463:b3ff:fe46:e3a1 > ff02::16: ICMP6, length 56
16:36:38.050570 ns1eth1 In ARP, Request who-has 10.0.1.1 tell 10.0.1.2, length 28
16:36:38.050685 ns1eth1 Out ARP, Reply 10.0.1.1 is-at 8e:01:f2:3e:03:d1, length 28
16:36:38.050880 ns1eth1 mptcp_connect.5253 In IP 10.0.1.2.54534 > 10.0.1.1.10004: Flags [S], seq 3642944204, win 65535, options [mss 1460,sackOK,TS val 1352303447 ecr 0,nop,wscale 8,mptcp 4 capable v1 flags [H]], length 0, ParentProc [timeout.5252]
16:36:38.050997 ns1eth1 mptcp_connect.5253 Out IP 10.0.1.1.10004 > 10.0.1.2.54534: Flags [S.], seq 1598583134, ack 3642944205, win 65535, options [mss 1460,sackOK,TS val 4096278096 ecr 1352303447,nop,wscale 8,mptcp 12 capable v1 flags [H] {0xf28be19c0c978849}], length 0, ParentProc [timeout.5252]
16:36:38.051114 ns1eth1 mptcp_connect.5253 In IP 10.0.1.2.54534 > 10.0.1.1.10004: Flags [.], seq 3642944205, ack 1598583135, win 1050, options [nop,nop,TS val 1352303447 ecr 4096278096,mptcp 20 capable v1 flags [H] {0xda33e37521a5eb9f,0xf28be19c0c978849}], length 0, ParentProc [timeout.5252]
16:36:38.063234 ns1eth1 mptcp_connect.5253 Out IP 10.0.1.1.10004 > 10.0.1.2.54534: Flags [P.], seq 1598583135:1598583235, ack 3642944205, win 1053, options [nop,nop,TS val 4096278102 ecr 1352303447,mptcp 22 dss ack 0 seq 0 subseq 0 len 0], length 100, ParentProc [timeout.5252]
16:36:38.063619 ns1eth1 mptcp_connect.5253 In IP 10.0.1.2.54534 > 10.0.1.1.10004: Flags [.], seq 3642944205, ack 1598583235, win 1050, options [nop,nop,TS val 1352303454 ecr 4096278102,mptcp 12 dss ack 16111094914477091189], length 0, ParentProc [timeout.5252]
16:36:38.064142 ns1eth1 1:2-events.181 In IP 10.0.3.2.55241 > 10.0.1.1.10004: Flags [S], seq 2848205064, win 65535, options [mss 1460,sackOK,TS val 3413286200 ecr 0,nop,wscale 8,mptcp 12 join id 1 token 0x3fc647ff nonce 0x50a65ac3], length 0, ParentProc [kthreadd.2]
16:36:38.064589 ns1eth3 Out ARP, Request who-has 10.0.3.2 tell 10.0.1.1, length 28
16:36:38.064748 ns1eth3 In ARP, Reply 10.0.3.2 is-at 46:4e:7e:c7:80:73, length 28
16:36:38.064862 ns1eth3 1:2-events.181 Out IP 10.0.1.1.10004 > 10.0.3.2.55241: Flags [S.], seq 1473949264, ack 2848205065, win 65535, options [mss 1460,sackOK,TS val 2572173167 ecr 3413286200,nop,wscale 8,mptcp 16 join id 0 hmac 0x767c9fb0b4aeaae7 nonce 0x0], length 0, ParentProc [kthreadd.2]
16:36:38.064989 ns1eth1 1:2-events.181 In IP 10.0.3.2.55241 > 10.0.1.1.10004: Flags [.], seq 2848205065, ack 1473949265, win 1050, options [nop,nop,TS val 3413286200 ecr 2572173167,mptcp 24 join hmac 0xa7d587d94069357dd3f4e7ae66040ac406545d1f], length 0, ParentProc [kthreadd.2]
16:36:38.065115 ns1eth3 1:2-events.181 Out IP 10.0.1.1.10004 > 10.0.3.2.55241: Flags [.], seq 1473949265, ack 2848205065, win 1053, options [nop,nop,TS val 2572173167 ecr 3413286200,mptcp 8 dss ack 1695918250], length 0, ParentProc [kthreadd.2]
16:36:38.065257 ns1eth1 1:2-events.181 In IP 10.0.2.2.49725 > 10.0.1.1.10004: Flags [S], seq 2207779350, win 65535, options [mss 1460,sackOK,TS val 4065758745 ecr 0,nop,wscale 8,mptcp 12 join id 2 token 0x3fc647ff nonce 0xf24388cb], length 0, ParentProc [kthreadd.2]
16:36:38.065378 ns1eth2 Out ARP, Request who-has 10.0.2.2 tell 10.0.1.1, length 28
16:36:38.065596 ns1eth2 In ARP, Reply 10.0.2.2 is-at 3e:40:c8:f2:60:b2, length 28
16:36:38.065710 ns1eth2 1:2-events.181 Out IP 10.0.1.1.10004 > 10.0.2.2.49725: Flags [S.], seq 2375890975, ack 2207779351, win 65535, options [mss 1460,sackOK,TS val 206937691 ecr 4065758745,nop,wscale 8,mptcp 16 join id 0 hmac 0x47817b6ed8236ef7 nonce 0x0], length 0, ParentProc [kthreadd.2]
16:36:38.065829 ns1eth1 1:2-events.181 In IP 10.0.2.2.49725 > 10.0.1.1.10004: Flags [.], seq 2207779351, ack 2375890976, win 1050, options [nop,nop,TS val 4065758745 ecr 206937691,mptcp 24 join hmac 0xe2238385480a065cb2c59800a93d349dd2901dde], length 0, ParentProc [kthreadd.2]
16:36:38.065945 ns1eth2 1:2-events.181 Out IP 10.0.1.1.10004 > 10.0.2.2.49725: Flags [.], seq 2375890976, ack 2207779351, win 1053, options [nop,nop,TS val 206937691 ecr 4065758745,mptcp 8 dss ack 1695918250], length 0, ParentProc [kthreadd.2]
panic: runtime error: index out of range [7] with length 0

goroutine 1 [running]:
encoding/binary.bigEndian.Uint64(...)
        /opt/hostedtoolcache/go/1.22.4/x64/src/encoding/binary/binary.go:183
github.com/x-way/pktdump.mpDSSPrint(0x1?, {0x1e, 0x1a, {0x0, 0x0, 0x0}, 0x2, 0x0, 0xc0004f7810, 0x0, ...}, ...)
        /home/runner/work/ptcpdump/ptcpdump/vendor/github.com/x-way/pktdump/mptcp.go:350 +0x12fc
github.com/x-way/pktdump.mptcpPrint(0x2752f40, {0x1e, 0x1a, {0x0, 0x0, 0x0}, 0x2, 0x0, 0xc0004f7810, 0x0, ...})
        /home/runner/work/ptcpdump/ptcpdump/vendor/github.com/x-way/pktdump/mptcp.go:639 +0x488
github.com/x-way/pktdump.formatMPTCP(...)
        /home/runner/work/ptcpdump/ptcpdump/vendor/github.com/x-way/pktdump/mptcp.go:661
github.com/x-way/pktdump.formatPacketTCP(0xc001cbe2c8, {0xc001be80b0, 0x8}, {0xc001be80b8, 0x8}, 0x41de85?, 0x1)
        /home/runner/work/ptcpdump/ptcpdump/vendor/github.com/x-way/pktdump/format.go:84 +0xc9f
github.com/x-way/pktdump.formatWithOptions({0x1680f40, 0xc001dc1ce0}, 0xc003648db8)
        /home/runner/work/ptcpdump/ptcpdump/vendor/github.com/x-way/pktdump/format.go:553 +0xf89
github.com/x-way/pktdump.FormatWithOptions({0x1680f40?, 0xc001dc1ce0?}, 0xc003648db8)
        /home/runner/work/ptcpdump/ptcpdump/vendor/github.com/x-way/pktdump/format.go:492 +0x25
github.com/mozillazg/ptcpdump/internal/writer.(*StdoutWriter).Write(0xc000c204b0, 0xc001bc6f30)
        /home/runner/work/ptcpdump/ptcpdump/internal/writer/stdout.go:89 +0xb9b
github.com/mozillazg/ptcpdump/internal/consumer.(*PacketEventConsumer).handlePacketEvent(0xc001d8b530, {{{0x13575112090, 0x0, {0x0, 0x0, 0x0}, 0x9, 0xc2, 0xc2, {0x1484, ...}, ...}}, ...})
        /home/runner/work/ptcpdump/ptcpdump/internal/consumer/net.go:60 +0x13f
github.com/mozillazg/ptcpdump/internal/consumer.(*PacketEventConsumer).Start(0xc001d8b530, {0x1674b68, 0xc0000a2400}, 0xc001cea960, 0x0)
        /home/runner/work/ptcpdump/ptcpdump/internal/consumer/net.go:43 +0x148
github.com/mozillazg/ptcpdump/cmd.capture({_, _}, _, {{0xc000054200, 0x1, 0x1}, {0x0, 0x0, 0x0}, {0x0, ...}, ...})
        /home/runner/work/ptcpdump/ptcpdump/cmd/capture.go:122 +0xb31
github.com/mozillazg/ptcpdump/cmd.run({{0xc000054200, 0x1, 0x1}, {0x0, 0x0, 0x0}, {0x0, 0x0}, 0x0, {0x0, ...}, ...})
        /home/runner/work/ptcpdump/ptcpdump/cmd/root.go:149 +0x270
github.com/mozillazg/ptcpdump/cmd.init.func1(0xc0000c4200?, {0xc0000a00e0?, 0x4?, 0x149807c?})
        /home/runner/work/ptcpdump/ptcpdump/cmd/root.go:42 +0xd8
github.com/spf13/cobra.(*Command).execute(0x1f35440, {0xc000040130, 0x2, 0x2})
        /home/runner/work/ptcpdump/ptcpdump/vendor/github.com/spf13/cobra/command.go:989 +0xab1
github.com/spf13/cobra.(*Command).ExecuteC(0x1f35440)
        /home/runner/work/ptcpdump/ptcpdump/vendor/github.com/spf13/cobra/command.go:1117 +0x3ff
github.com/spf13/cobra.(*Command).Execute(...)
        /home/runner/work/ptcpdump/ptcpdump/vendor/github.com/spf13/cobra/command.go:1041
github.com/mozillazg/ptcpdump/cmd.Execute(...)
        /home/runner/work/ptcpdump/ptcpdump/cmd/root.go:130
main.main()
        /home/runner/work/ptcpdump/ptcpdump/main.go:8 +0x1b

(note that I'm running the tests from a small VM where there is no Docker, etc.)

FYI, here is the trace presented by TCPDump:

16:36:30.420474 ip6gre0 Out IP6 :: > ff02::16: HBH ICMP6, multicast listener report v2, 2 group record(s), length 48                                                                                                                             
16:36:30.612431 ?     Out IP6 :: > ff02::1:ff04:315a: ICMP6, neighbor solicitation, who has fe80::ac67:36ff:fe04:315a, length 32                                                                     
16:36:30.652491 ?     Out IP6 :: > ff02::16: HBH ICMP6, multicast listener report v2, 2 group record(s), length 48                                                                                                                               
16:36:30.676551 ip6gre0 M   IP6 :: > ff02::1:fff2:60b2: ICMP6, neighbor solicitation, who has fe80::3c40:c8ff:fef2:60b2, length 32
16:36:30.740427 ip6tnl0 Out IP6 :: > ff02::1:ff3e:3d1: ICMP6, neighbor solicitation, who has fe80::8c01:f2ff:fe3e:3d1, length 32
16:36:30.804504 ?     Out IP6 :: > ff02::16: HBH ICMP6, multicast listener report v2, 2 group record(s), length 48
16:36:30.804525 ip6tnl0 M   IP6 :: > ff02::16: HBH ICMP6, multicast listener report v2, 2 group record(s), length 48
16:36:30.996439 ?     M   IP6 :: > ff02::16: HBH ICMP6, multicast listener report v2, 2 group record(s), length 48
16:36:30.996573 ?     Out IP6 :: > ff02::1:ff46:e3a1: ICMP6, neighbor solicitation, who has fe80::1463:b3ff:fe46:e3a1, length 32
16:36:30.996439 ?     M   IP6 :: > ff02::16: HBH ICMP6, multicast listener report v2, 2 group record(s), length 48
16:36:30.996573 ?     Out IP6 :: > ff02::1:ff46:e3a1: ICMP6, neighbor solicitation, who has fe80::1463:b3ff:fe46:e3a1, length 32
16:36:31.060502 ?     M   IP6 :: > ff02::1:ff14:2a5e: ICMP6, neighbor solicitation, who has fe80::642c:8cff:fe14:2a5e, length 32
16:36:31.060528 ip6gre0 Out IP6 :: > ff02::1:ffa1:5cad: ICMP6, neighbor solicitation, who has fe80::94ce:82ff:fea1:5cad, length 32
16:36:31.124532 ?     M   IP6 :: > ff02::16: HBH ICMP6, multicast listener report v2, 2 group record(s), length 48
16:36:31.188466 ip6tnl0 M   IP6 fe80::745a:49ff:fe3e:ac7b > ff02::16: HBH ICMP6, multicast listener report v2, 2 group record(s), length 48
16:36:31.188488 ip6tnl0 M   IP6 fe80::745a:49ff:fe3e:ac7b > ff02::2: ICMP6, router solicitation, length 16
16:36:31.252547 ?     M   IP6 :: > ff02::1:ffc7:8073: ICMP6, neighbor solicitation, who has fe80::444e:7eff:fec7:8073, length 32
16:36:31.636577 ?     Out IP6 fe80::ac67:36ff:fe04:315a > ff02::16: HBH ICMP6, multicast listener report v2, 2 group record(s), length 48
16:36:31.636602 ?     Out IP6 fe80::ac67:36ff:fe04:315a > ff02::2: ICMP6, router solicitation, length 16
16:36:31.700874 ip6gre0 M   IP6 fe80::3c40:c8ff:fef2:60b2 > ff02::16: HBH ICMP6, multicast listener report v2, 2 group record(s), length 48
16:36:31.700907 ip6gre0 M   IP6 fe80::3c40:c8ff:fef2:60b2 > ff02::2: ICMP6, router solicitation, length 16
16:36:31.764545 ip6tnl0 Out IP6 fe80::8c01:f2ff:fe3e:3d1 > ff02::16: HBH ICMP6, multicast listener report v2, 2 group record(s), length 48
16:36:31.764572 ip6tnl0 Out IP6 fe80::8c01:f2ff:fe3e:3d1 > ff02::2: ICMP6, router solicitation, length 16
16:36:31.956969 ip6tnl0 M   IP6 fe80::745a:49ff:fe3e:ac7b > ff02::16: HBH ICMP6, multicast listener report v2, 2 group record(s), length 48
16:36:31.964623 ip6gre0 M   IP6 fe80::3c40:c8ff:fef2:60b2 > ff02::16: HBH ICMP6, multicast listener report v2, 2 group record(s), length 48
16:36:32.020623 ?     Out IP6 fe80::1463:b3ff:fe46:e3a1 > ff02::16: HBH ICMP6, multicast listener report v2, 2 group record(s), length 48
16:36:32.020640 ?     Out IP6 fe80::1463:b3ff:fe46:e3a1 > ff02::2: ICMP6, router solicitation, length 16
16:36:32.052590 ?     Out IP6 fe80::ac67:36ff:fe04:315a > ff02::16: HBH ICMP6, multicast listener report v2, 2 group record(s), length 48
16:36:32.084998 ip6gre0 Out IP6 fe80::94ce:82ff:fea1:5cad > ff02::16: HBH ICMP6, multicast listener report v2, 2 group record(s), length 48
16:36:32.085023 ip6gre0 Out IP6 fe80::94ce:82ff:fea1:5cad > ff02::2: ICMP6, router solicitation, length 16
16:36:32.085045 ?     M   IP6 fe80::642c:8cff:fe14:2a5e > ff02::16: HBH ICMP6, multicast listener report v2, 2 group record(s), length 48
16:36:32.085053 ?     M   IP6 fe80::642c:8cff:fe14:2a5e > ff02::2: ICMP6, router solicitation, length 16
16:36:32.124609 ip6tnl0 Out IP6 fe80::8c01:f2ff:fe3e:3d1 > ff02::16: HBH ICMP6, multicast listener report v2, 2 group record(s), length 48
16:36:32.276731 ?     M   IP6 fe80::444e:7eff:fec7:8073 > ff02::16: HBH ICMP6, multicast listener report v2, 2 group record(s), length 48
16:36:32.276745 ?     M   IP6 fe80::444e:7eff:fec7:8073 > ff02::2: ICMP6, router solicitation, length 16
16:36:32.588907 ?     M   IP6 fe80::444e:7eff:fec7:8073 > ff02::16: HBH ICMP6, multicast listener report v2, 2 group record(s), length 48
16:36:32.660610 ip6gre0 Out IP6 fe80::94ce:82ff:fea1:5cad > ff02::16: HBH ICMP6, multicast listener report v2, 2 group record(s), length 48
16:36:32.852565 ?     M   IP6 fe80::642c:8cff:fe14:2a5e > ff02::16: HBH ICMP6, multicast listener report v2, 2 group record(s), length 48
16:36:32.853967 ?     Out IP6 fe80::1463:b3ff:fe46:e3a1 > ff02::16: HBH ICMP6, multicast listener report v2, 2 group record(s), length 48
16:36:35.320075 ip6tnl0 B   ARP, Request who-has 10.0.1.1 tell 10.0.1.2, length 28
16:36:35.320108 ip6tnl0 Out ARP, Reply 10.0.1.1 is-at 8e:01:f2:3e:03:d1, length 28
16:36:35.320110 ip6tnl0 In  IP 10.0.1.2.54534 > 10.0.1.1.10004: Flags [S], seq 3642944204, win 65535, options [mss 1460,sackOK,TS val 1352303447 ecr 0,nop,wscale 8,mptcp 4 capable v1], length 0
16:36:35.320136 ip6tnl0 Out IP 10.0.1.1.10004 > 10.0.1.2.54534: Flags [S.], seq 1598583134, ack 3642944205, win 65535, options [mss 1460,sackOK,TS val 4096278096 ecr 1352303447,nop,wscale 8,mptcp 12 capable v1 {0xf28be19c0c978849}], length 0
16:36:35.320148 ip6tnl0 In  IP 10.0.1.2.54534 > 10.0.1.1.10004: Flags [.], ack 1, win 1050, options [nop,nop,TS val 1352303447 ecr 4096278096,mptcp 20 capable v1 {0xda33e37521a5eb9f,0xf28be19c0c978849}], length 0
16:36:35.326433 ip6tnl0 Out IP 10.0.1.1.10004 > 10.0.1.2.54534: Flags [P.], seq 1:101, ack 1, win 1053, options [nop,nop,TS val 4096278102 ecr 1352303447,mptcp 22 dss ack 1695918250 seq 16111094914477091089 subseq 1 len 100,nop,nop], length 100
16:36:35.326476 ip6tnl0 In  IP 10.0.1.2.54534 > 10.0.1.1.10004: Flags [.], ack 101, win 1050, options [nop,nop,TS val 1352303454 ecr 4096278102,mptcp 12 dss ack 16111094914477091189], length 0
16:36:35.326553 ip6tnl0 In  IP 10.0.3.2.55241 > 10.0.1.1.10004: Flags [S], seq 2848205064, win 65535, options [mss 1460,sackOK,TS val 3413286200 ecr 0,nop,wscale 8,mptcp 12 join id 1 token 0x3fc647ff nonce 0x50a65ac3], length 0
16:36:35.326630 ?     Out ARP, Request who-has 10.0.3.2 tell 10.0.1.1, length 28
16:36:35.326642 ?     In  ARP, Reply 10.0.3.2 is-at 46:4e:7e:c7:80:73, length 28
16:36:35.326659 ?     Out IP 10.0.1.1.10004 > 10.0.3.2.55241: Flags [S.], seq 1473949264, ack 2848205065, win 65535, options [mss 1460,sackOK,TS val 2572173167 ecr 3413286200,nop,wscale 8,mptcp 16 join id 0 hmac 0x767c9fb0b4aeaae7 nonce 0x611fd8f4], length 0
16:36:35.326671 ip6tnl0 In  IP 10.0.3.2.55241 > 10.0.1.1.10004: Flags [.], ack 1, win 1050, options [nop,nop,TS val 3413286200 ecr 2572173167,mptcp 24 join hmac 0xa7d587d94069357dd3f4e7ae66040ac406545d1f], length 0
16:36:35.326702 ?     Out IP 10.0.1.1.10004 > 10.0.3.2.55241: Flags [.], ack 1, win 1053, options [nop,nop,TS val 2572173167 ecr 3413286200,mptcp 8 dss ack 1695918250], length 0
16:36:35.326721 ip6tnl0 In  IP 10.0.2.2.49725 > 10.0.1.1.10004: Flags [S], seq 2207779350, win 65535, options [mss 1460,sackOK,TS val 4065758745 ecr 0,nop,wscale 8,mptcp 12 join id 2 token 0x3fc647ff nonce 0xf24388cb], length 0
16:36:35.326749 ip6gre0 Out ARP, Request who-has 10.0.2.2 tell 10.0.1.1, length 28
16:36:35.326754 ip6gre0 In  ARP, Reply 10.0.2.2 is-at 3e:40:c8:f2:60:b2, length 28
16:36:35.326773 ip6gre0 Out IP 10.0.1.1.10004 > 10.0.2.2.49725: Flags [S.], seq 2375890975, ack 2207779351, win 65535, options [mss 1460,sackOK,TS val 206937691 ecr 4065758745,nop,wscale 8,mptcp 16 join id 0 hmac 0x47817b6ed8236ef7 nonce 0x82208c64], length 0
16:36:35.326780 ip6tnl0 In  IP 10.0.2.2.49725 > 10.0.1.1.10004: Flags [.], ack 1, win 1050, options [nop,nop,TS val 4065758745 ecr 206937691,mptcp 24 join hmac 0xe2238385480a065cb2c59800a93d349dd2901dde], length 0
16:36:35.326804 ip6gre0 Out IP 10.0.1.1.10004 > 10.0.2.2.49725: Flags [.], ack 1, win 1053, options [nop,nop,TS val 206937691 ecr 4065758745,mptcp 8 dss ack 1695918250], length 0
16:36:35.346330 ip6tnl0 In  IP 10.0.1.2.54534 > 10.0.1.1.10004: Flags [P.], seq 1:101, ack 101, win 1050, options [nop,nop,TS val 1352303473 ecr 4096278102,mptcp 26 dss ack 16111094914477091189 seq 13326749661043992746 subseq 1 len 100,nop,nop], length 100
16:36:35.346383 ip6tnl0 Out IP 10.0.1.1.10004 > 10.0.1.2.54534: Flags [.], ack 101, win 1053, options [nop,nop,TS val 4096278122 ecr 1352303473,mptcp 12 dss ack 13326749661043992846], length 0
16:36:35.527080 ip6tnl0 Out IP 10.0.1.1.10004 > 10.0.1.2.54534: Flags [P.], seq 101:1053, ack 101, win 1053, options [nop,nop,TS val 4096278303 ecr 1352303473,mptcp 26 dss ack 13326749661043992846 seq 16111094914477091189 subseq 101 len 952,nop,nop], length 952
16:36:35.527109 ip6tnl0 Out IP 10.0.1.1.10004 > 10.0.1.2.54534: Flags [.], ack 101, win 1053, options [nop,nop,TS val 4096278303 ecr 1352303473,mptcp 26 dss fin ack 13326749661043992846 seq 16111094914477092141 subseq 0 len 1,nop,nop], length 0
16:36:35.527114 ?     Out IP 10.0.1.1.10004 > 10.0.3.2.55241: Flags [.], ack 1, win 1053, options [nop,nop,TS val 2572173167 ecr 3413286200,mptcp 26 dss fin ack 13326749661043992846 seq 16111094914477092141 subseq 0 len 1,nop,nop], length 0
16:36:35.527124 ip6gre0 Out IP 10.0.1.1.10004 > 10.0.2.2.49725: Flags [.], ack 1, win 1053, options [nop,nop,TS val 206937691 ecr 4065758745,mptcp 26 dss fin ack 13326749661043992846 seq 16111094914477092141 subseq 0 len 1,nop,nop], length 0
16:36:35.527133 ip6tnl0 In  IP 10.0.1.2.54534 > 10.0.1.1.10004: Flags [.], ack 1053, win 1052, options [nop,nop,TS val 1352303654 ecr 4096278303,mptcp 12 dss ack 16111094914477092142], length 0
16:36:35.527138 ip6tnl0 In  IP 10.0.3.2.55241 > 10.0.1.1.10004: Flags [.], ack 1, win 1052, options [nop,nop,TS val 3413286400 ecr 2572173167,mptcp 12 dss ack 16111094914477092142], length 0
16:36:35.527140 ip6tnl0 In  IP 10.0.2.2.49725 > 10.0.1.1.10004: Flags [.], ack 1, win 1052, options [nop,nop,TS val 4065758945 ecr 206937691,mptcp 12 dss ack 16111094914477092142], length 0
16:36:35.547706 ip6tnl0 In  IP 10.0.1.2.54534 > 10.0.1.1.10004: Flags [P.], seq 101:1053, ack 1053, win 1055, options [nop,nop,TS val 1352303675 ecr 4096278303,mptcp 26 dss ack 16111094914477092142 seq 13326749661043992846 subseq 101 len 952,nop,nop], length 952
16:36:35.547735 ip6tnl0 Out IP 10.0.1.1.10004 > 10.0.1.2.54534: Flags [.], ack 1053, win 1052, options [nop,nop,TS val 4096278324 ecr 1352303675,mptcp 12 dss ack 13326749661043993798], length 0
16:36:35.547765 ip6tnl0 In  IP 10.0.1.2.54534 > 10.0.1.1.10004: Flags [.], ack 1053, win 1055, options [nop,nop,TS val 1352303675 ecr 4096278324,mptcp 26 dss fin ack 16111094914477092142 seq 13326749661043993798 subseq 0 len 1,nop,nop], length 0
16:36:35.547772 ip6tnl0 In  IP 10.0.3.2.55241 > 10.0.1.1.10004: Flags [.], ack 1, win 1055, options [nop,nop,TS val 3413286400 ecr 2572173167,mptcp 26 dss fin ack 16111094914477092142 seq 13326749661043993798 subseq 0 len 1,nop,nop], length 0
16:36:35.547775 ip6tnl0 In  IP 10.0.2.2.49725 > 10.0.1.1.10004: Flags [.], ack 1, win 1055, options [nop,nop,TS val 4065758945 ecr 206937691,mptcp 26 dss fin ack 16111094914477092142 seq 13326749661043993798 subseq 0 len 1,nop,nop], length 0
16:36:35.547940 ip6tnl0 Out IP 10.0.1.1.10004 > 10.0.1.2.54534: Flags [.], ack 1053, win 1055, options [nop,nop,TS val 4096278324 ecr 1352303675,mptcp 12 dss ack 13326749661043993799], length 0
16:36:35.547944 ?     Out IP 10.0.1.1.10004 > 10.0.3.2.55241: Flags [.], ack 1, win 1055, options [nop,nop,TS val 2572173388 ecr 3413286400,mptcp 12 dss ack 13326749661043993799], length 0
16:36:35.547948 ip6gre0 Out IP 10.0.1.1.10004 > 10.0.2.2.49725: Flags [.], ack 1, win 1055, options [nop,nop,TS val 206937912 ecr 4065758945,mptcp 12 dss ack 13326749661043993799], length 0
16:36:35.547952 ip6tnl0 In  IP 10.0.1.2.54534 > 10.0.1.1.10004: Flags [F.], seq 1053, ack 1053, win 1055, options [nop,nop,TS val 1352303675 ecr 4096278324,mptcp 12 dss ack 16111094914477092142], length 0
16:36:35.547955 ip6tnl0 In  IP 10.0.3.2.55241 > 10.0.1.1.10004: Flags [F.], seq 1, ack 1, win 1055, options [nop,nop,TS val 3413286421 ecr 2572173388,mptcp 12 dss ack 16111094914477092142], length 0
16:36:35.547956 ip6tnl0 In  IP 10.0.2.2.49725 > 10.0.1.1.10004: Flags [F.], seq 1, ack 1, win 1055, options [nop,nop,TS val 4065758966 ecr 206937912,mptcp 12 dss ack 16111094914477092142], length 0
16:36:35.547971 ip6tnl0 Out IP 10.0.1.1.10004 > 10.0.1.2.54534: Flags [F.], seq 1053, ack 1054, win 1055, options [nop,nop,TS val 4096278324 ecr 1352303675,mptcp 12 dss ack 13326749661043993799], length 0
16:36:35.548007 ip6tnl0 In  IP 10.0.1.2.54534 > 10.0.1.1.10004: Flags [.], ack 1054, win 1055, options [nop,nop,TS val 1352303675 ecr 4096278324,mptcp 12 dss ack 16111094914477092142], length 0
16:36:35.548015 ?     Out IP 10.0.1.1.10004 > 10.0.3.2.55241: Flags [F.], seq 1, ack 2, win 1055, options [nop,nop,TS val 2572173388 ecr 3413286421,mptcp 12 dss ack 13326749661043993799], length 0
16:36:35.548018 ip6tnl0 In  IP 10.0.3.2.55241 > 10.0.1.1.10004: Flags [.], ack 2, win 1055, options [nop,nop,TS val 3413286421 ecr 2572173388,mptcp 12 dss ack 16111094914477092142], length 0
16:36:35.548020 ip6gre0 Out IP 10.0.1.1.10004 > 10.0.2.2.49725: Flags [F.], seq 1, ack 2, win 1055, options [nop,nop,TS val 206937912 ecr 4065758966,mptcp 12 dss ack 13326749661043993799], length 0
16:36:35.548023 ip6tnl0 In  IP 10.0.2.2.49725 > 10.0.1.1.10004: Flags [.], ack 2, win 1055, options [nop,nop,TS val 4065758966 ecr 206937912,mptcp 12 dss ack 16111094914477092142], length 0
16:36:35.732614 ip6gre0 M   IP6 fe80::3c40:c8ff:fef2:60b2 > ff02::2: ICMP6, router solicitation, length 16
16:36:35.796545 ip6tnl0 Out IP6 fe80::8c01:f2ff:fe3e:3d1 > ff02::2: ICMP6, router solicitation, length 16
16:36:35.988779 ?     Out IP6 fe80::1463:b3ff:fe46:e3a1 > ff02::2: ICMP6, router solicitation, length 16
16:36:35.988792 ip6tnl0 M   IP6 fe80::745a:49ff:fe3e:ac7b > ff02::2: ICMP6, router solicitation, length 16
16:36:36.309338 ?     M   IP6 fe80::444e:7eff:fec7:8073 > ff02::2: ICMP6, router solicitation, length 16

@mozillazg
Copy link
Copy Markdown
Owner Author

mozillazg commented Sep 29, 2024

@matttbe Thanks for your review and test, I'll try running the same testing VM to fix the panic.

@matttbe
Copy link
Copy Markdown

matttbe commented Sep 29, 2024

I'll try running the same testing VM to fix the panic.

Thanks!

FYI, the mptcp-upstream-virtme docker image is there to build the Linux kernel, and start a VM with access to the kernel source code:

cd <kernel source code>
docker run -v "${PWD}:${PWD}:rw" -w "${PWD}" -v "${PWD}/.home:/root:rw" --rm \
  -it --privileged --pull always mptcp/mptcp-upstream-virtme-docker:latest \
  manual-btf

You can place ptcpdump binary in the kernel source dir to get access to it from the VM.

With the manual-btf mode (I guess BTF is needed for ptcpdump), you will have access to a shell where you can generate traffic. There are different tests that validate MPTCP:

  • KSelftests: some simple userspace tests that are in the kernel sources:

    • All MPTCP selftests are creating dedicated network namespaces to run these tests. It is easy to run them: 
      cd tools/testing/selftests/net/mptcp/
<script.sh>  ##  e.g. ./diag.sh
    • Because the tests are executed from a dedicated network namespace, we cannot simply launch ptcpdump -i any -- <script.sh>, apparently.
    • Many of these scripts support a -c option to launch tcpdump in the background inside the network namespaces, e.g. ./mptcp_join.sh -c
    • These scripts can be modified to run ptcpdump instead / in parallel, e.g. for mptcp_join.sh: 
      diff --git a/tools/testing/selftests/net/mptcp/mptcp_join.sh b/tools/testing/selftests/net/mptcp/mptcp_join.sh
index e8d0a01b4144..6a0475e6277a 100755
--- a/tools/testing/selftests/net/mptcp/mptcp_join.sh
+++ b/tools/testing/selftests/net/mptcp/mptcp_join.sh
@@ -918,8 +918,9 @@ do_transfer()
                echo "Capturing traffic for test $MPTCP_LIB_TEST_COUNTER into $capfile"
                ip netns exec ${listener_ns} tcpdump -i any -s 65535 -B 32768 $capuser -w $capfile > "$capout" 2>&1 &
                cappid=$!
+               ip netns exec ${listener_ns} ../../../../../ptcpdump -i any &
 
-               sleep 1
+               sleep 3
        fi
 
        NSTAT_HISTORY=/tmp/${listener_ns}.nstat ip netns exec ${listener_ns} \
      (I had to increase the sleep time, because ptcpdump takes more time to be ready in this VM, maybe because it checks for containerd, cgroups, etc. that are not available? Maybe I missed an option to avoid such timeout?)
    • To reproduce the issue I had, with the above patch, I ran the following command to execute only one test creating multiple paths with MPTCP: 
      ./mptcp_join.sh -c "multiple subflows"

  • Packetdrill: it will create a tun interface and inject packets to validate specific aspects in very small/quick tests:

    • typically, we launch the tests with run_all.py, but it will execute the tests inside a new network namespace, which doesn't seem to be supported by ptcpdump (nor tcpdump): 
      cd /opt/packetdrill/gtests/net/
run_all.py -lvP 4 mptcp
    • packetdrill can be executed directly, but it didn't seem to work for me, probably because of the tun interface that is created and used, and maybe not supported by ptcpdump?: 
      cd /opt/packetdrill/gtests/net/mptcp
ptcpdump -i any -- packetdrill -v mp_capable/v1_mp_capable_connct_no_cs.pkt
    • Note that run_all.py can be executed with the --capture option (e.g. -c ~), which will launch tcpdump inside the netns: you can modify /opt/packetdrill/gtests/net/packetdrill/in_netns.sh to launch ptcpdump instead. I didn't try because I guess I will have the same behaviour as with the previous command.

@mozillazg
Copy link
Copy Markdown
Owner Author

mozillazg commented Sep 30, 2024 •
edited
Loading

@matttbe I believe the panic is fixed. you can download the latest binary from https://github.com/mozillazg/ptcpdump/actions/runs/11102065291. I tested with using ptcpdump to read all pcap files were generated via run packetdrill tests and mptcp_join.sh -c

Because the tests are executed from a dedicated network namespace, we cannot simply launch ptcpdump -i any -- <script.sh>, apparently.

I'll support this case in the feature. #153

(I had to increase the sleep time, because ptcpdump takes more time to be ready in this VM, maybe because it checks for containerd, cgroups, etc. that are not available? Maybe I missed an option to avoid such timeout?)

I'll add an new option to allow skip these checks in the feature. #58

packetdrill can be executed directly, but it didn't seem to work for me, probably because of the tun interface that is created and used, and maybe not supported by ptcpdump

Yes, ptcpdump doesn't support this case yet. It'll be supported in the feature. #154

@matttbe
Copy link
Copy Markdown

matttbe commented Sep 30, 2024

@mozillazg thank you for the update!

I had a quick look: no more crashes, thanks!

I also took a bit more time to look at differences with TCPDump, and I noticed two issues (packets carrying data display a wrong DSS (dss ack 0 seq 0 subseq 0 len 0), and the SYN+ACK+MP_JOIN has no nonce) + some details (probably no need to fix most of them), please see below.

Here is an example when running mptcp_join.sh's unused signal address subtest:

ptcpdump:

$ ./mptcp_join.sh -c "unused signal address"
(...)
08:34:32.259852 ns1eth1 mptcp_connect.5891 In IP 10.0.1.2.51466 > 10.0.1.1.10011: Flags [S], seq 3968763826, win 65535, options [mss 1460,sackOK,TS val 1496177835 ecr 0,nop,wscale 8,mptcp 4 capable v1 flags [H]], length 0, ParentProc [timeout.5890]
08:34:32.261652 ns1eth1 mptcp_connect.5891 Out IP 10.0.1.1.10011 > 10.0.1.2.51466: Flags [S.], seq 372087587, ack 3968763827, win 65535, options [mss 1460,sackOK,TS val 4240152484 ecr 1496177835,nop,wscale 8,mptcp 12 capable v1 flags [H] {0xc53d54f2accb167a}], length 0, ParentProc [timeout.5890]
08:34:32.263921 ns1eth1 mptcp_connect.5891 In IP 10.0.1.2.51466 > 10.0.1.1.10011: Flags [.], seq 3968763827, ack 372087588, win 1050, options [nop,nop,TS val 1496177835 ecr 4240152484,mptcp 20 capable v1 flags [H] {0xabe9ccd1f53859e7,0xc53d54f2accb167a}], length 0, ParentProc [timeout.5890]
08:34:32.265774 ns1eth1 mptcp_connect.5891 Out IP 10.0.1.1.10011 > 10.0.1.2.51466: Flags [.], seq 372087588, ack 3968763827, win 1053, options [nop,nop,TS val 4240152484 ecr 1496177835,mptcp 16 add-addr v1 id 1 10.0.2.1:0 hmac 0xd43bfb3a9c0d5ac1], length 0, ParentProc [timeout.5890]
08:34:32.267692 ns1eth1 mptcp_connect.5891 In IP 10.0.1.2.51466 > 10.0.1.1.10011: Flags [.], seq 3968763827, ack 372087588, win 1050, options [nop,nop,TS val 1496177835 ecr 4240152484,mptcp 8 add-addr v1 id 1 10.0.2.1:0 hmac 0x0], length 0, ParentProc [timeout.5890]
08:34:32.271608 ns1eth1 mptcp_connect.5891 Out IP 10.0.1.1.10011 > 10.0.1.2.51466: Flags [P.], seq 372087588:372087688, ack 3968763827, win 1053, options [nop,nop,TS val 4240152484 ecr 1496177835,mptcp 22 dss ack 0 seq 0 subseq 0 len 0 csum 0x0], length 100, ParentProc [timeout.5890]
08:34:32.273673 ns1eth1 mptcp_connect.5891 In IP 10.0.1.2.51466 > 10.0.1.1.10011: Flags [.], seq 3968763827, ack 372087688, win 1050, options [nop,nop,TS val 1496177835 ecr 4240152484,mptcp 12 dss ack 8205239806650827240], length 0, ParentProc [timeout.5890]
08:34:32.280271 ns1eth1 mptcp_connect.5891 In IP 10.0.1.2.51466 > 10.0.1.1.10011: Flags [P.], seq 3968763827:3968763927, ack 372087688, win 1050, options [nop,nop,TS val 1496177840 ecr 4240152484,mptcp 26 dss ack 0 seq 0 subseq 0 len 0 csum 0x0], length 100, ParentProc [timeout.5890]
08:34:32.282242 ns1eth1 mptcp_connect.5891 Out IP 10.0.1.1.10011 > 10.0.1.2.51466: Flags [.], seq 372087688, ack 3968763927, win 1053, options [nop,nop,TS val 4240152489 ecr 1496177840,mptcp 12 dss ack 4248319828782699746], length 0, ParentProc [timeout.5890]
08:34:32.661958 ns1eth1 mptcp_connect.5891 Out IP 10.0.1.1.10011 > 10.0.1.2.51466: Flags [P.], seq 372087688:372088640, ack 3968763927, win 1053, options [nop,nop,TS val 4240152686 ecr 1496177840,mptcp 26 dss ack 0 seq 0 subseq 0 len 0 csum 0x0], length 952, ParentProc [timeout.5890]
08:34:32.665301 ns1eth1 mptcp_connect.5891 Out IP 10.0.1.1.10011 > 10.0.1.2.51466: Flags [.], seq 372088640, ack 3968763927, win 1053, options [nop,nop,TS val 4240152686 ecr 1496177840,mptcp 26 dss ack 4248319828782699746 seq 8205239806650828192 subseq 0 len 1,nop,nop], length 0, ParentProc [timeout.5890]
08:34:32.667892 ns1eth1 mptcp_connect.5891 In IP 10.0.1.2.51466 > 10.0.1.1.10011: Flags [.], seq 3968763927, ack 372088640, win 1052, options [nop,nop,TS val 1496178037 ecr 4240152686,mptcp 12 dss ack 8205239806650828193], length 0, ParentProc [timeout.5890]
08:34:32.679579 ns1eth1 mptcp_connect.5891 In IP 10.0.1.2.51466 > 10.0.1.1.10011: Flags [P.], seq 3968763927:3968764879, ack 372088640, win 1061, options [nop,nop,TS val 1496178045 ecr 4240152686,mptcp 26 dss ack 0 seq 0 subseq 0 len 0 csum 0x0], length 952, ParentProc [timeout.5890]
08:34:32.682173 ns1eth1 mptcp_connect.5891 Out IP 10.0.1.1.10011 > 10.0.1.2.51466: Flags [.], seq 372088640, ack 3968764879, win 1052, options [nop,nop,TS val 4240152694 ecr 1496178045,mptcp 12 dss ack 4248319828782700698], length 0, ParentProc [timeout.5890]
08:34:32.684014 ns1eth1 mptcp_connect.5891 In IP 10.0.1.2.51466 > 10.0.1.1.10011: Flags [.], seq 3968764879, ack 372088640, win 1061, options [nop,nop,TS val 1496178045 ecr 4240152694,mptcp 26 dss ack 8205239806650828193 seq 4248319828782700698 subseq 0 len 1,nop,nop], length 0, ParentProc [timeout.5890]
08:34:32.685933 ns1eth1 mptcp_connect.5891 Out IP 10.0.1.1.10011 > 10.0.1.2.51466: Flags [.], seq 372088640, ack 3968764879, win 1052, options [nop,nop,TS val 4240152695 ecr 1496178045,mptcp 12 dss ack 4248319828782700699], length 0, ParentProc [timeout.5890]
08:34:32.689919 ns1eth1 mptcp_connect.5891 In IP 10.0.1.2.51466 > 10.0.1.1.10011: Flags [F.], seq 3968764879, ack 372088640, win 1061, options [nop,nop,TS val 1496178046 ecr 4240152695,mptcp 12 dss ack 8205239806650828193], length 0, ParentProc [timeout.5890]
08:34:32.693772 ns1eth1 mptcp_connect.5891 Out IP 10.0.1.1.10011 > 10.0.1.2.51466: Flags [F.], seq 372088640, ack 3968764880, win 1061, options [nop,nop,TS val 4240152697 ecr 1496178046,mptcp 12 dss ack 4248319828782700699], length 0, ParentProc [timeout.5890]
08:34:32.695618 ns1eth1 mptcp_connect.5891 In IP 10.0.1.2.51466 > 10.0.1.1.10011: Flags [.], seq 3968764880, ack 372088641, win 1061, options [nop,nop,TS val 1496178048 ecr 4240152697,mptcp 12 dss ack 8205239806650828193], length 0, ParentProc [timeout.5890]

tcpdump (with -S to print absolute, rather than relative, TCP sequence numbers → even if it is harder to read, it is now similar to what ptcpdump is doing, which is easier to compare):

08:34:29.707925 ip6tnl0 In  IP 10.0.1.2.51466 > 10.0.1.1.10011: Flags [S], seq 3968763826, win 65535, options [mss 1460,sackOK,TS val 1496177835 ecr 0,nop,wscale 8,mptcp 4 capable v1], length 0
08:34:29.707949 ip6tnl0 Out IP 10.0.1.1.10011 > 10.0.1.2.51466: Flags [S.], seq 372087587, ack 3968763827, win 65535, options [mss 1460,sackOK,TS val 4240152484 ecr 1496177835,nop,wscale 8,mptcp 12 capable v1 {0xc53d54f2accb167a}], length 0
08:34:29.707960 ip6tnl0 In  IP 10.0.1.2.51466 > 10.0.1.1.10011: Flags [.], ack 372087588, win 1050, options [nop,nop,TS val 1496177835 ecr 4240152484,mptcp 20 capable v1 {0xabe9ccd1f53859e7,0xc53d54f2accb167a}], length 0
08:34:29.708004 ip6tnl0 Out IP 10.0.1.1.10011 > 10.0.1.2.51466: Flags [.], ack 3968763827, win 1053, options [nop,nop,TS val 4240152484 ecr 1496177835,mptcp 16 add-addr v1 id 1 10.0.2.1 hmac 0xd43bfb3a9c0d5ac1], length 0
08:34:29.708010 ip6tnl0 In  IP 10.0.1.2.51466 > 10.0.1.1.10011: Flags [.], ack 372087588, win 1050, options [nop,nop,TS val 1496177835 ecr 4240152484,mptcp 8 add-addr v1-echo id 1 10.0.2.1], length 0
08:34:29.708150 ip6tnl0 Out IP 10.0.1.1.10011 > 10.0.1.2.51466: Flags [P.], seq 372087588:372087688, ack 3968763827, win 1053, options [nop,nop,TS val 4240152484 ecr 1496177835,mptcp 22 dss ack 785863806 seq 8205239806650827140 subseq 1 len 100,nop,nop], length 100
08:34:29.708162 ip6tnl0 In  IP 10.0.1.2.51466 > 10.0.1.1.10011: Flags [.], ack 372087688, win 1050, options [nop,nop,TS val 1496177835 ecr 4240152484,mptcp 12 dss ack 8205239806650827240], length 0
08:34:29.713132 ip6tnl0 In  IP 10.0.1.2.51466 > 10.0.1.1.10011: Flags [P.], seq 3968763827:3968763927, ack 372087688, win 1050, options [nop,nop,TS val 1496177840 ecr 4240152484,mptcp 26 dss ack 8205239806650827240 seq 4248319828782699646 subseq 1 len 100,nop,nop], length 100
08:34:29.713179 ip6tnl0 Out IP 10.0.1.1.10011 > 10.0.1.2.51466: Flags [.], ack 3968763927, win 1053, options [nop,nop,TS val 4240152489 ecr 1496177840,mptcp 12 dss ack 4248319828782699746], length 0
08:34:29.909566 ip6tnl0 Out IP 10.0.1.1.10011 > 10.0.1.2.51466: Flags [P.], seq 372087688:372088640, ack 3968763927, win 1053, options [nop,nop,TS val 4240152686 ecr 1496177840,mptcp 26 dss ack 4248319828782699746 seq 8205239806650827240 subseq 101 len 952,nop,nop], length 952
08:34:29.909632 ip6tnl0 Out IP 10.0.1.1.10011 > 10.0.1.2.51466: Flags [.], ack 3968763927, win 1053, options [nop,nop,TS val 4240152686 ecr 1496177840,mptcp 26 dss fin ack 4248319828782699746 seq 8205239806650828192 subseq 0 len 1,nop,nop], length 0
08:34:29.909650 ip6tnl0 In  IP 10.0.1.2.51466 > 10.0.1.1.10011: Flags [.], ack 372088640, win 1052, options [nop,nop,TS val 1496178037 ecr 4240152686,mptcp 12 dss ack 8205239806650828193], length 0
08:34:29.918242 ip6tnl0 In  IP 10.0.1.2.51466 > 10.0.1.1.10011: Flags [P.], seq 3968763927:3968764879, ack 372088640, win 1061, options [nop,nop,TS val 1496178045 ecr 4240152686,mptcp 26 dss ack 8205239806650828193 seq 4248319828782699746 subseq 101 len 952,nop,nop], length 952
08:34:29.918333 ip6tnl0 Out IP 10.0.1.1.10011 > 10.0.1.2.51466: Flags [.], ack 3968764879, win 1052, options [nop,nop,TS val 4240152694 ecr 1496178045,mptcp 12 dss ack 4248319828782700698], length 0
08:34:29.918378 ip6tnl0 In  IP 10.0.1.2.51466 > 10.0.1.1.10011: Flags [.], ack 372088640, win 1061, options [nop,nop,TS val 1496178045 ecr 4240152694,mptcp 26 dss fin ack 8205239806650828193 seq 4248319828782700698 subseq 0 len 1,nop,nop], length 0
08:34:29.918432 ip6tnl0 Out IP 10.0.1.1.10011 > 10.0.1.2.51466: Flags [.], ack 3968764879, win 1052, options [nop,nop,TS val 4240152695 ecr 1496178045,mptcp 12 dss ack 4248319828782700699], length 0
08:34:29.918574 ip6tnl0 In  IP 10.0.1.2.51466 > 10.0.1.1.10011: Flags [F.], seq 3968764879, ack 372088640, win 1061, options [nop,nop,TS val 1496178046 ecr 4240152695,mptcp 12 dss ack 8205239806650828193], length 0
08:34:29.920772 ip6tnl0 Out IP 10.0.1.1.10011 > 10.0.1.2.51466: Flags [F.], seq 372088640, ack 3968764880, win 1061, options [nop,nop,TS val 4240152697 ecr 1496178046,mptcp 12 dss ack 4248319828782700699], length 0
08:34:29.920806 ip6tnl0 In  IP 10.0.1.2.51466 > 10.0.1.1.10011: Flags [.], ack 372088641, win 1061, options [nop,nop,TS val 1496178048 ecr 4240152697,mptcp 12 dss ack 8205239806650828193], length 0

If we do a "word diff" after having removed the headers (e.g. here), we can easily see the issue I mentioned, and a few more details:

  • The PUSH packets carrying data indicate that the DSS is 0, that seems wrong. ptcpdump displays: dss ack 0 seq 0 subseq 0 len 0, while tcpdump displays 32-bit counters (dss ack 785863806 seq 8205239806650827140 subseq 1 len 100) and 64-bit ones correctly (dss ack 8205239806650827240 seq 4248319828782699646 subseq 1 len 100)
  • In these PUSH packets, ptcpdump also indicates 0x0 for the csum, that's maybe not needed if it is not used (csum 0x0)
  • In the ACK packets, ptcpdump correctly display the dss ack.
  • In the DATA FIN packets (F flag), the DSN are correctly displayed by ptcpdump, but it is not mentioned the F flag is set (dss fin in tcpdump). I think it is important to mention that it is a DATA FIN somehow: either with fin or flag F.
  • In the 3 first packets, we can see flags [H] in ptcpdump, not in tcpdump. This flag H should always be set, I understand why it is not printed on tcpdump side, but it is also correct to mention it, because it should be set. So I'm fine with or without it.
  • The 4th packet is an ADD_ADDR without port. ptcpdump display :0 (10.0.2.1:0), not tcpdump. Maybe it is not needed to display the port? A detail, I'm sure it is fine with or without
  • The 5th packet is an ADD_ADDR echo (with the E flag set, and not HMAC), but ptcpdump doesn't mention that: it displays hmac 0x0, which is probably not needed, but a detail.

Also, when running other subtests from mptcp_join.sh, I noticed:

  • signal address: the new subflows are no longer linked to mptcp_connect, and there is no more info about the parent proc (ParentProc [...]): I guess that's normal, because these subflows are created by the kernel, e.g. 
    09:19:11.577848 ns1eth1 mptcp_connect.6260 In IP 10.0.1.2.48508 > 10.0.1.1.10012: Flags [.], seq 879946743, ack 3520608292, win 1050, options [nop,nop,TS val 1498859695 ecr 4242834344,mptcp 8 add-addr v1 id 1 10.0.2.1:0 hmac 0x0], length 0, ParentProc [timeout.6259]
09:19:11.579715 ns1eth2 In IP 10.0.2.2.40279 > 10.0.2.1.10012: Flags [S], seq 3316375630, win 65535, options [mss 1460,sackOK,TS val 436832514 ecr 0,nop,wscale 8,mptcp 12 join id 1 token 0x622f41c7 nonce 0x7a0a68cb], length 0
  • in the same tests, ptcpdump displays the SYN+ACK+MP_JOIN with a nonce of 0x0 while tcpdump got nonce 0x2ad0d586 (looks like an issue): 
    [S.], seq 581755826, ack 3316375631, win 65535, options [mss 1460,sackOK,TS val 3064065665 ecr 436832514,nop,wscale 8,mptcp 16 join id 1 hmac 0x5c152784f455fb06 nonce 0x0], length 0
  • single subflow, backup: an MP_PRIO is sent, initiated by the userspace via Netlink (ip mptcp endpoint change command), then triggered by the MPTCP worker. Because of that, the parent proc and command are different, but that seems normal: 
    09:34:14.827286 ns1eth1 1:2-events.181 In IP 10.0.3.2.57227 > 10.0.1.1.10064: Flags [.], seq 3033298228, ack 2463986344, win 1050, options [nop,nop,TS val 3560743630 ecr 2719630597,mptcp 12 dss ack 5905293352544175922,mptcp 3 prio non-backup,nop], length 0, ParentProc [kthreadd.2]
  • checksum test 1 1: ptcpdump will display flags [A] csum, while tcpdump only displays csum: that's because the A flag means checksum, a detail
  • fastclose test: the packet order is different when comparing ptcpdump and tcpdump, but I guess that's normal, that's due to how the RESET is processed by the network stack.
  • single subflow allow join id0 ns1: ptcpdump correctly shows the C flag, while tcpdump → that's good :-)
  • (TFO is not supported, used in mptcp_connect.sh), but I see GOPacket doesn't support it, and not directly linked to MPTCP anyway, it is just that MPTCP supports it ; still ptcpdump displays that the SYN has a length > 0, which is the main info to know)

@matttbe
Copy link
Copy Markdown

matttbe commented Sep 30, 2024

Thank you for having fixed the crash and opened the new tickets :)

@mozillazg
Copy link
Copy Markdown
Owner Author

mozillazg commented Sep 30, 2024

@matttbe Thanks for your review and testing! I'll fix the display related issues.

@matttbe
Copy link
Copy Markdown

matttbe commented Sep 30, 2024

Thank you! :)

@mozillazg
Copy link
Copy Markdown
Owner Author

mozillazg commented Sep 30, 2024

packets carrying data display a wrong DSS (dss ack 0 seq 0 subseq 0 len 0)

this issue is fixed and it was caused by gopacket: mozillazg/gopacket@2e5a32c

the SYN+ACK+MP_JOIN has no nonce

it is fixed.

  • The PUSH packets carrying data indicate that the DSS is 0, that seems wrong. ptcpdump displays: dss ack 0 seq 0 subseq 0 len 0, while tcpdump displays 32-bit counters (dss ack 785863806 seq 8205239806650827140 subseq 1 len 100) and 64-bit ones correctly (dss ack 8205239806650827240 seq 4248319828782699646 subseq 1 len 100)
  • In these PUSH packets, ptcpdump also indicates 0x0 for the csum, that's maybe not needed if it is not used (csum 0x0)
  • In the DATA FIN packets (F flag), the DSN are correctly displayed by ptcpdump, but it is not mentioned the F flag is set (dss fin in tcpdump). I think it is important to mention that it is a DATA FIN somehow: either with fin or flag F.
  • The 4th packet is an ADD_ADDR without port. ptcpdump display :0 (10.0.2.1:0), not tcpdump. Maybe it is not needed to display the port? A detail, I'm sure it is fine with or without
  • The 5th packet is an ADD_ADDR echo (with the E flag set, and not HMAC), but ptcpdump doesn't mention that: it displays hmac 0x0, which is probably not needed, but a detail.
  • in the same tests, ptcpdump displays the SYN+ACK+MP_JOIN with a nonce of 0x0 while tcpdump got nonce 0x2ad0d586 (looks like an issue):
  • checksum test 1 1: ptcpdump will display flags [A] csum, while tcpdump only displays csum: that's because the A flag means checksum, a detail

These issues are fixed.

  • In the 3 first packets, we can see flags [H] in ptcpdump, not in tcpdump. This flag H should always be set, I understand why it is not printed on tcpdump side, but it is also correct to mention it, because it should be set. So I'm fine with or without it.

OK, I may improve it in the feature.

@matttbe
Copy link
Copy Markdown

matttbe commented Sep 30, 2024

packets carrying data display a wrong DSS (dss ack 0 seq 0 subseq 0 len 0)

this issue is fixed and it was caused by gopacket: mozillazg/gopacket@2e5a32c

Good catch!

I just updated the upstream PR: gopacket/gopacket#66

I added you as a co-author, and I also fixed another similar issue with MP_TCPRST, see here. (The U flag should be set to 0 anyway)

the SYN+ACK+MP_JOIN has no nonce

it is fixed.

Thanks! I confirm that!

  • The PUSH packets carrying data indicate that the DSS is 0, that seems wrong. ptcpdump displays: dss ack 0 seq 0 subseq 0 len 0, while tcpdump displays 32-bit counters (dss ack 785863806 seq 8205239806650827140 subseq 1 len 100) and 64-bit ones correctly (dss ack 8205239806650827240 seq 4248319828782699646 subseq 1 len 100)
  • In these PUSH packets, ptcpdump also indicates 0x0 for the csum, that's maybe not needed if it is not used (csum 0x0)
  • In the DATA FIN packets (F flag), the DSN are correctly displayed by ptcpdump, but it is not mentioned the F flag is set (dss fin in tcpdump). I think it is important to mention that it is a DATA FIN somehow: either with fin or flag F.
  • The 4th packet is an ADD_ADDR without port. ptcpdump display :0 (10.0.2.1:0), not tcpdump. Maybe it is not needed to display the port? A detail, I'm sure it is fine with or without
  • The 5th packet is an ADD_ADDR echo (with the E flag set, and not HMAC), but ptcpdump doesn't mention that: it displays hmac 0x0, which is probably not needed, but a detail.
  • in the same tests, ptcpdump displays the SYN+ACK+MP_JOIN with a nonce of 0x0 while tcpdump got nonce 0x2ad0d586 (looks like an issue):
  • checksum test 1 1: ptcpdump will display flags [A] csum, while tcpdump only displays csum: that's because the A flag means checksum, a detail

These issues are fixed.

Great, thanks! It looks excellent now!

  • In the 3 first packets, we can see flags [H] in ptcpdump, not in tcpdump. This flag H should always be set, I understand why it is not printed on tcpdump side, but it is also correct to mention it, because it should be set. So I'm fine with or without it.

OK, I may improve it in the feature.

Sure, I'm fine with or without. Probably easier to keep it.

@mozillazg
Copy link
Copy Markdown
Owner Author

mozillazg commented Oct 2, 2024

checksum test 1 1: ptcpdump will display flags [A] csum, while tcpdump only displays csum: that's because the A flag means checksum, a detail

FYI. I updated it again to match the latest tcpdump:

flags [AH] csum

https://github.com/the-tcpdump-group/tcpdump/blob/master/tests/mptcp-fclose.out#L3-L5

Another mptcp related bug from gopacket was found: mozillazg/gopacket@112c3fb

@matttbe matttbe mentioned this pull request Oct 2, 2024
Merged
@matttbe
Copy link
Copy Markdown

matttbe commented Oct 2, 2024

Thank you for the recent modifications!

Another mptcp related bug from gopacket was found: mozillazg/gopacket@112c3fb

Good catch! I just updated the gopacket PR with the same fix.

@mozillazg mozillazg merged commit c6fbd56 into master Oct 3, 2024
@mozillazg mozillazg deleted the mptcp branch October 3, 2024 02:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Add MPTCP support

2 participants

@mozillazg @matttbe