Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Issues with backup flow #315

Open
Monika-Research opened this issue Nov 15, 2022 · 42 comments
Open

Issues with backup flow #315

Monika-Research opened this issue Nov 15, 2022 · 42 comments
Labels
Projects

Comments

@Monika-Research
Copy link

Hi Team,
Below is my setup details:
2 rpis - Ubuntu 22.04 server, 5.15.0-1016 kernel
Clients IP addresses - 192.168.11.2/24, 192.168.12.2/24
Server IP addresses - 192.168.11.3/24, 192.168.12.3/24
Stopped mptcpd and using ip mptcp to set path management rules

My aim is to create a primary flow between 192.168.11.2 & 192.168.11.3 and a backup subflow between 192.168.12.2 & 192.168.12.3 network. That is, in the middle of a data transmission, if I put down the primary interface, I am expecting the backup flow to come as a savior. However I failed to create a backup flow from 192.168.12.2 to 192.168.12.3.

Following is the configuration used
Client config:
ip mptcp limits set add_addr_accepted 1 subflows 1
ip mptcp endpoint add 192.168.12.2 dev bat1 subflow backup

Server config:
ip mptcp limits set add_addr_accepted 1 subflows 1
ip mptcp endpoint add 192.168.12.3 dev bat1 signal backup

The above config results in following
[SF_ESTABLISHED] token=d262648 remid=0 locid=1 saddr4=192.168.12.2 daddr=192.168.11.3 sport=41215 dport=5201 backup=1 ifindex=9
[SF_ESTABLISHED] token=753246 remid=0 locid=1 saddr4=192.168.12.2 daddr=192.168.12.3 sport=41860 dport=5201 backup=0

  1. I don't know why it is trying to create a flow between 192.168.12.2 & 192.168.11.3. Its not fullmesh I assume.
  2. Why 192.168.12.2 - 192.168.12.3 is getting created as non-backup subflow? I see data getting transmitted over this flow.

I checked the configurations suggested in the following discussions
#237
#191

However I couldn't get 192.168.12.2-192.168.12.3 as backup flow.
In case I remove signal option from server as suggested in one of the links above, I get a backup flow from 192.168.12.2-192.168.11.3.

Requesting your suggestions please in creating and putting the correct backup flow to use when the primary flow fails during a transmission.

Thanks,
Monika

@matttbe matttbe added the bug label Nov 17, 2022
@matttbe matttbe added this to Needs triage in MPTCP Bugs via automation Nov 17, 2022
@matttbe
Copy link
Member

matttbe commented Nov 21, 2022

Hi Monika,

Thank you for the bug report.

Stopped mptcpd and using ip mptcp to set path management rules

Did you disable the service not to have it started at boot time or did you stop it later? I mean: did you check there was no rules previously added by this daemon?

I don't know why it is trying to create a flow between 192.168.12.2 & 192.168.11.3. Its not fullmesh I assume.

What are the (IP) routes and rules you have in place? (ip route show table all && ip rule)?
On the client side, how do you ask to connect to the other side? a connect() to a specific IP? A bind() to force a source IP?
On the server side, do you restrict to one IP when you do the bind()?

Why 192.168.12.2 - 192.168.12.3 is getting created as non-backup subflow? I see data getting transmitted over this flow.

I wonder if this has not been "fixed" in v5.17.
Do you have packet traces to see the beginning of a connection?
Is the primary connection "faulty"? (losses, etc.)

Requesting your suggestions please in creating and putting the correct backup flow to use when the primary flow fails during a transmission.

The ideal for us would be to install a recent kernel, ideally compiled from our export branch or from the last RC one from kernel.org. If you stay on kernel v5.15, that will be difficult for us to help. A lot has changed and maybe a more recent version supports a missing feature needed here.

For Ubuntu, there are packages regularly built: https://kernel.ubuntu.com/~kernel-ppa/mainline/
But I don't know if you can use them on a RPI.
But building a kernel is not complex when you have a good guide, e.g. https://www.raspberrypi.com/documentation/computers/linux_kernel.html

@matttbe
Copy link
Member

matttbe commented Dec 8, 2022

Hi @Monika-Research

By chance, do you have any answers to the questions from my previous message?

@Monika-Research
Copy link
Author

Hi Matt,
Firstly, sorry for a late response.

I tried to use a recent kernel 6.1.2-v8+. I compiled the kernel source from https://github.com/raspberrypi/linux
However, I am facing issues with mptcpize now.

When I try to mptcpize iperf, I get the following error
mptcpize run -d iperf3 -s
mptcpwrap: changing socket protocol from 0x0 to 0x106 (IPPROTO_MPTCP) for family 0xa type 0x1 fd -1
iperf3: error -unable to start listener for connections: Protocol not supported

I tried to enable mptcp support with iperf3 - mptcpize enable iperf3
And restart iperf3 service - systemctl restart iperf3.service

But still the issue exists. So stuck here and not able to check the backup flow issue faced earlier with 5.15 kernel version.

Thanks
Monika

@matttbe
Copy link
Member

matttbe commented Jan 9, 2023

Hi Monika,

If you got EPROTONOSUPPORT 93 Protocol not supported error, it likely means MPTCP has not been compiled.
When you do the compilation, make sure to select MPTCP option, for example via make menuconfig or similar.

@matttbe
Copy link
Member

matttbe commented Jan 9, 2023

If you used the default config for a RPI and MPTCP is not enabled by default, please ask them to enable it by default as it is done in most distribution. And it makes sense to have that on a RPI.

@Monika-Research
Copy link
Author

Monika-Research commented Jan 9, 2023

Hi Matt,
Thanks for your immediate response.
I enabled the configs - CONFIG_MPTCP=y and CONFIG_MPTCP_IPV6=y.
sysctl net.mptcp.enabled gives me 1

@matttbe
Copy link
Member

matttbe commented Jan 9, 2023

mmh, strange...

Could you run python3 and write this?

import socket
s = socket.socket(socket.AF_INET6, socket.SOCK_STREAM, socket.IPPROTO_MPTCP)
s.bind(("::1", 1234))
s.listen(1)

Do you have the same EPROTONOSUPPORT error?

I'm not sure how else you could get Protocol not supported error. I guess you didn't use SELinux rules or similar.

@Monika-Research
Copy link
Author

Executing "import socket
s = socket.socket(socket.AF_INET6, socket.SOCK_STREAM, socket.IPPROTO_MPTCP)"

gives the same error - OSError: [Errno 93] Protocol not supported.

Can you brief me about SELinux rules?

@matttbe
Copy link
Member

matttbe commented Jan 9, 2023

If I'm not mistaken, there are 3 possible errors when creating a socket:

  • ENOPROTOOPT 92 Protocol not available: sysctl net.mptcp.enabled is set to 0
  • EPROTONOSUPPORT 93 Protocol not supported: MPTCP is not compiled on >=v5.6
  • EINVAL 22 Invalid argument: MPTCP is not available on <5.6
  • (ENOMEM, etc. but that's not MPTCP specific)

I don't know all paths producing a EPROTONOSUPPORT error. The only one that comes to my mind is if you didn't enable CONFIG_IPV6 but I guess it is. You can also try to create and AF_INET socket instead but I guess you will have the same issue.

If you can see the enable sysctl, you probably have everything you need: a kernel supporting MPTCP and the config enabled.

SELinux can be used to not allow you to create MPTCP sockets but it needs manual configuration, I guess it is not done automatically on a RPI and you didn't do that. You could cause some similar errors with BPF, another libc (or changing the behaviour of socket() with LD_PRELOAD), etc. but I guess you didn't do that.

One last thing to check maybe is if you can see mptcp when running: sysctl net.ipv4.tcp_available_ulp.

If not, I'm not sure where else to look at...

@Monika-Research
Copy link
Author

Monika-Research commented Jan 9, 2023

Thank you so much for a comprehensive response.
I am afraid if I enabled CONFIG_IPV6, because using AF_INET works fine without any error. Do I need to recompile the kernel if the config enabled? Or is there any other shortcut?

@Monika-Research
Copy link
Author

Monika-Research commented Jan 9, 2023

I would like to add that AF_INET6 with TCP works fine and it fails only with MPTCP. I missed to add CONFIG_MPTCP_IPV6 I guess.

@matttbe
Copy link
Member

matttbe commented Jan 9, 2023

I missed to add CONFIG_MPTCP_IPV6 I guess.

I thought you did but if not, yes it would be better to enable this and recompile the kernel. Maybe apps create a v6 sockets (in fact, a mix v4/v6) even if v4 is used after.

@Monika-Research
Copy link
Author

Hmmm.. I get it. Compilation in progress. I will keep you posted. Much appreciate your support.

@Monika-Research
Copy link
Author

Hi Matt,
It worked after enabling IPv6. I will try backup flow and update you.
Many Thanks

@AlejandraOliver
Copy link

Hi @Monika-Research, could you resolve the problem?

@Monika-Research
Copy link
Author

Hi @AlejandraOliver, I still couldn't get backup to working even with kernel version 6.1.2

@Monika-Research
Copy link
Author

Hi @matttbe, I am still facing issue with backup flow. Any leads?

Thanks,
Monika

@matttbe
Copy link
Member

matttbe commented Jun 1, 2023

Hi @Monika-Research

What is the issue exactly (there are a few in the comments)? You set the backup flag on one endpoint but it is not taken into account?

Do you mind trying to reproduce the bug on a recent v6.3 kernel release (last one is the 6.3.5 from what I can see) with a recent version of ip (ideally a v6.3 release, it is easy to compile if needed)?

Then we will need the output of ip mptcp endpoint but also the output of ip mptcp monitor and packet traces (e.g. tcpdump -i <iface linked to the endpoint set as backup> -w <file> + compress as zip to be able to send it here) while trying to reproduce the issue (ideally with a minimum of data being transferred).

@Monika-Research
Copy link
Author

Monika-Research commented Jun 1, 2023

Hi Matt,

What is the issue exactly (there are a few in the comments)? You set the backup flag on one endpoint but it is not taken into account?

Yes, it is backup=0 always in the "ip mptcp monitor" and data is transmitted over that subflow.

Do you mind trying to reproduce the bug on a recent v6.3 kernel release (last one is the 6.3.5 from what I can see) with a recent version of ip (ideally a v6.3 release, it is easy to compile if needed)?

Sure. I compiled the export branch and it is 6.4 I believe. should be fine?

@matttbe
Copy link
Member

matttbe commented Jun 1, 2023

Yes, it is backup=0 always in the "ip mptcp monitor" and data is transmitted over that subflow.

Strange, looks like a bug or a config issue.

Sure. I compiled the export branch and it is 6.4 I believe. should be fine?

Yes, using the export branch (make sure it is up to date) is even better!
Do not forget to also use a recent version for ip

@AlejandraOliver
Copy link

AlejandraOliver commented Jun 1, 2023

Hi @matttbe , I have compiled the kernel from the repository and have mptcp working correctly, but I have a question with schedulers. I'm testing with them and I would like to set one of the interfaces as a backup so that it will only be activated in case the others go down. Do you know the command I should use to do it,? I've tried

ip mptcp endpoint add IP dev interface backup

but I think it's not working.

Thanks.

@matttbe
Copy link
Member

matttbe commented Jun 1, 2023

@AlejandraOliver you need to specify if this endpoint will be used to create subflow (subflow: client side) or to signal addresses (signal: server side), so you should use something like:

ip mptcp endpoint add <IP> dev <interface> subflow backup  ## client
ip mptcp endpoint add <IP> dev <interface> signal  backup  ## server

As mentioned in my previous comment, make sure both the kernel (uname -a) and ip (ip -Version) are up to date (>= 6.3). If you still have the issue, then we will need the output of ip mptcp endpoint + ip mptcp monitor + tcpdump (see above)).

@matttbe
Copy link
Member

matttbe commented Jun 1, 2023

FYI, you can see how we are validating this feature there:

backup_tests()
{
# single subflow, backup
if reset "single subflow, backup"; then
pm_nl_set_limits $ns1 0 1
pm_nl_set_limits $ns2 0 1
pm_nl_add_endpoint $ns2 10.0.3.2 flags subflow,backup
run_tests $ns1 $ns2 10.0.1.1 0 0 0 slow nobackup
chk_join_nr 1 1 1
chk_prio_nr 0 1
fi
# single address, backup
if reset "single address, backup"; then
pm_nl_set_limits $ns1 0 1
pm_nl_add_endpoint $ns1 10.0.2.1 flags signal
pm_nl_set_limits $ns2 1 1
run_tests $ns1 $ns2 10.0.1.1 0 0 0 slow backup
chk_join_nr 1 1 1
chk_add_nr 1 1
chk_prio_nr 1 1
fi
# single address with port, backup
if reset "single address with port, backup"; then
pm_nl_set_limits $ns1 0 1
pm_nl_add_endpoint $ns1 10.0.2.1 flags signal port 10100
pm_nl_set_limits $ns2 1 1
run_tests $ns1 $ns2 10.0.1.1 0 0 0 slow backup
chk_join_nr 1 1 1
chk_add_nr 1 1
chk_prio_nr 1 1
fi
if reset "mpc backup"; then
pm_nl_add_endpoint $ns2 10.0.1.2 flags subflow,backup
run_tests $ns1 $ns2 10.0.1.1 0 0 0 slow
chk_join_nr 0 0 0
chk_prio_nr 0 1
fi
if reset "mpc backup both sides"; then
pm_nl_add_endpoint $ns1 10.0.1.1 flags subflow,backup
pm_nl_add_endpoint $ns2 10.0.1.2 flags subflow,backup
run_tests $ns1 $ns2 10.0.1.1 0 0 0 slow
chk_join_nr 0 0 0
chk_prio_nr 1 1
fi
if reset "mpc switch to backup"; then
pm_nl_add_endpoint $ns2 10.0.1.2 flags subflow
run_tests $ns1 $ns2 10.0.1.1 0 0 0 slow backup
chk_join_nr 0 0 0
chk_prio_nr 0 1
fi
if reset "mpc switch to backup both sides"; then
pm_nl_add_endpoint $ns1 10.0.1.1 flags subflow
pm_nl_add_endpoint $ns2 10.0.1.2 flags subflow
run_tests $ns1 $ns2 10.0.1.1 0 0 0 slow backup
chk_join_nr 0 0 0
chk_prio_nr 1 1
fi
}

@AlejandraOliver
Copy link

AlejandraOliver commented Jun 1, 2023

@AlejandraOliver you need to specify if this endpoint will be used to create subflow (subflow: client side) or to signal addresses (signal: server side), so you should use something like:

ip mptcp endpoint add dev subflow backup ## client
ip mptcp endpoint add dev signal backup ## server

Hi @matttbe, I don't quite understand what depends on whether you use subflow or signal.


I have a virtual scenario in VirtualBox with 2 Ubuntu 22.04 LTS machines. On both I have the v6.3-rc6 kernel from the export branch compiled and installed. In addition, I have already configured the routing, and I have also compiled and loaded in the kernel the different types of schedulers (bpf_red, bpf_rr,bpf_first and bpf_bkup).

Each machine has 4 interfaces:

MACHINE1 (client)
eth0 is used as NAT (connection to the outside).
eth1: has the IP 10.0.0.1 inside the internal network 10.0.0.0/24
eth2: has the IP 192.51.100.1 within the internal network 192.51.0.0/24
eth3; has IP 172.16.0.1 within internal network 172.16.0.0/24

MACHINE2 (server)
eth0 is used as NAT (connection with the outside).
eth1: has IP 10.0.0.2 within the internal network 10.0.0.0/24
eth2: has the IP 192.51.100.2 within the internal network 192.51.0.0/24
eth3; has IP 172.16.0.2 within internal network 172.16.0.0/24


My goal is for both machines to communicate via the 3 interfaces (eth1, eth2 and eth3). The problem is that when testing with the schedulers, for example the redundant, if I put one of the interfaces as a backup and the others go down, this interface is not reactivated.
I think that I don't know how to configure the path manager correctly (set the routes as subflow, backup or signal using the command $ip mptcp endpoint add dev subflow, backup, signal).

I would be very grateful if you could show me an example of how you would configure the 3 interfaces in client and server, with one of them as a backup. Like, what path manager commands (ip mptcp endpoint add dev subflow backup-
ip mptcp endpoint add dev signal backup) I need to use, because when I set the connection, the flag backup is set to 0, like @Monika-Research said.

So I think It's maybe because I don't set the backup subflow correctly, or a configuration problem not yet solved in the export branch.

Thanks a lot in advance!!!

@matttbe
Copy link
Member

matttbe commented Jun 2, 2023

@AlejandraOliver you need to specify if this endpoint will be used to create subflow (subflow: client side) or to signal addresses (signal: server side), so you should use something like:

ip mptcp endpoint add dev subflow backup ## client ip mptcp endpoint add dev signal backup ## server

Hi @matttbe, I don't quite understand what depends on whether you use subflow or signal.

The path-manager needs to know if the device is interested by either:

  • using additional IP addresses to create new subflows → typically it is the client that creates new subflows
  • signalling the other peer it has additional IP addresses available, so this other peer can create new subflows to these new addresses → typically it is the server that accepts new connections requests.

So in short, you should use the subflow flag on MACHINE1 (client) and signal on MACHINE2 (server) in addition to the backup one.

I have also compiled and loaded in the kernel the different types of schedulers (bpf_red, bpf_rr,bpf_first and bpf_bkup).

These schedulers are just test toys (maybe with bugs) and except the last one, they don't take into account the backup flag if I'm not mistaken. Except if you want to improve these schedulers (see #75), we don't recommend using them for the moment.

I would be very grateful if you could show me an example of how you would configure the 3 interfaces in client and server, with one of them as a backup.

Something like that (quickly done, not tested):

client:

ip mptcp limits set subflows 4 ## high just to be on the safe side

## ip mptcp endpoint add 10.0.0.1 dev eth1 subflow backup ## no need to configure it if your app uses this IP for the initial subflow (except if you want to add the backup flag)
ip mptcp endpoint add 192.51.100.1 dev eth2 subflow backup ## as backup?
ip mptcp endpoint add 172.16.0.1 dev eth3 subflow backup   ## as backup?

# no need to configure the routes as you are in the same LAN

server:

ip mptcp limits set add_addr_accepted 4 ## high just to be on the safe side

## ip mptcp endpoint add 10.0.0.2 dev eth1 signal ## no need to configure it if your app uses this IP for the initial subflow (except if you want to add the backup flag)
ip mptcp endpoint add 192.51.100.2 dev eth2 signal backup ## as backup?
ip mptcp endpoint add 172.16.0.2 dev eth3 signal backup   ## as backup?

# no need to configure the routes as you are in the same LAN

If you still have the issue, you know what to share ;-)

@AlejandraOliver
Copy link

AlejandraOliver commented Jun 2, 2023

@matttbe Following the configurations you have mentioned, I still have the problem that even if the endpoint is set as backup, data is transmitted.
I have created a repo with all the files that you said


I have the following configuration:
Client:

ip mptcp limits set subflows 4 add_addr_accepted 4 

ip mptcp endpoint add 192.51.100.1 dev eth2 subflow 
ip mptcp endpoint add 172.16.0.1 dev eth3 subflow backup 

Server:

ip mptcp limits set subflows 4 add_addr_accepted 4 

ip mptcp endpoint add 192.51.100.2 dev eth2 signal
ip mptcp endpoint add 172.16.0.2 dev eth3 signal backup 

If I captured in Wireshark, I see the following things:

  1. The backup flag is set to 1 in the SYN and SYN+ACK packets containing the MP_JOIN option between the interfaces 172.0.0.1-10.0.0.2.
  2. There is no SYN and SYN+ACK packets (with MP_JOIN option) between 172.0.0.1-192.51.100.2.
  3. The backup flag is set to 0 in the SYN and SYN+ACK packets containing the MP_JOIN option between the interfaces 172.0.0.1-172.0.0.2.

In 1, interfaces of different network should not create subflow, I think.


I have a question apart of this. If all the interfaces between the 2 machines were in the same internal network, I need to configure any type of routing before doing these path management commands? Or is simply having each interface assigned with an IP address is enough?

Thank you very much.

@matttbe
Copy link
Member

matttbe commented Jun 2, 2023

I have created a repo with all the files that you said

Thanks! You can also put the binaries in a .zip and upload it here ;-)

The traces you made with tcpdump don't contain the SYN (and a .pcap by using -w option would be more helpful :) ). So I will just rely on what you saw with Wireshark.

Also the output of ip mptcp monitor seems to suggest you have an old version of ip (but you were able to put the backup flag so I guess that's OK)

And finally, I guess you have mptcpd running which already set endpoints → maybe there are some conflicts there and the backup flag is not taken into account. If you do the configurations manually, stop (and disable) mptcp service. (or configure things in mptcpd)

If I captured in Wireshark, I see the following things:

1. The backup flag is set to 1 in the SYN and SYN+ACK packets containing the MP_JOIN option between the interfaces 172.0.0.1-10.0.0.2.

2. There is no SYN and SYN+ACK packets (with MP_JOIN option) between 172.0.0.1-192.51.100.2.

3. The backup flag is set to 0 in the SYN and SYN+ACK packets containing the MP_JOIN option between the interfaces 172.0.0.1-172.0.0.2.

In 1, interfaces of different network should not create subflow, I think.

mmh, a few things:

  • so it looks like the backup only works the first time but not after, right? Strange but maybe there is a bug.
  • the PM doesn't know which endpoints go well with which other ones. I guess in your case, you just want one subflow per interface sharing the same LAN? You can add firewall rules to forbid creating connections to another network.
  • for the connection between 172.16.0.1 and 10.0.0.2 for example, if you don't add any other rules, I guess it will use the interface eth1 which is linked to 10.0.0.0/24, no? You can check with:
ip route get from 172.16.0.1 10.0.0.2 ## then check the "dev" in the output

I have a question apart of this. If all the interfaces between the 2 machines were in the same internal network, I need to configure any type of routing before doing these path management commands? Or is simply having each interface assigned with an IP address is enough?

We need routing rules to tell the kernel that the packets from a specific IP address have to be routed through the device linked to this IP address. So in your case, to have the traffic from 172.16.0.1 using the eth3 device and not another one to be able to receive the replies.

Typically, a default route is configured which means that all the traffic is routed through a specific interface. There are also routes per other IP addresses you have to say that the traffic within the same prefix is routed through the right interface. If the two devices have IPs within the same prefix, you certainly already have such routes:

$ ip route
(...)
10.0.0.0/16 dev eth1 proto kernel scope link src 10.0.0.1 linkdow
(...)

But typically you don't have rules to force routing the traffic from a specific IP to the interface linked to this IP:

ip rule from <local IP> table <table>
ip route add default via <gateway IP> table <table>

e.g.

ip rule from 172.16.0.1 table 42
ip route add default via 172.16.0.2 table 42

@matttbe
Copy link
Member

matttbe commented Jun 2, 2023

I disabled mptcpd but, what should be the output of ip mptcp endpoint show now? I'm very confused.

only the endpoints you added. You can always use ip mptcp endpoint flush and re-add them manually.

@AlejandraOliver
Copy link

I disabled mptcpd and now the output of ip mptcp endpoint show is:

Client:

192.51.100.1 id 8 subflow dev enp0s9 (this is eth2)
172.16.0.1 id 9 subflow backup dev enp0s10 (this is eth3)

Server

192.51.100.2 id 8 signal dev enp0s9 (this is eth2)
172.16.0.2 id 9 signal backup dev enp0s10 (this is eth3)

@matttbe
Copy link
Member

matttbe commented Jun 2, 2023

looks good now. We can see the backup flag.

@AlejandraOliver
Copy link

AlejandraOliver commented Jun 2, 2023

Now, no data is retransmitted initially through the backup subflow (in Wireshark the backup flag is set to 1 in both directions). This is perfect but, when eth1 and eth2 go down, it should be reactivated, right?

@matttbe
Copy link
Member

matttbe commented Jun 2, 2023

This is perfect but, when eth1 and eth2 go down, it should be reactivated, right?

It should yes. It doesn't? with the default scheduler?

@AlejandraOliver
Copy link

AlejandraOliver commented Jun 2, 2023

No, when I put down eth1 and eth2, eth3 is still in backup. I'm using default scheduler

@matttbe
Copy link
Member

matttbe commented Jun 2, 2023

What do you mean by saying "it is still in backup"?
The flag will not change but it if the application is still pushing data, you should see traffic on eth3 (can be seen with tcpdump, wireshark, ifstat, bwmon, interface counters, etc.)

@AlejandraOliver
Copy link

I mean that data is not transmitted on that interface (I'm seeing this with ifstat)

@AlejandraOliver
Copy link

From what I've been seeing in Wireshark and ifstat, it reactivates for a very short period of time (in wireshark two packets are sent with the PUSH flag set) and then it cuts off and resets to zero.

@matttbe
Copy link
Member

matttbe commented Jun 2, 2023

then it cuts off and resets to zero.

do you mean a RST is sent?

Could you share the packet traces please?
Plus the output of:

  • sysctl net.mptcp
  • nstat -z just before disabling the interfaces and after 1 second
  • ss -natiMe also before and after like ↑

@AlejandraOliver
Copy link

AlejandraOliver commented Jun 2, 2023

The Wireshark capture is in my repo with name prueba.rar.

The output of nstat -z and ss -natiMe are in my repo too (I can't share any zip here, I don't know why :(

The output of sysctl net.mptcp:

net.mptcp.add_addr_timeout = 120
net.mptcp.allow_join_initial_addr_port = 1
net.mptcp.checksum_enabled = 0
net.mptcp.enabled = 1
net.mptcp.pm_type = 0
net.mptcp.scheduler = default
net.mptcp.stale_loss_cnt = 4

I think what happens is that it is reactivated to transmit the packets that were left unsent in the downed subflows, and when it does so, it stops transmitting new data.

@AlejandraOliver
Copy link

AlejandraOliver commented Jun 2, 2023

And finally, I guess you have mptcpd running which already set endpoints → maybe there are some conflicts there and the backup flag is not taken into account. If you do the configurations manually, stop (and disable) mptcp service. (or configure things in mptcpd)

One thing about that. If I want to set the same enviroment but with using mptcpd, I need to install one of the kernel of the export branch? Otherwise, installing a official kernel >v5.6, I wouldn't have any kind of scheduler, only the default scheduling that is preinstalled in the kernel, right?

Thanks!!

@matttbe
Copy link
Member

matttbe commented Jun 6, 2023

Thank you for the new info.

I started to write this reply before but I got interrupted. I didn't check the traces and log yet but here is already what I already wrote (and I have a few meetings coming, not sure I can check them soon)

The Wireshark capture is in my repo with name prueba.rar.

Please next time use .gz or .zip (or .xz or .7z but they are not accepted by GitHub I think) but not a proprietary file format.

I think what happens is that it is reactivated to transmit the packets that were left unsent in the downed subflows, and when it does so, it stops transmitting new data.

Interesting. I don't think we have tests where we disable one interface while data are being transferred. We should I suppose.

One thing about that. If I want to set the same enviroment but with using mptcpd, I need to install one of the kernel of the export branch? Otherwise, installing a official kernel >v5.6, I wouldn't have any kind of scheduler, only the default scheduling that is preinstalled in the kernel, right?

mptcpd doesn't handle the different packet schedulers (yet?), only the path-managers.
Please try to use at least the latest stable kernel (or latest LTS).

@AlejandraOliver
Copy link

Okay, I will keep checking it. Thank you for your time!

@Monika-Research
Copy link
Author

Hi,

I have some dependency on kernel version and ip version with our products/devices. But I really wanted to get the backup option working. What can be done in this case? Can someone point to the exact patches required to get it working? Much appreciate guidance. Thank you.

matttbe pushed a commit that referenced this issue Mar 27, 2024
In case when is64 == 1 in emit(A64_REV32(is64, dst, dst), ctx) the
generated insn reverses byte order for both high and low 32-bit words,
resuling in an incorrect swap as indicated by the jit test:

[ 9757.262607] test_bpf: #312 BSWAP 16: 0x0123456789abcdef -> 0xefcd jited:1 8 PASS
[ 9757.264435] test_bpf: #313 BSWAP 32: 0x0123456789abcdef -> 0xefcdab89 jited:1 ret 1460850314 != -271733879 (0x5712ce8a != 0xefcdab89)FAIL (1 times)
[ 9757.266260] test_bpf: #314 BSWAP 64: 0x0123456789abcdef -> 0x67452301 jited:1 8 PASS
[ 9757.268000] test_bpf: #315 BSWAP 64: 0x0123456789abcdef >> 32 -> 0xefcdab89 jited:1 8 PASS
[ 9757.269686] test_bpf: #316 BSWAP 16: 0xfedcba9876543210 -> 0x1032 jited:1 8 PASS
[ 9757.271380] test_bpf: #317 BSWAP 32: 0xfedcba9876543210 -> 0x10325476 jited:1 ret -1460850316 != 271733878 (0xa8ed3174 != 0x10325476)FAIL (1 times)
[ 9757.273022] test_bpf: #318 BSWAP 64: 0xfedcba9876543210 -> 0x98badcfe jited:1 7 PASS
[ 9757.274721] test_bpf: #319 BSWAP 64: 0xfedcba9876543210 >> 32 -> 0x10325476 jited:1 9 PASS

Fix this by forcing 32bit variant of rev32.

Fixes: 1104247 ("bpf, arm64: Support unconditional bswap")
Signed-off-by: Artem Savkov <asavkov@redhat.com>
Tested-by: Puranjay Mohan <puranjay12@gmail.com>
Acked-by: Puranjay Mohan <puranjay12@gmail.com>
Acked-by: Xu Kuohai <xukuohai@huawei.com>
Message-ID: <20240321081809.158803-1-asavkov@redhat.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
matttbe pushed a commit that referenced this issue May 20, 2024
Recent additions in BPF like cpu v4 instructions, test_bpf module
exhibits the following failures:

  test_bpf: #82 ALU_MOVSX | BPF_B jited:1 ret 2 != 1 (0x2 != 0x1)FAIL (1 times)
  test_bpf: #83 ALU_MOVSX | BPF_H jited:1 ret 2 != 1 (0x2 != 0x1)FAIL (1 times)
  test_bpf: #84 ALU64_MOVSX | BPF_B jited:1 ret 2 != 1 (0x2 != 0x1)FAIL (1 times)
  test_bpf: #85 ALU64_MOVSX | BPF_H jited:1 ret 2 != 1 (0x2 != 0x1)FAIL (1 times)
  test_bpf: #86 ALU64_MOVSX | BPF_W jited:1 ret 2 != 1 (0x2 != 0x1)FAIL (1 times)

  test_bpf: #165 ALU_SDIV_X: -6 / 2 = -3 jited:1 ret 2147483645 != -3 (0x7ffffffd != 0xfffffffd)FAIL (1 times)
  test_bpf: #166 ALU_SDIV_K: -6 / 2 = -3 jited:1 ret 2147483645 != -3 (0x7ffffffd != 0xfffffffd)FAIL (1 times)

  test_bpf: #169 ALU_SMOD_X: -7 % 2 = -1 jited:1 ret 1 != -1 (0x1 != 0xffffffff)FAIL (1 times)
  test_bpf: #170 ALU_SMOD_K: -7 % 2 = -1 jited:1 ret 1 != -1 (0x1 != 0xffffffff)FAIL (1 times)

  test_bpf: #172 ALU64_SMOD_K: -7 % 2 = -1 jited:1 ret 1 != -1 (0x1 != 0xffffffff)FAIL (1 times)

  test_bpf: #313 BSWAP 16: 0x0123456789abcdef -> 0xefcd
  eBPF filter opcode 00d7 (@2) unsupported
  jited:0 301 PASS
  test_bpf: #314 BSWAP 32: 0x0123456789abcdef -> 0xefcdab89
  eBPF filter opcode 00d7 (@2) unsupported
  jited:0 555 PASS
  test_bpf: #315 BSWAP 64: 0x0123456789abcdef -> 0x67452301
  eBPF filter opcode 00d7 (@2) unsupported
  jited:0 268 PASS
  test_bpf: #316 BSWAP 64: 0x0123456789abcdef >> 32 -> 0xefcdab89
  eBPF filter opcode 00d7 (@2) unsupported
  jited:0 269 PASS
  test_bpf: #317 BSWAP 16: 0xfedcba9876543210 -> 0x1032
  eBPF filter opcode 00d7 (@2) unsupported
  jited:0 460 PASS
  test_bpf: #318 BSWAP 32: 0xfedcba9876543210 -> 0x10325476
  eBPF filter opcode 00d7 (@2) unsupported
  jited:0 320 PASS
  test_bpf: #319 BSWAP 64: 0xfedcba9876543210 -> 0x98badcfe
  eBPF filter opcode 00d7 (@2) unsupported
  jited:0 222 PASS
  test_bpf: #320 BSWAP 64: 0xfedcba9876543210 >> 32 -> 0x10325476
  eBPF filter opcode 00d7 (@2) unsupported
  jited:0 273 PASS

  test_bpf: #344 BPF_LDX_MEMSX | BPF_B
  eBPF filter opcode 0091 (@5) unsupported
  jited:0 432 PASS
  test_bpf: #345 BPF_LDX_MEMSX | BPF_H
  eBPF filter opcode 0089 (@5) unsupported
  jited:0 381 PASS
  test_bpf: #346 BPF_LDX_MEMSX | BPF_W
  eBPF filter opcode 0081 (@5) unsupported
  jited:0 505 PASS

  test_bpf: #490 JMP32_JA: Unconditional jump: if (true) return 1
  eBPF filter opcode 0006 (@1) unsupported
  jited:0 261 PASS

  test_bpf: Summary: 1040 PASSED, 10 FAILED, [924/1038 JIT'ed]

Fix them by adding missing processing.

Fixes: daabb2b ("bpf/tests: add tests for cpuv4 instructions")
Signed-off-by: Christophe Leroy <christophe.leroy@csgroup.eu>
Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
Link: https://msgid.link/91de862dda99d170697eb79ffb478678af7e0b27.1709652689.git.christophe.leroy@csgroup.eu
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
No open projects
MPTCP Bugs
  
Needs triage
Development

No branches or pull requests

3 participants