UDP client traffic exhibits very low performance/does not aggregate

[xzjq]

Expected Behavior

UDP traffic from clients can aggregate bandwidth to achieve performance greater than a single WAN, as TCP traffic does.

Current Behavior

UDP traffic from clients is generally far lower than even a single WAN.

Discussion:
Various clients on the network use VPNs that are configured to send traffic via UDP. Speeds are very slow compared to TCP based traffic. For example, a client using OpenVPN via TCP might reach 150 Mbps / 10 Mbps (down/up), but switching it to UDP reduces this performance by approximately an order of magnitude (*if aggregated)

If using shadowsocks/glorytun TCP, aggregation will work at this substantial speed penalty, which is slower than any individual WAN.

If switched to v2ray (as per optimization guide) the UDP traffic only uses a single interface (the master); ironically, this is faster than the aggregated UDP performance. Again, both of these scenarios are substantially slower than WAN-aggregated TCP-based traffic.

I have tried various permutations of shadowsocks/v2ray vless/v2ray vmess & OMR vpns (including none). Overall, the issue seems very similar to #2366.

Using TCP-based VPNs on the clients is not ideal, not least because long-running TCP connections disaggregate as well, e.g. #2936, which requires the client device TCP VPNs to be reset multiple times a day to restore multipath-aggregated performance.

Specifications

• OpenMPTCProuter version: 0.60beta1-5.4
• OpenMPTCProuter VPS version: 0.1029-test 5.4.207-mptcp [VPS has 4 cores & 4 GB RAM]
• OpenMPTCProuter platform: x86_64 [6 cores, 3 GB RAM]

[Ysurac]

UDP will always be slower than TCP for now. There is no good way to aggregate UDP traffic (I'm open if you know something).
Did you try using snapshot release on 6.1 kernel ? 5.4 based release will go to legacy?
V2Ray and XRay should aggregate UDP, how do you test UDP aggregation ?

[xzjq]

Thank you for getting back to me so quickly. I really appreciate your work.

I have tried the 6.1 kernel version for this release, and it is giving me trouble. On a different virtualized omr on the same hardware node that is running the 5.4 kernel omr referenced in this ticket (and of course using a different VPS) using the same WANs you see above it does not connect with mptcp, at all, and fails MPTCP Support Check. So, clearly that issue is not the local hardware/network or the WANs. I am in the process of investigating that further.

I test aggregation for UDP traffic by observing bandwidth consumption. The non-aggregated UDP traffic will use, say, 30 Mbps on the master WAN and the traffic on the other two WANs is ~0. When the UDP traffic is aggregated, I will see 6-8 Mbps on each WAN for about 20 Mbps on speedtest.

Now, that same VPN client traffic to the same OpenVPN server via TCP will use 60-70 Mbps on all three WANs, for a total of 150+ Mbps used on the OMR proxy (*initially, though after an hour or two it typically disaggregates one or two WANs and the disaggregated WANs show ~0 traffic again, as per #2936).

So initially the TCP-based VPN clients can get, say, 100 Mbps but after a few hours, after one or more WANs are dropped from aggregation for that ongoing connection, (but the WANs still show a green check in status) the client VPNs won't get more than ~30 Mbps until the OpenMPTCPRouter Settings Wizard "Save & Apply" is clicked and the client VPNs are reconnected. Then it's back to fully aggregated speed until it falls apart again.

UDP performance wouldn't be as high of a priority for me if TCP disaggregation could be prevented/recovered from without requiring kicking omr + client vpns multiple times a day (which necessitates logging back into multiple end user apps, etc).

[Ysurac]

For 6.1, server part must also be installed with 6.1 VPS script.
The MPTCP support check tab need to be updated/removed for this kernel.

As 5.4 will go legacy (not my fault, it's a nightmare to maintain an old kernel...), changes will only be made on 6.1.
Can you try again with 6.1 based snapshot ?

[xzjq]

Yes, for that, a brand new 6.1 VPS was installed using the "wget -O - http://www.openmptcprouter.com/server-beta/debian.sh | UPSTREAM6="yes" sh" instructions from #2961, and it was therefore upgraded from debian 11 to 12. Local omr vm is 0.60beta1-6.1 and vps is 0.1029-test 6.1.0-13-amd64

I am in the process of creating another VPS w/ 6.1 to test.

I am not versed in history of MPTCP, but when I skimmed their site it seemed to me the version that was included in the mainline kernel appears to be missing some features, and I wondered if it were as robust (especially after the MPTCP Support Check failure). Makes sense to stop supporting the old version though.

[Ysurac]

Use snapshot instead of beta for both router and GPS, I fixed many issues since beta.
5.15 was missing some features for OMR and not so stable, 6.1 release is usable, 6.6 will be even better (working on it for next next release).

[xzjq]

Great, thank you, I will try that. Will the snapshot vps script also perform the debian 11=>12 upgrade? Is it acceptable to start with a VPS on debian 12, or does the script expect to perform upgrade operations?

[Ysurac]

The snapshot script upgrade to Debian 12 is needed, better to start directly on Debian 12 if available.

[ccmks]

Hi @Ysurac where can I find the snapshot for both OMR and VPS?

Do you have any change log so far on the snapshot?

[Ysurac]

You can find all here: https://github.com/Ysurac/openmptcprouter/wiki/Snapshots
Changes are visible by commit messages, but not yet a real changelog.

[ccmks]

I tried to use the following in Hyper-V Gen 2

https://snapshots.openmptcprouter.com/6.1/x86_64/targets/x86/64/openmptcprouter-v0.60beta1-6.1-r0+24041-74e7f8ebbd-x86-64-generic-squashfs-combined-efi.vhdx

This won't boot at all. Do you know why?

[Ysurac]

You have a UEFI boot ?

[ccmks]

I got it working now, I need to change the boot order to boot from HDD as it was trying to boot from bunch of virtual NIC

[ccmks]

I do see there are bunch of proxy

What are the difference? Which one that gives most stable for UDP?

[Ysurac]

As it's new, you have to test what work best in your case.
I would like to use something better than doing some UDP over TCP, but there is no proxy with QUIC Multipath support for now.

[ccmks]

In the past that UDP works on V2RAY, but not for long then it would break again until I have to press "save & apply" button in wizard configuration.

With this new version, is there any improvement?

[ccmks]

I tried all V2RAY and X2RAY, none of them can work with UDP very well.

I am using one of my voip provider to make phone call it won't work. The workaround is to establish L2TP VPN over OMR, then the voip is working. However they don't seem to recovery connection if I disconnect and reconnect the multiple WAN alternatively.

Another thing I noticed that only OpenVPN TCP is working for UDP if I am using shadowsocks, the glory tun, MLVPN and DSVPN not working

I am using the snapshot with 6.1 kernel, on x86 Hyper-V environment

[xzjq]

However they don't seem to recovery connection if I disconnect and reconnect the multiple WAN alternatively.

That sounds like #2936.

[xzjq]

V2Ray and XRay should aggregate UDP, how do you test UDP aggregation ?

Okay, back to testing this, now on a 6.1-based OMR. XRay Shadowsocks 2022 does not aggregate UDP. During speedtests performed on a client device UDP-based VPN, OMR bandwidth page shows traffic only on master WAN and ~0 traffic on the other two WANs.

Ignorant question: would something like udp2tcp provide simpler UDP traffic tunneling over an MPTCP stream, and thus higher bandwidth aggregation? None of the VPN options in OMR I've tried seem to aggregate well.

[Ysurac]

XRay-Shadowsocks 2022 use Shadowsocks 2022 that is a socks 5 proxy, UDP is not over TCP so this is not aggregated or need to use a VPN.
udp2tcp and most tools like this are using fake TCP and can't be aggregated or else there is problem with packets ordering.
Something like Quic Multipath or UDP multipath would be a real solution, but nothing really available for now...

[xzjq]

Thanks for the explanation. I am confused by what was meant by this comment:

V2Ray and XRay should aggregate UDP

If Xray-Shadowsocks 2022 does not aggregate UDP, then which V2Ray or XRay methods do aggregate UDP?

[Ysurac]

As indicated in the wizard, XRay/V2Ray VLESS, VMESS and Trojan protocols can aggregate UDP as they are doing UDP over TCP.

[xzjq]

Thanks. Okay, I set VPN to None and then tested V2Ray/XRay Trojan, VMESS, and VLESS (+ VLESS Reality).

Unfortunately, no UDP aggregation: i.e. master WAN exhibiting 80-100 Mbps down and the other two WANs exhibiting ~30 Kbps. Similar non-aggregation on upload.

[Ysurac]

You have enabled "V2Ray/XRay UDP" in "Advanced settings" tab in System->OpenMPTCProuter ? It's disabled by default.

[xzjq]

Ah, thank you. That does aggregate now. Unfortunately, the aggregated UDP XRay (etc) performance from using all 3 WANs is less than half of using XRay (etc) in the non-aggregated state where it routes all traffic via master WAN.

That is to say, my client device UDP VPN client can get ~140 Mbps down in non-aggregated V2Ray/XRay, where all traffic goes via master WAN, but when UDP is enabled it drops to ~50 Mbps total (though that traffic is spread over all 3 WANs).

Is this the expected result/fundamental limitation, or does this seem like it could be tuned for improvement?

[ccmks]

unfortunately, UDP is currently well known not to be able to achieve aggregated speed Might need to use different solutions to solve UDP, like utilizing TCP tunnel for UDP traffic

…

On Wed, Jan 3, 2024, 7:15 AM xzjq ***@***.***> wrote: Ah, thank you. That does aggregate now. Unfortunately, the aggregated UDP XRay (etc) performance from using all 3 WANs is less than half of using XRay (etc) in the non-aggregated state where it routes all traffic via master WAN. That is to say, my client device UDP VPN client can get ~140 Mbps down in non-aggregated V2Ray/XRay, where all traffic goes via master WAN, but when UDP is enabled it drops to ~50 Mbps total (though that traffic is spread over all 3 WANs). Is this the expected result/fundamental limitation, or does this seem like it could be tuned for improvement? — Reply to this email directly, view it on GitHub <#3067 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALYEL6RNKI54DFUXU4OOD43YMSIJNAVCNFSM6AAAAABAL4FAWCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNZUGY3TIMZSGI> . You are receiving this because you commented.Message ID: ***@***.***>

[xzjq]

It seems that VMESS/VLESS are tunneling UDP over TCP when configured this way, and thus could attain MPTCP aggregated speed.

However, given that the total aggregated speed of the three WANs in this case is ~1/3 of the non-aggregated master WAN, this is not a benefit. It isn't clear whether this is a structural issue of UDP over TCP on VMESS/VLESS, etc, or if there is a need for optimization with my configuration.

[OscarParzon]

hello everyone.
Reading the thread, I have to understand that this problem will affect the SRT protocol (Low Latency Audio and Video Transport) ?
SRT is a UDP-based low-latency transmission protocol with ARQ packet loss recovery

[xzjq]

In general, OMR only benefits clients that are using native TCP traffic. Generally speaking, all other types of traffic (e.g. UDP) pay a significant performance-reducing penalty compared to just using a single one of the WANs directly. I suppose YMMV in edge cases such as if all your WANs are terrible.

If you have a firewall, you could consider only routing TCP traffic via OMR and routing UDP traffic via your best individual WAN for that (which may or may not be your OMR master WAN).

[ilikenwf]

XRay-Shadowsocks 2022 use Shadowsocks 2022 that is a socks 5 proxy, UDP is not over TCP so this is not aggregated or need to use a VPN. udp2tcp and most tools like this are using fake TCP and can't be aggregated or else there is problem with packets ordering. Something like Quic Multipath or UDP multipath would be a real solution, but nothing really available for now...

There is this but it looks unmaintained. https://github.com/greensea/mptunnel

[yue2388253]

Hi,

It seems that the current UDP aggregation solutions, including V2Ray and Glorytun TCP, are not performing well. Could anyone provide insights into why these methods might not be achieving optimal results?

Additionally, I am considering implementing an MPQUIC-based protocol (e.g., XQUIC ) as a VPN solution to improve UDP traffic aggregation. I would appreciate any thoughts on whether this could be a viable and effective approach.

Best,
Lin

[Ysurac]

XRay or V2Ray with UDP proxy enabled are sometimes better then the VPNs. All these are UDP over TCP, it's far from ideal.

XQUIC doesn't provide a real VPN, else I would implement it. If you develop a working VPN based on a library that support Mutlipath Quic, I would then be happy to add it in OpenMPTCProuter.

XRay and V2Ray are using Quic-Go that doesn't provide Multipath Quic support (only a very old fork using an old draft of Multipath Quic exist...), else this can be cool if they have this support...