Mwan3track not working properly on wireguard interface #10712
Assignees
Comments
|
@feckert Any update about this issue? |
|
Sorry I have not further explored the subject. are strange. First of all we should investigate why this message are shown during mwan3 restart. |
These errors are always there, maybe it is caused by my mwan3 and firewall configs. Is there any thing I can do for these errors? |
|
@feckert I'm not goot at iptables, is there any way to get the ping working with wireguard interfaces? |
|
I have seen that you do not use the ping from busybox. I think this has everything to do with the interface binding that doesn't work by ip-utils, because the version is over 10 years old. In the pullrequest #12229 from @aaronjg there are some changes that would fix this. Please give them a try. And if it does not work please reopen this Issue. |
|
@feckert |
|
Can you confirm that busybox ping works when mwan3 is not running? If that doesn't work, can you provide some additional diagnostics:
|
|
@aaronjg And the output when mwan3 was running: |
|
Is that the correct src ip? It looks like it is a wan, but has a private IP address. Can you share the output of I am wondering if your VPN set up two half internet routes rather than one default route. Those would then be copied over to the other routing tables and could cause issues. |
|
@aaronjg |
|
It looks like you have no routes beyond the subnet for the wireguard interface. If you don't have a default route, then mwan3 isn't going to be able to copy anything over into the relevant table. I think you need to configure your wireguard to add the default routes, or do it manually after the link comes up. I'm curious how it is working without mwan3 enabled. Can you turn mwan3 off and post the same results from the iptables logs? |
|
The log when mwan3 stoped: And how to add the default route for wireguard interface? |
|
Sorry, I don't use wireguard so I can't help you there. With OtherVPNs, the routes are often pushed to you and the VPN client then sets them up. You could do it manually with |
|
I think it's not about the default route thing, I've add the defalut route for wireguard interface, and restarted the mwan3, but the ping still not working. |
|
@feckert |
|
You may also need a static route to the wireguard endpoint in your table. It does not look like the wireguard routes were set up correctly. Also, it looks like you are running a development build. Can you try it with the latest stable release? |
|
@aaronjg |
|
@wackejohn I just ran into this issue myself. It appears that something is handled differently when the outgoing interface is 192.168.X.X. Can you try applying this commit, and see if it fixes things for you? |
|
@aaronjg And it seemed that the icmp packages didn't pass through the vpn. |
|
So, with the trace route it looks like the packets are going through your vpn (192.168.120.1) when you don’t specify the interface, but not when you do? Looks like the issue I ran into is different from the issue I had. |
|
@aaronjg the icmp packets should pass through the vpn (wana or wanb) wether specify the interface or not. |
|
yes, but there are different rules that are applied to route them out of the interface when you specify the interface vs when you don't. I am surprised that they are being blocked with the new patch. That should make the ICMP echo packets skip the mwan3 rules entirely when you are specifying the bound interface. What kernel version are you using? I'll do the best I can to help, but I may not be able to reproduce since you are on the snapshot... A few more diagnostics - can you repeat the firewall logging with the new patch? I want to see if these packets are somehow still getting marked. Can you try pinging an IP , eg 1.1.1.1, not covered by your mwan3 rule? What is the output of: |
|
So, not to try and create noise here, but I recently switched from OpenVPN to Wireguard with mwan3, evaluating Wireguard vs OpenVPN, I found the IPv6 side was a bit off, but the latest changes from @aaronjg, fixes those. I have two Wireguard VPN interfaces as clients load balanced with Mullvad and it works OK. I'm on 19.07.3 kernel 4.14.180 on a Linksys WRT3200ACM. Currently running mwan3 2.8.6, but I'm aware you've tried the changes and it hasn't worked. Have you tried stripping back the mwan3 config to a very minimal setup? |
|
James, not noise at all. Thanks for sharing your experience. What sort of
IPv4 address are you getting from Mullvad? I wonder if some of john’s
problems is that he is getting a private, 192.168.X.x address from his vpn.
…On Tuesday, June 16, 2020, James White ***@***.***> wrote:
So, not to try and create noise here, but I recently switched from OpenVPN
to Wireguard with mwan3, evaluating Wireguard vs OpenVPN, I found the IPv6
side was a bit off, but the latest changes from @aaronjg
<https://github.com/aaronjg>, fixes those.
I have two Wireguard VPN interfaces as clients load balanced with Mullvad
and it works OK. I'm on 19.07.3 on a Linksys WRT3200ACM. Currently running
2.8.6, but I'm aware you've tried the changes and it hasn't worked.
Have you tried stripping back the mwan3 config to a very minimal setup?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#10712 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAHD3CBW5ZUNCIK3PYH5NVLRW65LPANCNFSM4JXG4K5Q>
.
--
Sent from Gmail Mobile
|
|
@aaronjg Mullvad provides an IPv4 in the 10.0.0.0/8 range and a single /128 fc00::/7 ULA as they use NAT66. In order to have IPv4 and IPv6 working with mwan3, I have a The IPv6 side seems unrelated in this case, because it looks like @wackejohn can't even use the Wireguard inferface once mwan3 is enabled, but in my case: Perhaps it is related to the 192.168.0.0 range. |
|
@aaronjg My openwrt version: Kernel version: And the firewall log: |
|
@wackejohn My only thoughts would be the fact you're on the snapshot builds and the kernel branch is different 4.x vs 5.x. I'll happy provide any configuration of the Wireguard clients and network config I have setup to see if you find anything different that jumps out, but that's the only area I can see at that moment that could explain the difference in behaviour. |
My ISP provide the dynamic pppoe connection, the ip addr will change on ervery reconnect. It was my mistake that I copied the previous log. The right log: |
|
Got it. So now we can see that that it is for some reason trying to route the encrypted vpn traffic back through the VPN. You can see that the log with mwan3 is much longer than the log without. After the first successful ping, each ping attempt triggers a cascade of 17 packets, each 64 bytes longer than the previous. Is your VPN perhaps connected on port 443? If so can you delete the rule titled 'https?' |
Nope. My VPN was configred on port 51888 udp, as the log showed, while the https is tcp port. |
|
Many VPNs use port 443 (https port) to avoid being blocked by ISP firewalls. Is any part of your connection going over 443 or https? You included it in the minimal config - so I was wondering what it is needed for. Something still seems wrong from the IP tables log: From the routing table: Do you have any idea why the packets are getting assigned a source address of 180.112.50.137? From your routing table, it looks like the src for ppoe-wan should be 117.85.170.4 Another thing that looks weird is your mwan3 config has both a |
I'm using the wireguard vpn, and when I use other vpns with tun device, the
The https rule was generated by the mwan3, so it was one of the default rule of mwan3.
Because I capture the routing table and the log at the different time, so the ip addr was different, but this shouldn't matter, please ignore the different ip addr.
Yes, with the default rule of mwan3, the |
When things change like this, it makes it very difficult to help. To identify the issue, we have to look for anomolies and these changes look very much like something when wrong in mwan3.
Anyway, since you have added back the direct link route, and confirmed that the IP address from the log matches the correct outgoing IP address of your ppoe device - it appears that mwan3 is doing everything correctly. Unfortunately, this means that you have found a bug in wireguard or the kernel routing. You can actually see it here: So this packet gets marked correctly by iptables with 0x3f0. You never shared the output of The main routing table is (with 117.85.170.4 replaced with 180.112.50.137 presumably, and the corresponding gateway also replaced.): So that packet should match the fourth rule in the table and be routed out of ppoe. Somehow it appears to not matching that rule and being routed out of wana - which causes it to go back through ppoe. So it appears that this is an issue upstream, either with wireguard or the kernel. You should try to create a minimal configuration without mwan3 to reproduce this, and post a bug report on the main openwrt project. A minimal configuration will likely contain:
If you can't get it with those rules, you may need a few more commands to reproduce the behavior - but try to recreate the behavior with as few commands as possible since that will make it easier for the openwrt developers to help out. |
|
@wackejohn hi It should be like: and if you don't want all traffic go out via |
mwan3 does not copy over the default routes, it just inserts its own.
With only one default route, changing the metric on the default route will not change the traffic routing |
This was referenced Aug 20, 2020
Merged
|
ping -I should work, so it is bug in wireguard |
|
@ptpt52 The ttl was different from wana (121) to wanb (116): |
|
@wackejohn ttl is the issue? |
|
@ptpt52 |
|
@wackejohn |
|
@ptpt52 |
|
Please do more test, I notice one line ttl=116 in |
|
can you show your network? |
|
you could mask the ip |
|
|
update: the previous fix is not correct I post another fix here: https://github.com/x-wrt/x-wrt/blob/master/package/network/services/wireguard/patches/100-scrub-skb-sk-before-send-out.patch |
|
@ptpt52 |
|
yes that is what it should be
wackejohn <notifications@github.com> 于 2020年10月27日周二 08:49写道：
… @ptpt52 <https://github.com/ptpt52>
Working.
***@***.***_CZ:~# ping -I wana 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: seq=0 ttl=121 time=135.233 ms
64 bytes from 8.8.8.8: seq=1 ttl=121 time=135.210 ms
64 bytes from 8.8.8.8: seq=2 ttl=121 time=134.916 ms
64 bytes from 8.8.8.8: seq=3 ttl=121 time=135.305 ms
64 bytes from 8.8.8.8: seq=4 ttl=121 time=135.863 ms
64 bytes from 8.8.8.8: seq=5 ttl=121 time=135.562 ms
64 bytes from 8.8.8.8: seq=6 ttl=121 time=134.961 ms
64 bytes from 8.8.8.8: seq=7 ttl=121 time=135.389 ms
64 bytes from 8.8.8.8: seq=8 ttl=121 time=135.052 ms
^C
--- 8.8.8.8 ping statistics ---
9 packets transmitted, 9 packets received, 0% packet loss
round-trip min/avg/max = 134.916/135.276/135.863 ms
***@***.***_CZ:~# ping -I wanb 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: seq=0 ttl=116 time=136.593 ms
64 bytes from 8.8.8.8: seq=1 ttl=116 time=135.974 ms
64 bytes from 8.8.8.8: seq=2 ttl=116 time=136.143 ms
64 bytes from 8.8.8.8: seq=3 ttl=116 time=136.279 ms
64 bytes from 8.8.8.8: seq=4 ttl=116 time=136.264 ms
64 bytes from 8.8.8.8: seq=5 ttl=116 time=136.204 ms
64 bytes from 8.8.8.8: seq=6 ttl=116 time=135.992 ms
64 bytes from 8.8.8.8: seq=7 ttl=116 time=136.520 ms
64 bytes from 8.8.8.8: seq=8 ttl=116 time=136.105 ms
64 bytes from 8.8.8.8: seq=9 ttl=116 time=136.008 ms
64 bytes from 8.8.8.8: seq=10 ttl=116 time=135.789 ms
^C
--- 8.8.8.8 ping statistics ---
11 packets transmitted, 11 packets received, 0% packet loss
round-trip min/avg/max = 135.789/136.170/136.593 ms
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#10712 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABIVTYVKS6N6ZRXWRA7XP5TSMYKKVANCNFSM4JXG4K5Q>
.
|
|
@ptpt52 This is a wireguard problem and has nothing to do with the mwan3? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Maintainer: @feckert
Environment: OpenWrt SNAPSHOT, r11625-a512123a4b
Mwan3 version: 2.8.2-2
Description:
I'm using mwan3 and wireguard as client with latest openwrt snapshot.
And it seemed that the issue was caused by mwan3 itself, when mwan3 was running ,the
ping -I $Devicenot working, and then mwan3 would always mark the wireguard interface down, but the wireguard interface actually working...Below is the test resault:
And there is a discuss on the forum:
https://forum.openwrt.org/t/mwan3track-not-working-properly-on-wireguard/49557
The text was updated successfully, but these errors were encountered: