Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

mwan3 connected_v6_temp is full #15951

Closed
zhihuiyuze opened this issue Jun 25, 2021 · 4 comments
Closed

mwan3 connected_v6_temp is full #15951

zhihuiyuze opened this issue Jun 25, 2021 · 4 comments
Assignees
@feckert

Comments

@zhihuiyuze
Copy link

zhihuiyuze commented Jun 25, 2021 •
edited

Hi,i'm runing mwan3 (2.8.16-1) on openwrt 19.07.06.
I found that the console has an output
mwan3 _connected_v6_temp is full maxelem 65536 reached
This will cause mwan3 to fail to diverge normally and luci to be inaccessible, etc.
After investigation,i think it is caused by limition of IPV6 Routing Table.
The routing table is imported init kernal by bird2,So how to remove these restrictions? Is this the problem?
Here's the configuration file

root@home-1-OpenWrt:/etc/config# cat mwan3

config rule 'vpn_ca_udp'
        option dest_ip '104.218.*.*
        option dest_port '52471'
        option proto 'udp'
        option sticky '0'
        option use_policy 'balanced'

config rule 'vpn_ca'
        option dest_ip '104.218.*.*'
        option dest_port '52471'
        option proto 'tcp'
        option sticky '0'
        option use_policy 'balanced'

config rule 'vpn_hk'
        option dest_ip '103.152.*.*'
        option sticky '0'
        option proto 'all'
        option use_policy 'balanced'

config rule 'heipv6_update'
        option dest_ip '64.62.200.2'
        option proto 'all'
        option sticky '0'
        option use_policy 'wan_only'

config rule 'direct'
        option proto 'all'
        option sticky '0'
        option ipset 'direct'
        option use_policy 'balanced'

config rule 'https_china'
        option dest_port '443'
        option ipset 'china'
        option sticky '1'
        option proto 'tcp'
        option use_policy 'balanced'

config rule 'China_v4'
        option proto 'all'
        option sticky '0'
        option ipset 'china'
        option use_policy 'balanced'

config rule 'ssh'
        option dest_port '22'
        option proto 'tcp'
        option sticky '1'
        option use_policy 'balanced'

config rule 'https'
        option dest_port '443'
        option proto 'tcp'
        option sticky '1'
        option use_policy 'VPN'

config rule 'ipv4_tcp'
        option sticky '0'
        option proto 'tcp'
        option dest_port '1024:65535'
        option use_policy 'VPN'

config rule 'default_rule_v6'
        option dest_ip '::/0'
        option family 'ipv6'
        option proto 'all'
        option sticky '0'
        option use_policy 'default'

config rule 'default_rule_v4'
        option dest_ip '0.0.0.0/0'
        option family 'ipv4'
        option proto 'all'
        option sticky '0'
        option use_policy 'VPN'

config globals 'globals'
        option mmx_mask '0x3F00'
        option rtmon_interval '5'

config interface 'wan'
        option enabled '1'
        option family 'ipv4'
        option count '1'
        option timeout '2'
        option interval '5'
        option down '3'
        option up '8'
        option initial_state 'online'
        option track_method 'ping'
        option size '56'
        option max_ttl '60'
        option check_quality '0'
        option failure_interval '5'
        option recovery_interval '5'
        option reliability '1'
        list track_ip '223.5.5.5'
        list track_ip '1.2.4.8'
        list track_ip '1.1.1.1'

config policy 'wan_only'
        option last_resort 'default'
        list use_member 'wan1_m50_w50'

config policy 'balanced'
        option last_resort 'default'
        list use_member 'wan3_m50_w50'
        list use_member 'wan1_m50_w50'
        list use_member 'wan2_m50_w50'

config interface 'wan2'
        option initial_state 'online'
        option family 'ipv4'
        option track_method 'ping'
        option reliability '1'
        option count '1'
        option size '56'
        option max_ttl '60'
        option check_quality '0'
        option timeout '2'
        option interval '5'
        option failure_interval '5'
        option recovery_interval '5'
        option down '3'
        option up '3'
        option enabled '1'
        list track_ip '223.5.5.5'
        list track_ip '1.2.4.8'
        list track_ip '1.1.1.1'

config interface 'wan3'
        option enabled '1'
        option initial_state 'online'
        option family 'ipv4'
        option track_method 'ping'
        option reliability '1'
        option count '1'
        option size '56'
        option max_ttl '60'
        option check_quality '0'
        option timeout '2'
        option interval '5'
        option failure_interval '5'
        option recovery_interval '5'
        option down '3'
        option up '3'
        list track_ip '223.5.5.5'
        list track_ip '1.2.4.8'
        list track_ip '1.1.1.1'

config policy 'wan2_only'
        option last_resort 'default'
        list use_member 'wan2_m50_w50'

config policy 'wan3_only'
        option last_resort 'default'
        list use_member 'wan3_m50_w50'

config interface 'Unmetered_VPN'
        option enabled '1'
        option initial_state 'online'
        option family 'ipv4'
        option track_method 'ping'
        option reliability '1'
        option count '1'
        option size '56'
        option max_ttl '60'
        option check_quality '0'
        option timeout '2'
        option interval '5'
        option failure_interval '5'
        option recovery_interval '5'
        option down '3'
        option up '3'
        list track_ip '10.200.200.1'
        list track_ip '1.1.1.1'

config policy 'VPN'
        option last_resort 'default'
        list use_member 'hk_1_vpn_m50_w50'
        list use_member 'Unmetered_VPN_m60_w55'
        list use_member 'Unmetered_VPN_2_m60_w60'
        list use_member 'Unmetered_VPN_3_m60_w60'

config interface 'Unmetered_VPN_2'
        option initial_state 'online'
        option family 'ipv4'
        option track_method 'ping'
        option reliability '1'
        option count '1'
        option size '56'
        option max_ttl '60'
        option check_quality '0'
        option timeout '2'
        option interval '5'
        option failure_interval '5'
        option recovery_interval '5'
        option down '3'
        option up '3'
        option enabled '1'
        list track_ip '10.200.200.1'
        list track_ip '1.1.1.1'

config interface 'Unmetered_VPN_3'
        option enabled '1'
        option initial_state 'online'
        option family 'ipv4'
        option track_method 'ping'
        option reliability '1'
        option count '1'
        option size '56'
        option max_ttl '60'
        option check_quality '0'
        option timeout '2'
        option interval '5'
        option failure_interval '5'
        option recovery_interval '5'
        option down '3'
        option up '3'
        list track_ip '10.200.200.1'
        list track_ip '1.1.1.1'

config interface 'hk_1_vpn'
        option enabled '1'
        option initial_state 'online'
        option family 'ipv4'
        option track_method 'ping'
        option reliability '1'
        option count '1'
        option size '56'
        option max_ttl '60'
        option check_quality '0'
        option timeout '2'
        option interval '5'
        option failure_interval '5'
        option recovery_interval '5'
        option down '3'
        option up '3'
        list track_ip '10.200.210.1'
        list track_ip '1.1.1.1'
        list track_ip '8.8.8.8'

config policy 'Unmetered'
        option last_resort 'unreachable'
        list use_member 'Unmetered_VPN_m60_w55'
        list use_member 'Unmetered_VPN_2_m60_w60'
        list use_member 'Unmetered_VPN_3_m60_w60'

config policy 'hk1'
        option last_resort 'unreachable'
        list use_member 'hk_1_vpn_m50_w50'

config member 'Unmetered_VPN_m60_w55'
        option interface 'Unmetered_VPN'
        option metric '60'
        option weight '55'

config member 'Unmetered_VPN_2_m60_w60'
        option interface 'Unmetered_VPN_2'
        option metric '60'
        option weight '60'

config member 'Unmetered_VPN_3_m60_w60'
        option interface 'Unmetered_VPN_3'
        option metric '60'
        option weight '60'

config member 'hk_1_vpn_m50_w50'
        option interface 'hk_1_vpn'
        option metric '50'
        option weight '50'

config member 'hk_1_vpn_m60_w50'
        option interface 'hk_1_vpn'
        option metric '60'
        option weight '50'

config member 'wan1_m50_w50'
        option interface 'wan'
        option metric '50'
        option weight '50'

config member 'wan2_m50_w50'
        option metric '50'
        option weight '50'
        option interface 'wan2'

config member 'wan3_m50_w50'
        option interface 'wan3'
        option metric '50'
        option weight '50'

bird.conf

router id 192.168.7.1;

define LOCAL_ASN = 141011;
define OUR_PREFIXES= [
    2602:feda:ab0::/44{44,48},
    2406:840:e240::/44{44,48}
];

roa4 table r4;
roa6 table r6;

log syslog all;

protocol device {
        scan time 60;
}

#protocol static {
#        ipv6;  
#        route 2602:feda:ab5::1/128  via 2602:feda:ab3::1%'br-home_1_arch';  
#}

protocol static BGP_Prefix{
    ipv6;
    route 2602:feda:ab3::/48 reject;
}

protocol kernel {
    learn;
#    persist;
        ipv6 {
                import all;
                export filter {
#                    krt_prefsrc = 2602:feda:ab3::1;
                accept;
            };
        };
}

protocol direct {
        #interface "dummy*";
    ipv6 { 
    import all;
    export all;
    };
}

function net_len_too_long(){
    case net.type {
        NET_IP4: return net.len > 24; # IPv4 CIDR 大于 /24 为太长
        NET_IP6: return net.len > 48; # IPv6 CIDR 大于 /48 为太长
        else: print "net_len_too_long: unexpected net.type ", net.type, " ", net; return false;
    }
}

define BOGON_ASNS = [
    0,                      # RFC 7607
    23456,                  # RFC 4893 AS_TRANS
    64496..64511,           # RFC 5398 and documentation/example ASNs
    64512..65534,           # RFC 6996 Private ASNs
    65535,                  # RFC 7300 Last 16 bit ASN
    65536..65551,           # RFC 5398 and documentation/example ASNs
    65552..131071,          # RFC IANA reserved ASNs
    4200000000..4294967294, # RFC 6996 Private ASNs
    4294967295              # RFC 7300 Last 32 bit ASN
];
define BOGON_PREFIXES_V4 = [
    0.0.0.0/8+,             # RFC 1122 'this' network
    10.0.0.0/8+,            # RFC 1918 private space
    100.64.0.0/10+,         # RFC 6598 Carrier grade nat space
    127.0.0.0/8+,           # RFC 1122 localhost
    169.254.0.0/16+,        # RFC 3927 link local
    172.16.0.0/12+,         # RFC 1918 private space 
    192.0.2.0/24+,          # RFC 5737 TEST-NET-1
    192.88.99.0/24+,        # RFC 7526 deprecated 6to4 relay anycast. If you wish to allow this, change `24+` to `24{25,32}`(no more specific)
    192.168.0.0/16+,        # RFC 1918 private space
    198.18.0.0/15+,         # RFC 2544 benchmarking
    198.51.100.0/24+,       # RFC 5737 TEST-NET-2
    203.0.113.0/24+,        # RFC 5737 TEST-NET-3
    224.0.0.0/4+,           # multicast
    240.0.0.0/4+            # reserved
];
define BOGON_PREFIXES_V6 = [
    ::/8+,                  # RFC 4291 IPv4-compatible, loopback, et al 
    0100::/64+,             # RFC 6666 Discard-Only
    2001::/32{33,128},      # RFC 4380 Teredo, no more specific
    2001:2::/48+,           # RFC 5180 BMWG
    2001:10::/28+,          # RFC 4843 ORCHID
    2001:db8::/32+,         # RFC 3849 documentation
    2002::/16+,             # RFC 7526 deprecated 6to4 relay anycast. If you wish to allow this, change `16+` to `16{17,128}`(no more specific)
    3ffe::/16+,             # RFC 3701 old 6bone
    fc00::/7+,              # RFC 4193 unique local unicast
    fe80::/10+,             # RFC 4291 link local unicast
    fec0::/10+,             # RFC 3879 old site local unicast
    ff00::/8+               # RFC 4291 multicast
];

function is_bogon_prefix() {
    case net.type {
        NET_IP4: return net ~ BOGON_PREFIXES_V4;
        NET_IP6: return net ~ BOGON_PREFIXES_V6;
        else: print "is_bogon_prefix: unexpected net.type ", net.type, " ", net; return false;
    }
}

function is_bogon_asn() {
    if bgp_path ~ BOGON_ASNS then return true;
    return false;
}

protocol rpki {
#        debug all;

        roa4 { table r4; };
        roa6 { table r6; };

        # Please, do not use rpki-validator.realmv6.org in production
        remote "rtr.rpki.cloudflare.com" port 8282;

        retry keep 5;
        refresh keep 30;
        expire 600;
}

filter peer_in_v4 {
        if (roa_check(r4, net, bgp_path.last) = ROA_INVALID) then
        {
                print "Ignore RPKI invalid ", net, " for ASN ", bgp_path.last;
                reject;
        }
        accept;
}

filter peer_in_v6 {
        if is_bogon_asn() then {
                print "is bogon asn", net, " for ASN ", bgp_path.last;
                reject;
        }
        if is_bogon_prefix() then {
                print "is bogon prefix", net, " for ASN ", bgp_path.last;
                reject;
        }
        if (roa_check(r6, net, bgp_path.last) = ROA_INVALID) then
        {
                print "Ignore RPKI invalid ", net, " for ASN ", bgp_path.last;
                reject;
        }
        accept;
}

function bgp_export() {
    if net !~ OUR_PREFIXES then return false;
    if is_bogon_asn() then return false;
    if is_bogon_prefix() then return false;
    
    if net_len_too_long() then {
#        print "net ", net, " prefix too long";
            return false;
        }
    if proto = "BGP_Prefix" then return true;
    return true;
}



function bgp_export_all() {
    if bgp_export() then return true;

    if source != RTS_BGP then return false;
    return true;
}

template bgp tpl_bgp {
    graceful restart on;
    local as LOCAL_ASN;
    ipv6 {
        next hop self;
        import filter peer_in_v6;
        export filter{
            if net ~ OUR_PREFIXES then bgp_path.prepend(141011);
            if bgp_export() then accept;
            else reject;
        };
    };
}

template bgp tpl_ibgp {
    local as LOCAL_ASN;
    rr client;
    direct;
   ipv6 {
       next hop self;
        import filter {
            if is_bogon_asn() then {
                print "is bogon asn", net, " for ASN ", bgp_path.last;
                reject;
        }
            if is_bogon_prefix() then {
                print "is bogon prefix", net, " for ASN ", bgp_path.last;
                reject;
            }
            accept;
        };
        export filter {
            if bgp_export_all() then accept;
            if proto != RTS_BGP then reject;
            if is_bogon_asn() then reject;
            if is_bogon_prefix() then reject;    
            accept;
        };
    };
}

template bgp tpl_bgp_rs {
    graceful restart on;
    rs client;
    local as LOCAL_ASN;
    ipv6 {
        next hop self;
        import filter peer_in_v6;
        export filter{
            if bgp_export_all() then accept;
            if net ~ OUR_PREFIXES then bgp_path.prepend(141011);
        };
    };
}

protocol bgp home_1_arch from tpl_ibgp {  
    description "HOME_1_AECH BGP";
    source address 2602:feda:ab3::1;
    neighbor 2602:feda:ab3:0:3843:aeff:fe5b:18cb%'eth0' as 141011;
}
@zhihuiyuze
Copy link
Author

I try following Preventing IPv6 rules to disable mwan3 ipv6 support,but it did't works.

@feckert
Copy link
Member

feckert commented Jun 29, 2021

I try following Preventing IPv6 rules to disable mwan3 ipv6 support,but it did't works.

This is only valid for the mwan3 which is in the master or in the openwrt-21.02.

For the mwan3 in openwrt-19.07 you can change this line so that no IPV6 rules are created.

I can't tell you more, because I don't know bird2.

@feckert feckert self-assigned this Jun 29, 2021
@zhihuiyuze
Copy link
Author

zhihuiyuze commented Jul 14, 2021 •
edited

Hi,I have been upgrade to OpenWrt 21.02-SNAPSHOT r16235-8921e36ed8
After updating to this version, mwan3 cannot create routes normally.
I created a Wireguard rule so that all IPs from mainland China pass Wireguard. The configuration is the same as that of openwrt19, but it doesn’t work properly after 20.02.

Wed Jul 14 20:54:25 2021 user.info mwan3track[15717]: Check (ping) failed for target "1.1.1.1" on interface hk_1_vpn (hk_1_vpn). Current score: 6
Wed Jul 14 20:54:25 2021 user.notice mwan3track[15717]: Interface hk_1_vpn (hk_1_vpn) is disconnecting
Wed Jul 14 20:54:30 2021 user.notice mwan3track[15717]: Interface hk_1_vpn (hk_1_vpn) is connecting
Wed Jul 14 20:54:31 2021 user.info mwan3track[15717]: Lost 1 ping(s) on interface hk_1_vpn (hk_1_vpn). Current score: 5

But i can ping 1.1.1.1. on hk_1_vpn

PING 1.1.1.1 (1.1.1.1): 56 data bytes
64 bytes from 1.1.1.1: seq=0 ttl=61 time=53.439 ms
64 bytes from 1.1.1.1: seq=1 ttl=61 time=51.358 ms
64 bytes from 1.1.1.1: seq=2 ttl=61 time=48.282 ms
64 bytes from 1.1.1.1: seq=3 ttl=61 time=48.070 ms

Interface status:
 interface wan is online 00h:30m:17s, uptime 00h:31m:29s and tracking is active
 interface wan2 is offline and tracking is paused
 interface wan3 is offline and tracking is paused
 interface Unmetered_VPN is error (16) and tracking is active
 interface Unmetered_VPN_2 is error (24) and tracking is active
 interface Unmetered_VPN_3 is error (24) and tracking is active
 interface hk_1_vpn is error (16) and tracking is active

It’s not online here, but it’s online again when shown here

image

root@home-1-OpenWrt:/etc/rc.d# ip -4 route
default via 183.156.52.1 dev pppoe-wan proto static metric 120 
66.220.18.42 via 183.156.52.1 dev pppoe-wan proto static metric 120 
183.156.52.1 dev pppoe-wan proto kernel scope link src 183.156.52.190 
192.168.7.0/24 dev eth0 proto kernel scope link src 192.168.7.1 


Software-Version
-------------------------------------------------
OpenWrt - OpenWrt 21.02-SNAPSHOT r16235-8921e36ed8
LuCI - git-21.191.52202-07063ae


Output of "ip a show"
-------------------------------------------------
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 42:e0:77:ff:96:7c brd ff:ff:ff:ff:ff:ff
    inet 192.168.7.1/24 brd 192.168.7.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 240e:390:80c:3750::1/64 scope global dynamic noprefixroute 
       valid_lft 257545sec preferred_lft 171145sec
    inet6 2001:470:f050::1/64 scope global noprefixroute 
       valid_lft forever preferred_lft forever
    inet6 2602:feda:ab3::1/64 scope global noprefixroute 
       valid_lft forever preferred_lft forever
    inet6 fe80::40e0:77ff:feff:967c/64 scope link 
       valid_lft forever preferred_lft forever
3: ip6tnl0@NONE: <NOARP> mtu 1452 qdisc noop state DOWN group default qlen 1000
    link/tunnel6 :: brd ::
4: sit0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000
    link/sit 0.0.0.0 brd 0.0.0.0
5: gre0@NONE: <NOARP> mtu 1476 qdisc noop state DOWN group default qlen 1000
    link/gre 0.0.0.0 brd 0.0.0.0
6: gretap0@NONE: <BROADCAST,MULTICAST> mtu 1476 qdisc noop state DOWN group default qlen 1000
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
7: erspan0@NONE: <BROADCAST,MULTICAST> mtu 1464 qdisc noop state DOWN group default qlen 1000
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
8: ip6gre0@NONE: <NOARP> mtu 1448 qdisc noop state DOWN group default qlen 1000
    link/gre6 :: brd ::
9: siit0: <BROADCAST,MULTICAST,NOARP> mtu 1500 qdisc noop state DOWN group default qlen 10
    link/ether 0a:95:f9:e3:42:55 brd ff:ff:ff:ff:ff:ff
10: Unmetered_VPN: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1280 qdisc noqueue state UNKNOWN group default qlen 1000
    link/none 
    inet 10.200.200.2/32 brd 255.255.255.255 scope global Unmetered_VPN
       valid_lft forever preferred_lft forever
12: hk_1_vpn: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1280 qdisc noqueue state UNKNOWN group default qlen 1000
    link/none 
    inet 10.200.210.2/32 brd 255.255.255.255 scope global hk_1_vpn
       valid_lft forever preferred_lft forever
13: Unmetered_VPN_3: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1420 qdisc noqueue state UNKNOWN group default qlen 1000
    link/none 
    inet 10.200.200.4/32 brd 255.255.255.255 scope global Unmetered_VPN_3
       valid_lft forever preferred_lft forever
14: pppoe-wan: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1492 qdisc fq_codel state UNKNOWN group default qlen 3
    link/ppp 
    inet 183.156.52.190 peer 183.156.52.1/32 scope global pppoe-wan
       valid_lft forever preferred_lft forever
    inet6 240e:390:8c0:ed6e:8846:5369:1ac9:5099/64 scope global dynamic noprefixroute 
       valid_lft 259040sec preferred_lft 172640sec
    inet6 fe80::8846:5369:1ac9:5099/128 scope link 
       valid_lft forever preferred_lft forever
15: 6in4-wan6@NONE: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1480 qdisc noqueue state UNKNOWN group default qlen 1000
    link/sit 183.156.52.190 peer 66.220.18.42
    inet6 2001:470:c:e50::2/64 scope global 
       valid_lft forever preferred_lft forever
    inet6 fe80::b79c:34be/64 scope link 
       valid_lft forever preferred_lft forever
16: Unmetered_VPN_2: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1420 qdisc noqueue state UNKNOWN group default qlen 1000
    link/none 
    inet 10.200.200.3/32 brd 255.255.255.255 scope global Unmetered_VPN_2
       valid_lft forever preferred_lft forever


Output of "ip route show"
-------------------------------------------------
default via 183.156.52.1 dev pppoe-wan proto static metric 120 
66.220.18.42 via 183.156.52.1 dev pppoe-wan proto static metric 120 
183.156.52.1 dev pppoe-wan proto kernel scope link src 183.156.52.190 
192.168.7.0/24 dev eth0 proto kernel scope link src 192.168.7.1


Output of "ip rule show"
-------------------------------------------------
0:	from all lookup local
1001:	from all iif pppoe-wan lookup 1
1004:	from all iif Unmetered_VPN lookup 4
1005:	from all iif Unmetered_VPN_2 lookup 5
1006:	from all iif Unmetered_VPN_3 lookup 6
1007:	from all iif hk_1_vpn lookup 7
2001:	from all fwmark 0x100/0x3f00 lookup 1
2004:	from all fwmark 0x400/0x3f00 lookup 4
2005:	from all fwmark 0x500/0x3f00 lookup 5
2006:	from all fwmark 0x600/0x3f00 lookup 6
2007:	from all fwmark 0x700/0x3f00 lookup 7
2061:	from all fwmark 0x3d00/0x3f00 blackhole
2062:	from all fwmark 0x3e00/0x3f00 unreachable
3001:	from all fwmark 0x100/0x3f00 unreachable
3004:	from all fwmark 0x400/0x3f00 unreachable
3005:	from all fwmark 0x500/0x3f00 unreachable
3006:	from all fwmark 0x600/0x3f00 unreachable
3007:	from all fwmark 0x700/0x3f00 unreachable
32766:	from all lookup main
32767:	from all lookup default


Output of "ip route list table 1-250"
-------------------------------------------------
Table 1: default via 183.156.52.1 dev pppoe-wan proto static metric 120 
66.220.18.42 via 183.156.52.1 dev pppoe-wan proto static metric 120 
183.156.52.1 dev pppoe-wan proto kernel scope link src 183.156.52.190 
192.168.7.0/24 dev eth0 proto kernel scope link src 192.168.7.1
Table 4: 192.168.7.0/24 dev eth0 proto kernel scope link src 192.168.7.1
Table 7: 192.168.7.0/24 dev eth0 proto kernel scope link src 192.168.7.1


Output of "iptables -L -t mangle -v -n"
-------------------------------------------------
Chain PREROUTING (policy ACCEPT 23735 packets, 2596K bytes)
 pkts bytes target     prot opt in     out     source               destination         
 356K   61M mwan3_hook  all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain INPUT (policy ACCEPT 14590 packets, 1211K bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain FORWARD (policy ACCEPT 9026 packets, 1378K bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 TCPMSS     tcp  --  *      Unmetered_VPN  0.0.0.0/0            0.0.0.0/0            tcp flags:0x06/0x02 /* !fw3: Zone wan MTU fixing */ TCPMSS clamp to PMTU
    0     0 TCPMSS     tcp  --  Unmetered_VPN *       0.0.0.0/0            0.0.0.0/0            tcp flags:0x06/0x02 /* !fw3: Zone wan MTU fixing */ TCPMSS clamp to PMTU
    0     0 TCPMSS     tcp  --  *      Unmetered_VPN_2  0.0.0.0/0            0.0.0.0/0            tcp flags:0x06/0x02 /* !fw3: Zone wan MTU fixing */ TCPMSS clamp to PMTU
    0     0 TCPMSS     tcp  --  Unmetered_VPN_2 *       0.0.0.0/0            0.0.0.0/0            tcp flags:0x06/0x02 /* !fw3: Zone wan MTU fixing */ TCPMSS clamp to PMTU
    0     0 TCPMSS     tcp  --  *      Unmetered_VPN_3  0.0.0.0/0            0.0.0.0/0            tcp flags:0x06/0x02 /* !fw3: Zone wan MTU fixing */ TCPMSS clamp to PMTU
    0     0 TCPMSS     tcp  --  Unmetered_VPN_3 *       0.0.0.0/0            0.0.0.0/0            tcp flags:0x06/0x02 /* !fw3: Zone wan MTU fixing */ TCPMSS clamp to PMTU
    0     0 TCPMSS     tcp  --  *      6in4-wan6  0.0.0.0/0            0.0.0.0/0            tcp flags:0x06/0x02 /* !fw3: Zone wan MTU fixing */ TCPMSS clamp to PMTU
    0     0 TCPMSS     tcp  --  6in4-wan6 *       0.0.0.0/0            0.0.0.0/0            tcp flags:0x06/0x02 /* !fw3: Zone wan MTU fixing */ TCPMSS clamp to PMTU
    0     0 TCPMSS     tcp  --  *      hk_1_vpn  0.0.0.0/0            0.0.0.0/0            tcp flags:0x06/0x02 /* !fw3: Zone wan MTU fixing */ TCPMSS clamp to PMTU
    3   144 TCPMSS     tcp  --  hk_1_vpn *       0.0.0.0/0            0.0.0.0/0            tcp flags:0x06/0x02 /* !fw3: Zone wan MTU fixing */ TCPMSS clamp to PMTU
32082 1924K TCPMSS     tcp  --  *      pppoe-wan  0.0.0.0/0            0.0.0.0/0            tcp flags:0x06/0x02 /* !fw3: Zone wan MTU fixing */ TCPMSS clamp to PMTU
 1002 58000 TCPMSS     tcp  --  pppoe-wan *       0.0.0.0/0            0.0.0.0/0            tcp flags:0x06/0x02 /* !fw3: Zone wan MTU fixing */ TCPMSS clamp to PMTU

Chain OUTPUT (policy ACCEPT 6537 packets, 982K bytes)
 pkts bytes target     prot opt in     out     source               destination         
85342   12M mwan3_hook  all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain POSTROUTING (policy ACCEPT 11308 packets, 2187K bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain mwan3_connected (2 references)
 pkts bytes target     prot opt in     out     source               destination         
   24  7707 MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            match-set mwan3_connected dst MARK or 0x3f00

Chain mwan3_hook (2 references)
 pkts bytes target     prot opt in     out     source               destination         
 436K   72M CONNMARK   all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00 CONNMARK restore mask 0x3f00
 305K   16M mwan3_ifaces_in  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00
 102K 6435K mwan3_connected  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00
 102K 6435K mwan3_rules  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00
 442K   73M CONNMARK   all  --  *      *       0.0.0.0/0            0.0.0.0/0            CONNMARK save mask 0x3f00
 402K   67M mwan3_connected  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match ! 0x3f00/0x3f00

Chain mwan3_iface_in_Unmetered_VPN (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MARK       all  --  Unmetered_VPN *       0.0.0.0/0            0.0.0.0/0            match-set mwan3_connected src mark match 0x0/0x3f00 /* default */ MARK or 0x3f00
    0     0 MARK       all  --  Unmetered_VPN *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00 /* Unmetered_VPN */ MARK xset 0x400/0x3f00

Chain mwan3_iface_in_hk_1_vpn (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MARK       all  --  hk_1_vpn *       0.0.0.0/0            0.0.0.0/0            match-set mwan3_connected src mark match 0x0/0x3f00 /* default */ MARK or 0x3f00
  607 61227 MARK       all  --  hk_1_vpn *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00 /* hk_1_vpn */ MARK xset 0x700/0x3f00

Chain mwan3_iface_in_wan (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MARK       all  --  pppoe-wan *       0.0.0.0/0            0.0.0.0/0            match-set mwan3_connected src mark match 0x0/0x3f00 /* default */ MARK or 0x3f00
 203K 9376K MARK       all  --  pppoe-wan *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00 /* wan */ MARK xset 0x100/0x3f00

Chain mwan3_ifaces_in (1 references)
 pkts bytes target     prot opt in     out     source               destination         
 305K   16M mwan3_iface_in_wan  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00
 102K 6496K mwan3_iface_in_Unmetered_VPN  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00
 102K 6496K mwan3_iface_in_hk_1_vpn  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00

Chain mwan3_policy_Unmetered (0 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00 statistic mode random probability 0.52100000018 /* Unmetered_VPN_2 60 115 */ MARK xset 0x500/0x3f00
    0     0 MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00 /* Unmetered_VPN 55 55 */ MARK xset 0x400/0x3f00

Chain mwan3_policy_VPN (0 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00 /* hk_1_vpn 50 50 */ MARK xset 0x700/0x3f00

Chain mwan3_policy_balanced (0 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00 /* wan 50 50 */ MARK xset 0x100/0x3f00

Chain mwan3_policy_hk1 (0 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00 /* hk_1_vpn 50 50 */ MARK xset 0x700/0x3f00

Chain mwan3_policy_wan2_only (0 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00 /* default */ MARK or 0x3f00

Chain mwan3_policy_wan3_only (0 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00 /* default */ MARK or 0x3f00

Chain mwan3_policy_wan_only (0 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00 /* wan 50 50 */ MARK xset 0x100/0x3f00

Chain mwan3_rules (1 references)
 pkts bytes target     prot opt in     out     source               destination

root@home-1-OpenWrt:/etc/config# cat mwan3

config rule 'vpn_ca_udp'
        option dest_ip '104.218.61.232'
        option dest_port '52471'
        option proto 'udp'
        option sticky '0'
        option use_policy 'balanced'
        option family 'ipv4'

config rule 'vpn_ca'
        option dest_ip '104.218.61.232'
        option dest_port '52471'
        option proto 'tcp'
        option sticky '0'
        option use_policy 'balanced'
        option family 'ipv4'

config rule 'vpn_hk'
        option dest_ip '103.152.35.35'
        option sticky '0'
        option proto 'all'
        option family 'ipv4'
        option use_policy 'balanced'

config rule 'heipv6_update'
        option dest_ip '64.62.200.2'
        option proto 'all'
        option sticky '0'
        option use_policy 'wan_only'
        option family 'ipv4'

config rule 'direct'
        option proto 'all'
        option sticky '0'
        option ipset 'direct'
        option family 'ipv4'
        option use_policy 'balanced'

config rule 'China_v4'
        option proto 'all'
        option sticky '0'
        option ipset 'china'
        option family 'ipv4'
        option use_policy 'balanced'

config rule 'https_china'
        option dest_port '443'
        option ipset 'china'
        option sticky '1'
        option proto 'tcp'
        option family 'ipv4'
        option dest_ip '0.0.0.0/0'
        option use_policy 'balanced'

config rule 'ssh'
        option dest_port '22'
        option proto 'tcp'
        option sticky '1'
        option family 'ipv4'
        option dest_ip '0.0.0.0/0'
        option use_policy 'balanced'

config rule 'https'
        option dest_port '443'
        option proto 'tcp'
        option sticky '1'
        option use_policy 'VPN'
        option family 'ipv4'
        option dest_ip '0.0.0.0/0'

config rule 'ipv4_tcp'
        option sticky '0'
        option proto 'tcp'
        option dest_port '1024:65535'
        option use_policy 'VPN'
        option family 'ipv4'
        option dest_ip '0.0.0.0/0'

config rule 'default_rule_v4'
        option dest_ip '0.0.0.0/0'
        option family 'ipv4'
        option proto 'all'
        option sticky '0'
        option use_policy 'VPN'

config globals 'globals'
        option mmx_mask '0x3F00'
        option rtmon_interval '5'

config interface 'wan'
        option enabled '1'
        option family 'ipv4'
        option count '1'
        option timeout '2'
        option interval '5'
        option down '3'
        option up '8'
        option initial_state 'online'
        option track_method 'ping'
        option size '56'
        option max_ttl '60'
        option check_quality '0'
        option failure_interval '5'
        option recovery_interval '5'
        option reliability '1'
        list track_ip '223.5.5.5'
        list track_ip '1.2.4.8'
        list track_ip '1.1.1.1'

config policy 'wan_only'
        option last_resort 'default'
        list use_member 'wan1_m50_w50'

config policy 'balanced'
        option last_resort 'default'
        list use_member 'wan3_m50_w50'
        list use_member 'wan1_m50_w50'
        list use_member 'wan2_m50_w50'

config interface 'wan2'
        option initial_state 'online'
        option family 'ipv4'
        option track_method 'ping'
        option reliability '1'
        option count '1'
        option size '56'
        option max_ttl '60'
        option check_quality '0'
        option timeout '2'
        option interval '5'
        option failure_interval '5'
        option recovery_interval '5'
        option down '3'
        option up '3'
        option enabled '1'
        list track_ip '223.5.5.5'
        list track_ip '1.2.4.8'
        list track_ip '1.1.1.1'

config interface 'wan3'
        option enabled '1'
        option initial_state 'online'
        option family 'ipv4'
        option track_method 'ping'
        option reliability '1'
        option count '1'
        option size '56'
        option max_ttl '60'
        option check_quality '0'
        option timeout '2'
        option interval '5'
        option failure_interval '5'
        option recovery_interval '5'
        option down '3'
        option up '3'
        list track_ip '223.5.5.5'
        list track_ip '1.2.4.8'
        list track_ip '1.1.1.1'

config policy 'wan2_only'
        option last_resort 'default'
        list use_member 'wan2_m50_w50'

config policy 'wan3_only'
        option last_resort 'default'
        list use_member 'wan3_m50_w50'

config interface 'Unmetered_VPN'
        option enabled '1'
        option initial_state 'online'
        option family 'ipv4'
        option track_method 'ping'
        option reliability '1'
        option count '1'
        option size '56'
        option max_ttl '60'
        option check_quality '0'
        option timeout '2'
        option interval '5'
        option failure_interval '5'
        option recovery_interval '5'
        option down '3'
        option up '3'
        list track_ip '10.200.200.1'
        list track_ip '1.1.1.1'

config policy 'VPN'
        list use_member 'hk_1_vpn_m50_w50'
        list use_member 'Unmetered_VPN_m60_w55'
        list use_member 'Unmetered_VPN_2_m60_w60'
        list use_member 'Unmetered_VPN_3_m60_w60'
        option last_resort 'default'

config interface 'Unmetered_VPN_2'
        option initial_state 'online'
        option family 'ipv4'
        option track_method 'ping'
        option reliability '1'
        option count '1'
        option size '56'
        option max_ttl '60'
        option check_quality '0'
        option timeout '2'
        option interval '5'
        option failure_interval '5'
        option recovery_interval '5'
        option down '3'
        option up '3'
        option enabled '1'
        list track_ip '10.200.200.1'
        list track_ip '1.1.1.1'

config interface 'Unmetered_VPN_3'
        option enabled '1'
        option initial_state 'online'
        option family 'ipv4'
        option track_method 'ping'
        option reliability '1'
        option count '1'
        option size '56'
        option max_ttl '60'
        option check_quality '0'
        option timeout '2'
        option interval '5'
        option failure_interval '5'
        option recovery_interval '5'
        option down '3'
        option up '3'
        list track_ip '1.1.1.1'
        list track_ip '8.8.8.8'

config interface 'hk_1_vpn'
        option enabled '1'
        option initial_state 'online'
        option family 'ipv4'
        option track_method 'ping'
        option reliability '1'
        option count '1'
        option size '56'
        option max_ttl '60'
        option check_quality '0'
        option timeout '2'
        option interval '5'
        option failure_interval '5'
        option recovery_interval '5'
        option down '3'
        option up '3'
        list track_ip '1.1.1.1'

config policy 'Unmetered'
        option last_resort 'unreachable'
        list use_member 'Unmetered_VPN_m60_w55'
        list use_member 'Unmetered_VPN_2_m60_w60'
        list use_member 'Unmetered_VPN_3_m60_w60'

config policy 'hk1'
        option last_resort 'unreachable'
        list use_member 'hk_1_vpn_m50_w50'

config member 'Unmetered_VPN_m60_w55'
        option interface 'Unmetered_VPN'
        option metric '60'
        option weight '55'

config member 'Unmetered_VPN_2_m60_w60'
        option interface 'Unmetered_VPN_2'
        option metric '60'
        option weight '60'

config member 'Unmetered_VPN_3_m60_w60'
        option interface 'Unmetered_VPN_3'
        option metric '60'
        option weight '60'

config member 'hk_1_vpn_m50_w50'
        option interface 'hk_1_vpn'
        option metric '50'
        option weight '50'

config member 'hk_1_vpn_m60_w50'
        option interface 'hk_1_vpn'
        option metric '60'
        option weight '55'

config member 'wan1_m50_w50'
        option interface 'wan'
        option metric '50'
        option weight '50'

config member 'wan2_m50_w50'
        option metric '50'
        option weight '50'
        option interface 'wan2'

config member 'wan3_m50_w50'
        option interface 'wan3'
        option metric '50'
        option weight '50'

@zhihuiyuze
Copy link
Author

Hi,I have been upgrade to OpenWrt 21.02-SNAPSHOT r16235-8921e36ed8
After updating to this version, mwan3 cannot create routes normally.
I created a Wireguard rule so that all IPs from mainland China pass Wireguard. The configuration is the same as that of openwrt19, but it doesn’t work properly after 20.02.

Wed Jul 14 20:54:25 2021 user.info mwan3track[15717]: Check (ping) failed for target "1.1.1.1" on interface hk_1_vpn (hk_1_vpn). Current score: 6
Wed Jul 14 20:54:25 2021 user.notice mwan3track[15717]: Interface hk_1_vpn (hk_1_vpn) is disconnecting
Wed Jul 14 20:54:30 2021 user.notice mwan3track[15717]: Interface hk_1_vpn (hk_1_vpn) is connecting
Wed Jul 14 20:54:31 2021 user.info mwan3track[15717]: Lost 1 ping(s) on interface hk_1_vpn (hk_1_vpn). Current score: 5

But i can ping 1.1.1.1. on hk_1_vpn

PING 1.1.1.1 (1.1.1.1): 56 data bytes
64 bytes from 1.1.1.1: seq=0 ttl=61 time=53.439 ms
64 bytes from 1.1.1.1: seq=1 ttl=61 time=51.358 ms
64 bytes from 1.1.1.1: seq=2 ttl=61 time=48.282 ms
64 bytes from 1.1.1.1: seq=3 ttl=61 time=48.070 ms

Interface status:
 interface wan is online 00h:30m:17s, uptime 00h:31m:29s and tracking is active
 interface wan2 is offline and tracking is paused
 interface wan3 is offline and tracking is paused
 interface Unmetered_VPN is error (16) and tracking is active
 interface Unmetered_VPN_2 is error (24) and tracking is active
 interface Unmetered_VPN_3 is error (24) and tracking is active
 interface hk_1_vpn is error (16) and tracking is active

It’s not online here, but it’s online again when shown here

image

root@home-1-OpenWrt:/etc/rc.d# ip -4 route
default via 183.156.52.1 dev pppoe-wan proto static metric 120 
66.220.18.42 via 183.156.52.1 dev pppoe-wan proto static metric 120 
183.156.52.1 dev pppoe-wan proto kernel scope link src 183.156.52.190 
192.168.7.0/24 dev eth0 proto kernel scope link src 192.168.7.1 

Software-Version
-------------------------------------------------
OpenWrt - OpenWrt 21.02-SNAPSHOT r16235-8921e36ed8
LuCI - git-21.191.52202-07063ae


Output of "ip a show"
-------------------------------------------------
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 42:e0:77:ff:96:7c brd ff:ff:ff:ff:ff:ff
    inet 192.168.7.1/24 brd 192.168.7.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 240e:390:80c:3750::1/64 scope global dynamic noprefixroute 
       valid_lft 257545sec preferred_lft 171145sec
    inet6 2001:470:f050::1/64 scope global noprefixroute 
       valid_lft forever preferred_lft forever
    inet6 2602:feda:ab3::1/64 scope global noprefixroute 
       valid_lft forever preferred_lft forever
    inet6 fe80::40e0:77ff:feff:967c/64 scope link 
       valid_lft forever preferred_lft forever
3: ip6tnl0@NONE: <NOARP> mtu 1452 qdisc noop state DOWN group default qlen 1000
    link/tunnel6 :: brd ::
4: sit0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000
    link/sit 0.0.0.0 brd 0.0.0.0
5: gre0@NONE: <NOARP> mtu 1476 qdisc noop state DOWN group default qlen 1000
    link/gre 0.0.0.0 brd 0.0.0.0
6: gretap0@NONE: <BROADCAST,MULTICAST> mtu 1476 qdisc noop state DOWN group default qlen 1000
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
7: erspan0@NONE: <BROADCAST,MULTICAST> mtu 1464 qdisc noop state DOWN group default qlen 1000
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
8: ip6gre0@NONE: <NOARP> mtu 1448 qdisc noop state DOWN group default qlen 1000
    link/gre6 :: brd ::
9: siit0: <BROADCAST,MULTICAST,NOARP> mtu 1500 qdisc noop state DOWN group default qlen 10
    link/ether 0a:95:f9:e3:42:55 brd ff:ff:ff:ff:ff:ff
10: Unmetered_VPN: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1280 qdisc noqueue state UNKNOWN group default qlen 1000
    link/none 
    inet 10.200.200.2/32 brd 255.255.255.255 scope global Unmetered_VPN
       valid_lft forever preferred_lft forever
12: hk_1_vpn: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1280 qdisc noqueue state UNKNOWN group default qlen 1000
    link/none 
    inet 10.200.210.2/32 brd 255.255.255.255 scope global hk_1_vpn
       valid_lft forever preferred_lft forever
13: Unmetered_VPN_3: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1420 qdisc noqueue state UNKNOWN group default qlen 1000
    link/none 
    inet 10.200.200.4/32 brd 255.255.255.255 scope global Unmetered_VPN_3
       valid_lft forever preferred_lft forever
14: pppoe-wan: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1492 qdisc fq_codel state UNKNOWN group default qlen 3
    link/ppp 
    inet 183.156.52.190 peer 183.156.52.1/32 scope global pppoe-wan
       valid_lft forever preferred_lft forever
    inet6 240e:390:8c0:ed6e:8846:5369:1ac9:5099/64 scope global dynamic noprefixroute 
       valid_lft 259040sec preferred_lft 172640sec
    inet6 fe80::8846:5369:1ac9:5099/128 scope link 
       valid_lft forever preferred_lft forever
15: 6in4-wan6@NONE: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1480 qdisc noqueue state UNKNOWN group default qlen 1000
    link/sit 183.156.52.190 peer 66.220.18.42
    inet6 2001:470:c:e50::2/64 scope global 
       valid_lft forever preferred_lft forever
    inet6 fe80::b79c:34be/64 scope link 
       valid_lft forever preferred_lft forever
16: Unmetered_VPN_2: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1420 qdisc noqueue state UNKNOWN group default qlen 1000
    link/none 
    inet 10.200.200.3/32 brd 255.255.255.255 scope global Unmetered_VPN_2
       valid_lft forever preferred_lft forever


Output of "ip route show"
-------------------------------------------------
default via 183.156.52.1 dev pppoe-wan proto static metric 120 
66.220.18.42 via 183.156.52.1 dev pppoe-wan proto static metric 120 
183.156.52.1 dev pppoe-wan proto kernel scope link src 183.156.52.190 
192.168.7.0/24 dev eth0 proto kernel scope link src 192.168.7.1


Output of "ip rule show"
-------------------------------------------------
0:	from all lookup local
1001:	from all iif pppoe-wan lookup 1
1004:	from all iif Unmetered_VPN lookup 4
1005:	from all iif Unmetered_VPN_2 lookup 5
1006:	from all iif Unmetered_VPN_3 lookup 6
1007:	from all iif hk_1_vpn lookup 7
2001:	from all fwmark 0x100/0x3f00 lookup 1
2004:	from all fwmark 0x400/0x3f00 lookup 4
2005:	from all fwmark 0x500/0x3f00 lookup 5
2006:	from all fwmark 0x600/0x3f00 lookup 6
2007:	from all fwmark 0x700/0x3f00 lookup 7
2061:	from all fwmark 0x3d00/0x3f00 blackhole
2062:	from all fwmark 0x3e00/0x3f00 unreachable
3001:	from all fwmark 0x100/0x3f00 unreachable
3004:	from all fwmark 0x400/0x3f00 unreachable
3005:	from all fwmark 0x500/0x3f00 unreachable
3006:	from all fwmark 0x600/0x3f00 unreachable
3007:	from all fwmark 0x700/0x3f00 unreachable
32766:	from all lookup main
32767:	from all lookup default


Output of "ip route list table 1-250"
-------------------------------------------------
Table 1: default via 183.156.52.1 dev pppoe-wan proto static metric 120 
66.220.18.42 via 183.156.52.1 dev pppoe-wan proto static metric 120 
183.156.52.1 dev pppoe-wan proto kernel scope link src 183.156.52.190 
192.168.7.0/24 dev eth0 proto kernel scope link src 192.168.7.1
Table 4: 192.168.7.0/24 dev eth0 proto kernel scope link src 192.168.7.1
Table 7: 192.168.7.0/24 dev eth0 proto kernel scope link src 192.168.7.1


Output of "iptables -L -t mangle -v -n"
-------------------------------------------------
Chain PREROUTING (policy ACCEPT 23735 packets, 2596K bytes)
 pkts bytes target     prot opt in     out     source               destination         
 356K   61M mwan3_hook  all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain INPUT (policy ACCEPT 14590 packets, 1211K bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain FORWARD (policy ACCEPT 9026 packets, 1378K bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 TCPMSS     tcp  --  *      Unmetered_VPN  0.0.0.0/0            0.0.0.0/0            tcp flags:0x06/0x02 /* !fw3: Zone wan MTU fixing */ TCPMSS clamp to PMTU
    0     0 TCPMSS     tcp  --  Unmetered_VPN *       0.0.0.0/0            0.0.0.0/0            tcp flags:0x06/0x02 /* !fw3: Zone wan MTU fixing */ TCPMSS clamp to PMTU
    0     0 TCPMSS     tcp  --  *      Unmetered_VPN_2  0.0.0.0/0            0.0.0.0/0            tcp flags:0x06/0x02 /* !fw3: Zone wan MTU fixing */ TCPMSS clamp to PMTU
    0     0 TCPMSS     tcp  --  Unmetered_VPN_2 *       0.0.0.0/0            0.0.0.0/0            tcp flags:0x06/0x02 /* !fw3: Zone wan MTU fixing */ TCPMSS clamp to PMTU
    0     0 TCPMSS     tcp  --  *      Unmetered_VPN_3  0.0.0.0/0            0.0.0.0/0            tcp flags:0x06/0x02 /* !fw3: Zone wan MTU fixing */ TCPMSS clamp to PMTU
    0     0 TCPMSS     tcp  --  Unmetered_VPN_3 *       0.0.0.0/0            0.0.0.0/0            tcp flags:0x06/0x02 /* !fw3: Zone wan MTU fixing */ TCPMSS clamp to PMTU
    0     0 TCPMSS     tcp  --  *      6in4-wan6  0.0.0.0/0            0.0.0.0/0            tcp flags:0x06/0x02 /* !fw3: Zone wan MTU fixing */ TCPMSS clamp to PMTU
    0     0 TCPMSS     tcp  --  6in4-wan6 *       0.0.0.0/0            0.0.0.0/0            tcp flags:0x06/0x02 /* !fw3: Zone wan MTU fixing */ TCPMSS clamp to PMTU
    0     0 TCPMSS     tcp  --  *      hk_1_vpn  0.0.0.0/0            0.0.0.0/0            tcp flags:0x06/0x02 /* !fw3: Zone wan MTU fixing */ TCPMSS clamp to PMTU
    3   144 TCPMSS     tcp  --  hk_1_vpn *       0.0.0.0/0            0.0.0.0/0            tcp flags:0x06/0x02 /* !fw3: Zone wan MTU fixing */ TCPMSS clamp to PMTU
32082 1924K TCPMSS     tcp  --  *      pppoe-wan  0.0.0.0/0            0.0.0.0/0            tcp flags:0x06/0x02 /* !fw3: Zone wan MTU fixing */ TCPMSS clamp to PMTU
 1002 58000 TCPMSS     tcp  --  pppoe-wan *       0.0.0.0/0            0.0.0.0/0            tcp flags:0x06/0x02 /* !fw3: Zone wan MTU fixing */ TCPMSS clamp to PMTU

Chain OUTPUT (policy ACCEPT 6537 packets, 982K bytes)
 pkts bytes target     prot opt in     out     source               destination         
85342   12M mwan3_hook  all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain POSTROUTING (policy ACCEPT 11308 packets, 2187K bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain mwan3_connected (2 references)
 pkts bytes target     prot opt in     out     source               destination         
   24  7707 MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            match-set mwan3_connected dst MARK or 0x3f00

Chain mwan3_hook (2 references)
 pkts bytes target     prot opt in     out     source               destination         
 436K   72M CONNMARK   all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00 CONNMARK restore mask 0x3f00
 305K   16M mwan3_ifaces_in  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00
 102K 6435K mwan3_connected  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00
 102K 6435K mwan3_rules  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00
 442K   73M CONNMARK   all  --  *      *       0.0.0.0/0            0.0.0.0/0            CONNMARK save mask 0x3f00
 402K   67M mwan3_connected  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match ! 0x3f00/0x3f00

Chain mwan3_iface_in_Unmetered_VPN (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MARK       all  --  Unmetered_VPN *       0.0.0.0/0            0.0.0.0/0            match-set mwan3_connected src mark match 0x0/0x3f00 /* default */ MARK or 0x3f00
    0     0 MARK       all  --  Unmetered_VPN *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00 /* Unmetered_VPN */ MARK xset 0x400/0x3f00

Chain mwan3_iface_in_hk_1_vpn (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MARK       all  --  hk_1_vpn *       0.0.0.0/0            0.0.0.0/0            match-set mwan3_connected src mark match 0x0/0x3f00 /* default */ MARK or 0x3f00
  607 61227 MARK       all  --  hk_1_vpn *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00 /* hk_1_vpn */ MARK xset 0x700/0x3f00

Chain mwan3_iface_in_wan (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MARK       all  --  pppoe-wan *       0.0.0.0/0            0.0.0.0/0            match-set mwan3_connected src mark match 0x0/0x3f00 /* default */ MARK or 0x3f00
 203K 9376K MARK       all  --  pppoe-wan *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00 /* wan */ MARK xset 0x100/0x3f00

Chain mwan3_ifaces_in (1 references)
 pkts bytes target     prot opt in     out     source               destination         
 305K   16M mwan3_iface_in_wan  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00
 102K 6496K mwan3_iface_in_Unmetered_VPN  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00
 102K 6496K mwan3_iface_in_hk_1_vpn  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00

Chain mwan3_policy_Unmetered (0 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00 statistic mode random probability 0.52100000018 /* Unmetered_VPN_2 60 115 */ MARK xset 0x500/0x3f00
    0     0 MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00 /* Unmetered_VPN 55 55 */ MARK xset 0x400/0x3f00

Chain mwan3_policy_VPN (0 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00 /* hk_1_vpn 50 50 */ MARK xset 0x700/0x3f00

Chain mwan3_policy_balanced (0 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00 /* wan 50 50 */ MARK xset 0x100/0x3f00

Chain mwan3_policy_hk1 (0 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00 /* hk_1_vpn 50 50 */ MARK xset 0x700/0x3f00

Chain mwan3_policy_wan2_only (0 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00 /* default */ MARK or 0x3f00

Chain mwan3_policy_wan3_only (0 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00 /* default */ MARK or 0x3f00

Chain mwan3_policy_wan_only (0 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00 /* wan 50 50 */ MARK xset 0x100/0x3f00

Chain mwan3_rules (1 references)
 pkts bytes target     prot opt in     out     source               destination

root@home-1-OpenWrt:/etc/config# cat mwan3

config rule 'vpn_ca_udp'
        option dest_ip '104.218.61.232'
        option dest_port '52471'
        option proto 'udp'
        option sticky '0'
        option use_policy 'balanced'
        option family 'ipv4'

config rule 'vpn_ca'
        option dest_ip '104.218.61.232'
        option dest_port '52471'
        option proto 'tcp'
        option sticky '0'
        option use_policy 'balanced'
        option family 'ipv4'

config rule 'vpn_hk'
        option dest_ip '103.152.35.35'
        option sticky '0'
        option proto 'all'
        option family 'ipv4'
        option use_policy 'balanced'

config rule 'heipv6_update'
        option dest_ip '64.62.200.2'
        option proto 'all'
        option sticky '0'
        option use_policy 'wan_only'
        option family 'ipv4'

config rule 'direct'
        option proto 'all'
        option sticky '0'
        option ipset 'direct'
        option family 'ipv4'
        option use_policy 'balanced'

config rule 'China_v4'
        option proto 'all'
        option sticky '0'
        option ipset 'china'
        option family 'ipv4'
        option use_policy 'balanced'

config rule 'https_china'
        option dest_port '443'
        option ipset 'china'
        option sticky '1'
        option proto 'tcp'
        option family 'ipv4'
        option dest_ip '0.0.0.0/0'
        option use_policy 'balanced'

config rule 'ssh'
        option dest_port '22'
        option proto 'tcp'
        option sticky '1'
        option family 'ipv4'
        option dest_ip '0.0.0.0/0'
        option use_policy 'balanced'

config rule 'https'
        option dest_port '443'
        option proto 'tcp'
        option sticky '1'
        option use_policy 'VPN'
        option family 'ipv4'
        option dest_ip '0.0.0.0/0'

config rule 'ipv4_tcp'
        option sticky '0'
        option proto 'tcp'
        option dest_port '1024:65535'
        option use_policy 'VPN'
        option family 'ipv4'
        option dest_ip '0.0.0.0/0'

config rule 'default_rule_v4'
        option dest_ip '0.0.0.0/0'
        option family 'ipv4'
        option proto 'all'
        option sticky '0'
        option use_policy 'VPN'

config globals 'globals'
        option mmx_mask '0x3F00'
        option rtmon_interval '5'

config interface 'wan'
        option enabled '1'
        option family 'ipv4'
        option count '1'
        option timeout '2'
        option interval '5'
        option down '3'
        option up '8'
        option initial_state 'online'
        option track_method 'ping'
        option size '56'
        option max_ttl '60'
        option check_quality '0'
        option failure_interval '5'
        option recovery_interval '5'
        option reliability '1'
        list track_ip '223.5.5.5'
        list track_ip '1.2.4.8'
        list track_ip '1.1.1.1'

config policy 'wan_only'
        option last_resort 'default'
        list use_member 'wan1_m50_w50'

config policy 'balanced'
        option last_resort 'default'
        list use_member 'wan3_m50_w50'
        list use_member 'wan1_m50_w50'
        list use_member 'wan2_m50_w50'

config interface 'wan2'
        option initial_state 'online'
        option family 'ipv4'
        option track_method 'ping'
        option reliability '1'
        option count '1'
        option size '56'
        option max_ttl '60'
        option check_quality '0'
        option timeout '2'
        option interval '5'
        option failure_interval '5'
        option recovery_interval '5'
        option down '3'
        option up '3'
        option enabled '1'
        list track_ip '223.5.5.5'
        list track_ip '1.2.4.8'
        list track_ip '1.1.1.1'

config interface 'wan3'
        option enabled '1'
        option initial_state 'online'
        option family 'ipv4'
        option track_method 'ping'
        option reliability '1'
        option count '1'
        option size '56'
        option max_ttl '60'
        option check_quality '0'
        option timeout '2'
        option interval '5'
        option failure_interval '5'
        option recovery_interval '5'
        option down '3'
        option up '3'
        list track_ip '223.5.5.5'
        list track_ip '1.2.4.8'
        list track_ip '1.1.1.1'

config policy 'wan2_only'
        option last_resort 'default'
        list use_member 'wan2_m50_w50'

config policy 'wan3_only'
        option last_resort 'default'
        list use_member 'wan3_m50_w50'

config interface 'Unmetered_VPN'
        option enabled '1'
        option initial_state 'online'
        option family 'ipv4'
        option track_method 'ping'
        option reliability '1'
        option count '1'
        option size '56'
        option max_ttl '60'
        option check_quality '0'
        option timeout '2'
        option interval '5'
        option failure_interval '5'
        option recovery_interval '5'
        option down '3'
        option up '3'
        list track_ip '10.200.200.1'
        list track_ip '1.1.1.1'

config policy 'VPN'
        list use_member 'hk_1_vpn_m50_w50'
        list use_member 'Unmetered_VPN_m60_w55'
        list use_member 'Unmetered_VPN_2_m60_w60'
        list use_member 'Unmetered_VPN_3_m60_w60'
        option last_resort 'default'

config interface 'Unmetered_VPN_2'
        option initial_state 'online'
        option family 'ipv4'
        option track_method 'ping'
        option reliability '1'
        option count '1'
        option size '56'
        option max_ttl '60'
        option check_quality '0'
        option timeout '2'
        option interval '5'
        option failure_interval '5'
        option recovery_interval '5'
        option down '3'
        option up '3'
        option enabled '1'
        list track_ip '10.200.200.1'
        list track_ip '1.1.1.1'

config interface 'Unmetered_VPN_3'
        option enabled '1'
        option initial_state 'online'
        option family 'ipv4'
        option track_method 'ping'
        option reliability '1'
        option count '1'
        option size '56'
        option max_ttl '60'
        option check_quality '0'
        option timeout '2'
        option interval '5'
        option failure_interval '5'
        option recovery_interval '5'
        option down '3'
        option up '3'
        list track_ip '1.1.1.1'
        list track_ip '8.8.8.8'

config interface 'hk_1_vpn'
        option enabled '1'
        option initial_state 'online'
        option family 'ipv4'
        option track_method 'ping'
        option reliability '1'
        option count '1'
        option size '56'
        option max_ttl '60'
        option check_quality '0'
        option timeout '2'
        option interval '5'
        option failure_interval '5'
        option recovery_interval '5'
        option down '3'
        option up '3'
        list track_ip '1.1.1.1'

config policy 'Unmetered'
        option last_resort 'unreachable'
        list use_member 'Unmetered_VPN_m60_w55'
        list use_member 'Unmetered_VPN_2_m60_w60'
        list use_member 'Unmetered_VPN_3_m60_w60'

config policy 'hk1'
        option last_resort 'unreachable'
        list use_member 'hk_1_vpn_m50_w50'

config member 'Unmetered_VPN_m60_w55'
        option interface 'Unmetered_VPN'
        option metric '60'
        option weight '55'

config member 'Unmetered_VPN_2_m60_w60'
        option interface 'Unmetered_VPN_2'
        option metric '60'
        option weight '60'

config member 'Unmetered_VPN_3_m60_w60'
        option interface 'Unmetered_VPN_3'
        option metric '60'
        option weight '60'

config member 'hk_1_vpn_m50_w50'
        option interface 'hk_1_vpn'
        option metric '50'
        option weight '50'

config member 'hk_1_vpn_m60_w50'
        option interface 'hk_1_vpn'
        option metric '60'
        option weight '55'

config member 'wan1_m50_w50'
        option interface 'wan'
        option metric '50'
        option weight '50'

config member 'wan2_m50_w50'
        option metric '50'
        option weight '50'
        option interface 'wan2'

config member 'wan3_m50_w50'
        option interface 'wan3'
        option metric '50'
        option weight '50'

This problem has been solved. It seems that after openwrt is updated, you need to turn on the add route in iwiregard and allow 0.0.0/0 to open at the same time to use the default gateway

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@feckert feckert

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@feckert @zhihuiyuze