tunnel mtu problems #30

axn · 2018-05-21T21:19:07Z

MTU discovery does NOT work with bmx7 tunnels on openwrt devices

axn · 2018-05-21T21:42:51Z

The problem is that BMX7 by default uses a faked outer-ip-in-ip6 tunnel source address which prevents ICMPv6 Packet Too Big messages to reach the intended origin of the MTU-exceeding packet.

The problem need fixes on two sides:

Openwrt currently ships a patched ip4-in-ip6 kernel module that adds support for MAP-E-FMRs mesh mode. This patch actually breaks the possibility of using an ip4ip6 tunnel interface as a fall back interface accepting ip4-in-ip6 tunneled packets from any remote address which works out of the box with any normal (non-patched) kernel and can be configured by setting up an 'ip -6 tunnel' with type 'any' or 'ip4ip6' and a remote address of '::'.
I pushed a tentative fix for Re-Enabling any-remote tun6 addresses again
for openwrt-18.06 here:
axn/openwrt@1b8b264
and for openwrt-master here:
axn/openwrt@8ea3fc4
BMX7 must be modified to use its primary IPv6 address as outer ip-in-ip6 tunnel header.
Again, a tentative commit can be found here:
d854135
which can be enabled via --tunAnySrc=1

@aparcar
can you check if this fixes the problem reported by you?

aparcar · 2018-05-25T17:55:14Z

root@meshrc-e8b7b4:/etc/tinc/librenet6/hosts# bmx7 --tunAnySrc=1
[27585        8] ERROR apply_stream_opts: invalid argument: tunAnySrc=1

Am I missing something? It's freshly compiled

aparcar · 2018-05-25T18:23:48Z

I setup the following:

Two nodes A/B are connected to an AP.
Both run BMX7 only on an interface called librenet6.
The librenet6 interfaces connects via Tinc to a VM running BMX7 as well.
I can access Bs web interface while being connected to A.
No additional MTU changes where required.

After all, I can't seem to start BMX7 accepting the tunAnySrc parameter. The 666 patch patch doesn't seem to break anything and your AllowZeroTunAddr patch is broken? Honestly I'm a bit confused.

axn · 2018-05-26T05:12:35Z

TunAnySrc parameter is only give with bmx7_tun.so plugin. So try bmx7 plugin=bmx7_tun.so tunAnySrc=1 Am 25. Mai 2018 19:55:17 MESZ schrieb Paul Spooren <notifications@github.com>:

…

``` ***@***.***:/etc/tinc/librenet6/hosts# bmx7 --tunAnySrc=1 [27585 8] ERROR apply_stream_opts: invalid argument: tunAnySrc=1 ``` Am I missing something? It's freshly compiled -- You are receiving this because you authored the thread. Reply to this email directly or view it on GitHub: #30 (comment)

-- Diese Nachricht wurde von meinem Android-Gerät mit K-9 Mail gesendet.

axn · 2018-05-28T05:12:50Z

This works:

root@mlc1000:~# bmx7 -cpd8
PARAMETERS:
 plugin                 bmx7_config.so       (0) 
 plugin                 bmx7_json.so         (0) 
 plugin                 bmx7_sms.so          (0) 
 plugin                 bmx7_tun.so          (0) 
 plugin                 bmx7_topology.so     (0) 
 plugin                 bmx7_table.so        (0) 
 dev                    eth1                 (0) 
 unicastHna             2013:0:0:1000::/64   (0) 
 tunAnySrc              1                    (0) 
 tunDev                 default              (0) 
    /tun4Address        10.20.0.1/24         (0) 
    /tun6Address        2013:0:0:1000::1/64  (0) 
 tunOut                 ip6                  (0) 
    /network            2013::/16            (0) 
 tunOut                 ip4                  (0) 
    /network            10.20.0.0/16         (0) 
STATUS:
shortId  name    nodeKey cv revision primaryIp                               tun6Address         tun4Address  uptime     cpu txQ  nbs rts nodes 
F93ED1A4 mlc1000 RSA2048 21 d854135  fd70:f93e:d1a4:7ff8:dd9d:2404:1ec6:afa8 2013:0:0:1000::1/64 10.20.0.1/24 0:00:02:51 0.1 0/50 1   2   3/3   
INTERFACES:
dev  state linkKey    linkKeys          type     channel rateMax idx localIp                   rts helloSqn rxBpP   txBpP   
eth1 UP    DH2048M112 RSA896,DH2048M112 ethernet 0       1000M   1   fe80::a2cd:efff:fe10:1/64 2   26750    197/1.2 143/1.3 
LINKS:
shortId  name    linkKey    linkKeys          nbLocalIp                dev  rts rq  tq  txRate wTxRate wTxRateEff wTxThr wTxThrEff mcs sgi chw wSnr 
4307C006 mlc1001 DH2048M112 RSA896,DH2048M112 fe80::a2cd:efff:fe10:101 eth1 2   100 100 1000M  -1      -1         -1     -1        0   0   20  0   
ORIGINATORS:
shortId  name    as S s T t descSqn lastDesc descSize cv revision primaryIp                               dev  nbShortId nbName  metric hops ogmSqn lastRef 
F93ED1A4 mlc1000 nQ A A A A 612     171      705+768  21 d854135  fd70:f93e:d1a4:7ff8:dd9d:2404:1ec6:afa8 ---  ---       ---     257G   0    25     5       
4307C006 mlc1001 nA A A A A 412     170      705+768  21 d854135  fd70:4307:c006:1d87:72fd:f7d4:e2d9:48a8 eth1 4307C006  mlc1001 999M   1    26     0       
9BC3537B mlc1002 pA A A A A 513     22       705+768  21 d854135  fd70:9bc3:537b:6fb4:ca32:d887:9d9d:3047 eth1 4307C006  mlc1001 706M   2    4      4       
TUNNELS:
tunOut id       gwName net          min max hyst rating minBw tunName     tunRoute           remoteId remoteName advNet             advBw pathMtc tunMtc 
ip4    00000000 ---    10.20.0.0/16 16  128 20   100    960   X7C4default 10.20.1.0/24       4307C006 mlc1001    10.20.1.0/24       257G  999M    1199M  
ip4    00000000 ---    10.20.0.0/16 16  128 20   100    960   X7C4default 10.20.2.0/24       9BC3537B mlc1002    10.20.2.0/24       257G  706M    847M   
ip6    00000000 ---    2013::/16    16  128 20   100    960   X7C6default 2013:0:0:1001::/64 4307C006 mlc1001    2013:0:0:1001::/64 257G  999M    1199M  
ip6    00000000 ---    2013::/16    16  128 20   100    960   X7C6default 2013:0:0:1002::/64 9BC3537B mlc1002    2013:0:0:1002::/64 257G  706M    847M   
root@mlc1000:~# ping 10.20.1.1
PING 10.20.1.1 (10.20.1.1) 56(84) bytes of data.
64 bytes from 10.20.1.1: icmp_seq=1 ttl=63 time=2.43 ms
64 bytes from 10.20.1.1: icmp_seq=2 ttl=64 time=0.097 ms
^C
--- 10.20.1.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 0.097/1.265/2.433/1.168 ms
root@mlc1000:~#

altergui · 2018-12-12T15:23:41Z

here testing the code with @G10h4ck, here is our result:

the "old" bmx7 (without tunAnySrc commit, or also with tunAnySrc commit included but without specifying tunAnySrc=1) produces these tunnel interfaces:

34: X7main@NONE: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1460 qdisc noqueue state UNKNOWN group default qlen 1
    link/tunnel6 fd70:4887:bc2d:ce6f:7dad:f423:8323:c593 peer fd71:4887:bc2d:ce6f:7dad:f423:8323:c593
    inet 10.82.172.170/32 scope global X7main
       valid_lft forever preferred_lft forever
    inet6 2012:0:0:aa::/128 scope global 
       valid_lft forever preferred_lft forever
    inet6 fe80::d0bf:45ff:fedc:bcb5/64 scope link 
       valid_lft forever preferred_lft forever
[...snip...]
46: X7Out_570B504A@NONE: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1358 qdisc noqueue state UNKNOWN group default qlen 1
    link/tunnel6 fd71:570b:504a:601d:977f:dbd7:8c86:24b5 peer fd70:570b:504a:601d:977f:dbd7:8c86:24b5
    inet 10.82.172.170/32 scope global X7Out_570B504A
       valid_lft forever preferred_lft forever
    inet6 2012:0:0:aa::/128 scope global 
       valid_lft forever preferred_lft forever
    inet6 fd71:570b:504a:601d:977f:dbd7:8c86:24b5/128 scope global deprecated 
       valid_lft forever preferred_lft 0sec

X7main with source (real) fd70: and peer (fake) fd71:
and X7Out with source (fake) fd71: and destination (real) fd70:

the "new" bmx7 (after applying tunAnySrc=1 like this in two nodes of different LibreMesh clouds):

root@LiMe-7aacaa:~# bmx7 -c plugin=bmx7_tun.so tunAnySrc=1
INFO  uci_save_option: bmx7.general.tunAnySrc=1
INFO  : --tunAnySrc               1                             
root@LiMe-7aacaa:~# killall bmx7 && sleep 10

(regarding the killall bmx7: if bmx7 is not fully restarted, the new X7Out interfaces will use real addresses but the X7main is not recreated and will still have a fake "peer")

successfully can ping to the other node over the tunnel!!!

root@LiMe-7aacaa:~# ip r
10.13.47.0/24 dev X7C4main proto static metric 1024 
10.82.172.0/24 dev br-lan proto kernel scope link src 10.82.172.170 
root@LiMe-7aacaa:~# ping -c 3 10.13.47.16
PING 10.13.47.16 (10.13.47.16) 56(84) bytes of data.
64 bytes from 10.13.47.16: icmp_req=1 ttl=64 time=8.47 ms
64 bytes from 10.13.47.16: icmp_req=2 ttl=64 time=2.14 ms
64 bytes from 10.13.47.16: icmp_req=3 ttl=64 time=2.14 ms

--- 10.13.47.16 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 2.142/4.254/8.476/2.985 ms
root@LiMe-7aacaa:~# ip r
10.13.47.0/24 dev X7Out_570B504A proto static metric 1024 
10.82.172.0/24 dev br-lan proto kernel scope link src 10.82.172.170

and this created tunnel X7Out is using real source addresses, and the X7main has an "empty" (::) peer, as can be seen

55: X7main@NONE: <NOARP,UP,LOWER_UP> mtu 1460 qdisc noqueue state UNKNOWN group default qlen 1
    link/tunnel6 fd70:4887:bc2d:ce6f:7dad:f423:8323:c593 brd ::
    inet 10.82.172.170/32 scope global X7main
       valid_lft forever preferred_lft forever
    inet6 2012:0:0:aa::/128 scope global 
       valid_lft forever preferred_lft forever
    inet6 fe80::18e3:b8ff:fe83:8d0f/64 scope link 
       valid_lft forever preferred_lft forever
56: X7C4main: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UNKNOWN group default qlen 500
    link/none 
    inet 10.82.172.170/32 scope global X7C4main
       valid_lft forever preferred_lft forever
    inet6 fe80::86a6:4b18:7028:40c3/64 scope link stable-privacy 
       valid_lft forever preferred_lft forever
57: X7C6main: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UNKNOWN group default qlen 500
    link/none 
    inet6 2012:0:0:aa::/128 scope global 
       valid_lft forever preferred_lft forever
    inet6 fe80::ae29:e69d:ea20:b0b2/64 scope link stable-privacy 
       valid_lft forever preferred_lft forever
58: X7Out_570B504A@NONE: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1358 qdisc noqueue state UNKNOWN group default qlen 1
    link/tunnel6 fd70:4887:bc2d:ce6f:7dad:f423:8323:c593 peer fd70:570b:504a:601d:977f:dbd7:8c86:24b5
    inet 10.82.172.170/32 scope global X7Out_570B504A
       valid_lft forever preferred_lft forever
    inet6 2012:0:0:aa::/128 scope global 
       valid_lft forever preferred_lft forever
    inet6 fd70:4887:bc2d:ce6f:7dad:f423:8323:c593/128 scope global deprecated 
       valid_lft forever preferred_lft 0sec
    inet6 fe80::3c07:cff:fe6d:43bb/64 scope link 
       valid_lft forever preferred_lft forever

so, we have just reproduced independently @axn test, and it looks great! thanks a lot

what's pending is to test this "in the real world" (or at least with a reduced MTU segment in the middle of the path) and see if it allows PMTUD to work correctly, finally ending the infamous "dead" tunnels

in particular, we could not compile bmx6 at this particular commit bmx-routing/bmx6@5dc6678 on top of current openwrt, understandably since it's old code. But this means someone should port this tunAnySrc=1 patch to bmx6 as @pedro-nonfree suggested, so that we can compile a "fixed" bmx6 and test it in quintanalibre for example

axn · 2019-02-23T05:06:36Z

@altergui any updates from the "in the real world" tests of d854135 ?
And did you discover any incompatibilities with current master.
Otherwise I would like to merge d854135 into master and close this issue...

axn · 2019-03-08T17:05:05Z

I added d854135 plus another cc245a2 patch to testing https://github.com/bmx-routing/bmx7/commits/testing branch.

The latter commit replaces tunAnySrc=<0|1> with tunRealSrc=<0|1|2> parameter which should be used as follows:

Use tunRealSrc=0 (default) if your specific device suffers from the (openwrt) kernel problem reported above.
Use tunRealSrc=1 if the device kernel is fine (No matter which other kernel or tunRealSrc setting other nodes are running). tunRealSrc=1 should be compatible with all of them.
Use tunRealSrc=2 if your kernel is fine and other nodes (especially your GW or GW-clients) have tunRealSrc>=1 configured.

In the long term I'll make tunRealSrc=1 and later tunRealSrc=2 the default...

Hope this helps to smoothly get away with asymmetric tunnels and faked tunIps

Please @altergui use and report

dangowrt · 2019-03-08T23:57:44Z

Just updated the PR for openwrt-routing/packages.git after some testing:
openwrt/routing#454

pedro-nonfree mentioned this issue Jun 26, 2018

solving 'tunnel mtu problems" for bmx6 too bmx-routing/bmx6#18

Open

axn self-assigned this Feb 23, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

tunnel mtu problems #30

tunnel mtu problems #30

axn commented May 21, 2018 •

edited

Loading

axn commented May 21, 2018

aparcar commented May 25, 2018

aparcar commented May 25, 2018

axn commented May 26, 2018 via email

axn commented May 28, 2018 •

edited

Loading

altergui commented Dec 12, 2018

axn commented Feb 23, 2019

axn commented Mar 8, 2019

dangowrt commented Mar 8, 2019

tunnel mtu problems #30

tunnel mtu problems #30

Comments

axn commented May 21, 2018 • edited Loading

axn commented May 21, 2018

aparcar commented May 25, 2018

aparcar commented May 25, 2018

axn commented May 26, 2018 via email

axn commented May 28, 2018 • edited Loading

altergui commented Dec 12, 2018

axn commented Feb 23, 2019

axn commented Mar 8, 2019

dangowrt commented Mar 8, 2019

axn commented May 21, 2018 •

edited

Loading

axn commented May 28, 2018 •

edited

Loading