interfaces: 6RD tracking with /64 prefix

[paride]

I think I can correctly set up a 6rd tunnel (a /64):

root@net6501:~ # ifconfig wan_stf
wan_stf: flags=4041<UP,RUNNING,LINK2> metric 0 mtu 1280
	inet6 2001:b07:xx:yy:: prefixlen 64 
	nd6 options=1<PERFORMNUD>
	v4net 81.208.50.214/32 -> tv4br 81.208.50.214
	groups: stf 

root@net6501:~ # route -6 get default
   route to: default
destination: default
       mask: default
    gateway: 2001:b07:xx:yy::51d0:32d6
        fib: 0
  interface: wan_stf
      flags: <UP,GATEWAY,DONE,STATIC>
 recvpipe  sendpipe  ssthresh  rtt,msec    mtu        weight    expire
       0         0         0         0      1280         1         0 

root@net6501:~ # ping6 -c1 ipv6.google.com
PING6(56=40+8+8 bytes) 2001:b07:xx:yy:: --> 2a00:1450:4002:80a::200e
16 bytes from 2a00:1450:4002:80a::200e, icmp_seq=0 hlim=58 time=7.225 ms

--- ipv6.l.google.com ping6 statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/std-dev = 7.225/7.225/7.225/0.000 ms

But when I configure my LAN IPv6 to track the WAN interface (prefix id 0, I don't have other choice), the IPv6 configuration stops working: I can't ping ipv6.google.com anymore. I think this is happening because the default route gets overwritten (em1 is LAN):

root@net6501:~ # route -6 get default
   route to: default
destination: default
       mask: default
    gateway: 2001:b07:xx:yy::51d0:32d6
        fib: 0
  interface: em1
      flags: <UP,GATEWAY,DONE,STATIC>
 recvpipe  sendpipe  ssthresh  rtt,msec    mtu        weight    expire
       0         0         0         0      1500         1         0 

while I think I still should be getting interface: wan_stf here, as it was before. On the webui the Interface for the WAN_6RD gateway is still WAN (see attached screenshot). Is this an OPNsense issue, or am I misunderstanding something? Suggestions and feedback will be appreciated. Thank you!

[fichtner]

It’s a reported 6rd limitation. We don’t know yet if this can be worked around.

…

On 22. Aug 2018, at 11:57, Paride Legovini ***@***.***> wrote: I think I can correctly set up a 6rd tunnel (a /64): ***@***.***:~ # ifconfig wan_stf wan_stf: flags=4041<UP,RUNNING,LINK2> metric 0 mtu 1280 inet6 2001:b07:xx:yy:: prefixlen 64 nd6 options=1<PERFORMNUD> v4net 81.208.50.214/32 -> tv4br 81.208.50.214 groups: stf ***@***.***:~ # route -6 get default route to: default destination: default mask: default gateway: 2001:b07:xx:yy::51d0:32d6 fib: 0 interface: wan_stf flags: <UP,GATEWAY,DONE,STATIC> recvpipe sendpipe ssthresh rtt,msec mtu weight expire 0 0 0 0 1280 1 0 ***@***.***:~ # ping6 -c1 ipv6.google.com PING6(56=40+8+8 bytes) 2001:b07:xx:yy:: --> 2a00:1450:4002:80a::200e 16 bytes from 2a00:1450:4002:80a::200e, icmp_seq=0 hlim=58 time=7.225 ms --- ipv6.l.google.com ping6 statistics --- 1 packets transmitted, 1 packets received, 0.0% packet loss round-trip min/avg/max/std-dev = 7.225/7.225/7.225/0.000 ms But when I configure my LAN IPv6 to track the WAN interface (prefix id 0, I don't have other choice), the IPv6 configuration stops working: I can't ping ipv6.google.com anymore. I think this is happening because the default route gets overwritten (em0 is LAN): ***@***.***:~ # route -6 get default route to: default destination: default mask: default gateway: 2001:b07:xx:yy::51d0:32d6 fib: 0 interface: em1 flags: <UP,GATEWAY,DONE,STATIC> recvpipe sendpipe ssthresh rtt,msec mtu weight expire 0 0 0 0 1500 1 0 while I think I still should be getting interface: wan_stf here, as it was before. On the webui the Interface for the WAN_6RD gateway is still WAN (see attached screenshot). Is this an OPNsense issue, or am I misunderstanding something? Suggestions and feedback will be appreciated. Thank you! — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

[paride]

Thanks @fichtner. So does 6rd work properly on OPNsense only when more than a /64 is assigned?

The point is: it does work with the ISP provided router, so there has to be a way. I have tried to set up 6rd directly on the LAN interface (which is how I found out #2661), but routers are not advertised in this way (radvd is not started). Suggestions are welcome.

[fichtner]

6rd was just recently brought back with 18.7 and probably suffers from a longer code slumber. It probably shouldn’t pull a WAN address as is configurable in DHCPv6 mode to be able to yield the prefix to LAN. I’ll look into it in September when I’m back at the office.

[fichtner]

I don’t see code for stripping of the default route so this seems to be a side effect although I’m not sure where. Tracking on 6rd works differently than on „traditional“ dhcp6 and the code suggest tracking more than one address is not supported because the LAN address always ends up as xxxx::1. Still a bit of work ahead of us. 😊

[paride]

Thanks for looking into it, @fichtner. I am very willing to test stuff and provide feedback, just ping me.

[fichtner]

A code audit later it looks like 6rd has issues with setting up system routes in general. This will take a while, but working on it.

[fichtner]

@paride hi! :) is this still the case on 18.7.3 ? do you have means to try the development version and see if that makes a change if 18.7.3 hasn't ? I'm flying mostly blind for 6RD unfortunately. Appreciate all feedback.

[thomasjsn]

My 6RD stopped working with 18.7.2, but started working again with 18.7.3. Looked like route issue with 18.7.2.

[fichtner]

@thomasjsn yup, that was #2698

[paride]

@fichtner I will try 18.7.3 tomorrow and let you know. I am fine with trying the development version if there's the need to. Thanks a lot for your work.

[fichtner]

@paride super, thanks ❤️

[paride]

@fichtner With 18.7.3 I can still reproduce the issue as I initially described it: 6rd on WAN works fine, but when I configure my LAN interface to track WAN, the default route is hijacked.

But there is more: if I deconfigure the LAN IPv6 setting it to None, the IPv6 default route interface goes back to what is was before (wan_stf), but the IPv6 connection does not come back to life (e.g. I can't ping ipv6.google.com). At this point the only non-local inet6 address is assigned to wan_stf, as it should be.

I tried to disable the IPv6 default route from the web interface, and noticed that it is not actually disabled: by running route -6 get default I still get an "UP" gateway, and indeed if I try to ping ipv6.google.com I do not get an "address unreachable" error.

I can try the development version, please just hint me on how to install it (I remember reading the command in one of your comments here, but I can't recall it).

[fichtner]

Bummer, ok... here we go...

Do you have aliases? If so, please make a copy (config.xml download) for your configuration via System: Configuration: Backups. You need this for restoring your aliases on 18.7.3 as the development version uses the newer alias system. If you don't have aliases, download a backup just to be safe.

Changing is easy: System: Firewall: Settings: Set "release type" to development. Save. Check for updates. Install updates. Since this affects 6RD please reboot as the interfaces for 6RD have been renamed to e.g. em0_stf.

The same transition for going back. Import your config.xml to retain your old aliases. Reboot due to 6RD interface name changing back to the old format.

[paride]

@fichtner same problem with 19.1.a_227-i386 :(

(There is also another issue: the 6rd configuration only works if I configure it with the hack I described in #2662 (comment) and reboot. If I use the "6RD IPv4 Prefix address" option it doesn't work, despite the stf interface being configured in an almost identical way. But I think this is a different issue, and it is probably better to tackle one thing at a time...)

[fichtner]

ok, thank you for confirming. let's go back to 18.7.3 and solve this ticket first.

What I know is that "tracking" injects an IPv6 into the tracking interface with ::1. My suspicion is that this also causes the routing table to behave differently, but it's very hard to see why. Maybe it thinks the adapter is in the same subnet and thus traffic is not forwarded to the correct interface anymore?

[fichtner]

Is radvd running for you?

# pgrep radvd

[paride]

Maybe it thinks the adapter is in the same subnet and thus traffic is not forwarded to the correct interface anymore?

I think this is the case. If I manually remove the default route (route -6 del default), then add it again (route -6 add default <ip6>), and then I check how the route has been configured, I see that the chosen interface is the LAN interface, as if the route address belonged to its subnet.

If I set a default route with: route -6 add default -interface em0_stf (as in "far gateway"?) then it seems to work as expected. I still wasn't able to ping the ipv6 Internet from my LAN, but this may be a different issue.

Once I set up a tracking interface then radvd is started.

[fichtner]

Ok, I have a suspicion... please use "Manual configuration" for your tracking LAN, go to Services: Router Advertisements: [LAN] setting "Router Advertisements" to "unmanaged" and "Advertise Default Gateway" to off. I have the feeling that WAN snatches a route from radvd pointing to the LAN interface but that would only work for clients in LAN, not the WAN interface on your box.

[paride]

@fichtner done, but the gateway interface still changes to the LAN interface once the ipv6 tracking is set up.

[fichtner]

Can you provide a full ifconfig + netstat -nr ? ... can send over to franco@opnsense.org for privacy if you want.

[fichtner]

(in the misconfigured state and working state)

[paride]

@fichtner done

[fichtner]

That would also explain why you're the only one reporting... others have larger prefixes so that's not an issue... makes sense? :)

[paride]

Makes sense and setting prefixlen to /128 does work! The gateway interface stays set to em0_stf now. Still this is not truly a solution, right?

I did try to add the default ipv6 gateway using route the -ifp modifier (route -6 add default -ifp em0_stf <gw ipv6>), but the gateway interface is set to em0 in this case...

[fichtner]

Actually it is. I'd prefer 128. It means it's a host-only address. Transit interface for LAN only. How are your clients in this scenario? http://test-ipv6.com/ and http://ipv6-test.com/ are good tests.

Besides, there's precedent in GIF and GRE tunnels which are very similar to 6RD:

core/src/etc/inc/interfaces.inc

Line 849 in 772ea4a

mwexec("/sbin/ifconfig {$gifif} inet6 " . escapeshellarg($gif['tunnel-local-addr']) . " " . escapeshellarg($gif['tunnel-remote-addr']) . " prefixlen 128");

I know that IPv6 implementations refuse to accept forced gateway interfaces and produce strange errors as a result.

[paride]

@fichtner If I set prefixlen to something different from 64 (I tried: /128, /63, /96) my clients do not get autoconfigured anymore...

[fichtner]

# clog /var/log/routing.log | grep radvd

Is radvd starting when we have the wrong prefix? There is a Base6Interface parameter in radvd that we could use to follow the real interface maybe.

[paride]

@fichtner it does start but complains: Sep 24 16:57:55 net6501 radvd[64234]: prefix length should be 64 for em1.

@fichtner I'm sorry this turned out more convoluted than expected, but hopefully I won't be the only OPNsense user to benefit of this work.

[fichtner]

I appreciate the help in getting to the bottom of this. Already learned a few things along the way. Give me a day to propose a test patch. :)

[fichtner]

@paride alright, 49a4981 is a radically different approach to try. Run

# opnsense-patch 49a498195

to try (should also reboot) and patch again to remove.

[paride]

@fichtner will do; does it apply on 18.7.4 or to the development version? (I still have to revert to the stable version...).

[fichtner]

I used dev, but both should work

[paride]

@fichtner seems to behave as before: the interface of the default gateway is set to LAN once I configure LAN to track WAN. Don't we have the same prefixlen = 64 problem?

[fichtner]

The issue is the same of course, but I was hoping to side-step it through this... but that would further point to "it's impossible" with only a /64 from the ISP. I have no other ideas. :(

[paride]

@fichtner well, the ISP router somehow manages to do it, there must be a way. Can't be force the inet6 address assigned to LAN to have a 128bit prefix? I don't see it specified in the patch.

[fichtner]

I was hoping to get rid of the other code, but let's try the hybrid solution then.

[paride]

@fichtner any news on this? I have some more free time on these weeks, so I'm fine with testing stuff and breaking my setup :)

[icebalm]

I just hit this problem today, glad I found this thread. 6rd works from opnsense itself, as soon as you turn on track interface on the LAN it dies.... Running 18.7.7.

[fichtner]

November has been extremely busy, sorry. Bumping this to feature.

[fichtner]

19.1 will not allow tracking from LAN so we hope to get more people involved to find a suitable solution.

[tbandixen]

I think I'm having the same issue.
ifconfig:

wan_stf: flags=4041<UP,RUNNING,LINK2> metric 0 mtu 1480
        inet6 xxxx:xxxx:xxxx:xxxx:: prefixlen 64

If I set track interface on LAN, I can't ping ipv6.google.com from the opnsense.

OPNsense 19.1.5_1-amd64
FreeBSD 11.2-RELEASE-p9-HBSD
OpenSSL 1.0.2r 26 Feb 2019

[fichtner]

I'm dropping this issue. With a "dead end" prefix of /64 we won't be able to do anything clean and since 19.1 the system refuses to allow tracking.