dhcp6c over PPPoE IPv4 WAN fails on 23.1.r1

[csutcliff]

Important notices

Before you add a new report, we ask you kindly to acknowledge the following:

• I have read the contributing guide lines at https://github.com/opnsense/core/blob/master/CONTRIBUTING.md
• I am convinced that my issue is new after having checked both open and closed issues at https://github.com/opnsense/core/issues?q=is%3Aissue

Describe the bug
DHCPv6 with Use IPv4 connectivity (PPPoE WAN) doesn't function on upgrade from 22.7.10_2 to 23.1.r1

To Reproduce

Steps to reproduce the behavior:

• Upgrade to 23.1.r1
• Observe no IPv6 connectivity and error log entries for dhcp6c transmit failed: Network is down. IPv4 connectivity is otherwise normal.

Describe alternatives you considered

manually running dhcp6c via /usr/local/sbin/dhcp6c -f -d -D -c /var/etc/dhcp6c_wan.conf -p /var/run/dhcp6c_pppoe0.pid pppoe0 also repeats the error transmit failed: Network is down

Additional context

Reverting to 22.7.10_2 fixes the problem

[fichtner]

Are you sure you’re not falling into the „feature trap“ that is the new PPPoEv6 mode and DHCPv6 never really worked? There isn’t much of a drift between DHCPv6 on 22.7 and 23.1. I’m using the latter every day.

Cheers,
Franco

[csutcliff]

Quite possibly. I'm with Zen in the UK so I had previously had it set up as per the docs

I saw the line interfaces: add PPPoEv6 mode to prevent IPv6 CP negotiation over PPPoE in other IPv6 modes in the release notes but I wasn't clear what it meant in practice.

I've just tried setting the WAN IPv6 Configuration Type to PPPoEv6 but there is still no connection and a new error appears in the logs:

/system_gateways.php: The command '/sbin/route add -'inet6' default 'fe80::827f:f8ff:fe74:b4f3%igb0'' returned exit code '1', the output was 'route: writing to routing socket: Network is unreachable add net default: gateway fe80::827f:f8ff:fe74:b4f3%igb0 fib 0: Network is unreachable'

What would be the recommended way of configuring this in 23.1?

[fichtner]

%igb0 is wrong it seems and should be %pppoe0. Perhaps that was the case for a long time but never surfaced. Let me try to take a closer look tomorrow and perhaps propose a patch.

You could always revert the patch with the PPPoEv6 separately but it’s too late here and no computer to grab the necessary info.

Cheers,
Franco

[fichtner]

@csutcliff can you list the files on your system for me? Not entirely sure how to reproduce yet.

# ls /tmp/igb*_* /tmp/pppoe*_*

[fichtner]

I also need:

# grep dhcp6usev4iface /conf/config.xml

[csutcliff]

With the interface set to PPPoEv6 :

# ls /tmp/igb*_* /tmp/pppoe*_*
/tmp/pppoe0_nameserver	/tmp/pppoe0_oldip	/tmp/pppoe0_router	/tmp/pppoe0_routerv6	/tmp/pppoe0_uptime

# grep dhcp6usev4iface /conf/config.xml

With it set to DHCPv6 :

# ls /tmp/igb*_* /tmp/pppoe*_*
/tmp/pppoe0_nameserver	/tmp/pppoe0_oldip	/tmp/pppoe0_router	/tmp/pppoe0_uptime

# grep dhcp6usev4iface /conf/config.xml
      <dhcp6usev4iface>1</dhcp6usev4iface>

With the interface set to PPPoEv6 the entire DHCPv6 client configuration section disappears including Use IPv4 connectivity

[fichtner]

Everything looks as expected. Can you show the contents of pppoe0_routerv6 in PPPoEv6 mode? Somehow it’s mismatching in get_real_interface() but don’t see how just yet.

[csutcliff]

# cat /tmp/pppoe0_routerv6
fe80::827f:f8ff:fe74:b4f3

[fichtner]

Ok thanks that’s the one I expected. I will try to reproduce the mismatch now. It’s definitely in the code.

[csutcliff]

I checked the routing table and there was no default route for ipv6. I added a manual route for '::/0' via the PPPoEv6 gateway and now it appears to be working aside from the gateway monitoring which is still showing as offline. I am monitoring a cloudflare IP 2606:4700:4700::1001 as I was in 22.7 because monitoring the gateway IP was unreliable.

WAN_PPPOEV6
fe80::827f:f8ff:fe74:b4f3	~	~	~	Offline

pinging that IP manually from opnsense or a LAN client works.

[fichtner]

Ok, as stated I think that

/sbin/route add -'inet6' default 'fe80::827f:f8ff:fe74:b4f3%igb0'

should be

/sbin/route add -'inet6' default 'fe80::827f:f8ff:fe74:b4f3%pppoe0'

and matches what you just said. Not sure where DHCPv6 plays a role here yet. Did you have interface tracking enabled? Did you get a delegated prefix before?

[fichtner]

PS: The monitor likely has the same issue with the interface mismatch via get_real_interface()

[csutcliff]

Yes that makes sense, the log files contain errors

/system_gateways.php: The required WAN_PPPOEV6 IPv6 interface address could not be found, skipping.

I have the LAN IPv6 as Static. DHCPv6 might be a red herring, it's how the WAN was configured in <23.1 but as you said earlier it might have never really been working.

[fichtner]

@csutcliff found it.... can you try 9934ef0?

# opnsense-patch 9934ef0

[csutcliff]

I had just made the same edit locally myself but yes I can confirm it works.

[fichtner]

Yay, it looks like forcing "dhcp6usev4iface" would have had the same effect but making this explicit in get_real_interface() is likely more robust in the long run. I've labeled this "cleanup" since not in production yet but I'll make sure this hits RC2 and naturally 23.1 with a migration note so it works as expected for all involved.

Thanks for the report and the use case. Static IPv6 LAN makes sense here after setting up basic connectivity via PPPoE. :)

Cheers,
Franco

[csutcliff]

My parting thought:

Would it be beneficial to have the older PPPoE IPv4 and DHCPv6 IPv6 with Use IPv4 Connectivity set migrated automatically to PPPoE IPv4 and PPPoE IPv6 or are there use cases where you would want to retain the former?

[fichtner]

It would solely depend on the ISP side so we cannot know beforehand. For me the separation is important to get rid of side effects due to double-IPv6 assignment in some rare cases. We've observed these types of bugs before.

[csutcliff]

that makes sense, just migration notes then.

[fichtner]

@csutcliff just to let you know we had to follow up with a partial revert that makes migration notes unnecessary f60333622aa

At least one provider requires both PPPoEv6 and DHCPv6 enabled to get a delegated prefix. But the PPPoEv6 mode will stay and rightfully turn off DHCPv6 at least. I'll try to keep working on this to separate the cases a bit better where it is possible.