IPv6 connection not working on clients after 24h reconnect

[aponert]

Forum link for reference: https://forum.opnsense.org/index.php?topic=6946.msg31674#msg31674

After the 24 hours reconnect, IPv6 is no more working on the clients. As the forum user mentions it seems that the clients are assigned an old prefix although after reconnect the prefix changed.
I can also confirm that issue. v 18.1_1

[fichtner]

Could be #2164, although that was the case since 2016 if true.

[aponert]

I can try to apply opnsense-patch f09eaf7. Will report the next 2-3 days if it helps.

[fichtner]

@theq86 thanks! But this will very likely hit 18.1.2 tomorrow anyway.

[aponert]

Well, seems f09eaf7 does not fix this issue. Again, no correct v6 connection.

[fichtner]

Ok, we'll keep looking. Thanks for the double-check.

[aponert]

What happens during "periodic interface reset"?

I tried to find out something more about this issue.

First, what I stated earlier is wrong, all clients get the correct IPv6 address. But after periodic interface reset (crontabbed every morning at 4 o' clock on my box) IPv6 internet stops working.

Some facts:
Client -> LAN address -> ping works
Client -> WAN address -> ping works
Client -> Internet -> does not work.

Then I tcpdump'ed the connections on the client, the opnsense' pppoe1 interface and on a public server.

Client: obviously, ping request is shown, response not
OPNsense (pppoe1): same here, request is shown, response is not
internet server: receives ping, sends reply

So the way towards the server is working. but the way back does not, not even the pppoe interface receives the echo reply.

After OPNsense reboot, everything is fine until the next reset

[fichtner]

Multiple tickets and patches landing on master at the moment. Can you reevaluate on the development version of 18.1.3 when that is out?

[aponert]

Yes. Tell me when I can test it and how to update/downgrade to/from the dev version.

[fichtner]

I've been unspecific regarding commits because there have been a couple that seem to improve things in tandem. You can switch to the dev version from System: Firmware: Settings and hit save and update. Then...

# opnsense-code core
# cd /usr/core && make upgrade

Cheers,
Franco

[aponert]

I can test it at the weekend. These changes will hit 18.1.4 then?

[fichtner]

At least all in dev version of 18.1.4. We don't necessarily put them into the release without testing. :)

[fichtner]

@theq86 how's 18.1.4 working for you?

[aponert]

I will test it tonight. Until now I still had the full reboots in my crontab.

[aponert]

Unfortunately not better in 18.1.4. :-(

[fichtner]

Step by step. How about this? 64057c13

# opnsense-patch 64057c1

[fichtner]

Same one, github truncates the hash after posting...

[fichtner]

Logs should be a lot better now on 18.1.4, too. Can you post sanitised "newwanip" logs of your next test?

[aponert]

yes. I'll post them here in the evening.

[aponert]

The issue is still there. No IPv6 after periodic interface reset, v4 working ok.

Ok, I applied the above patch and of course I'm on 18.1.4.
I'll attach my pppoe and system log for the appropriate time.
pppoe.log
system.log

[fichtner]

Where does "62.214.63.93" come from? Can you verify in ifconfig where this is? It picks up the wrong IP, so that's why it fails quite simply oO

[fichtner]

If it's not in ifconfig, it's set in a gateway under system: gateways: single maybe...

[fichtner]

# grep 62.214.63.93 /conf/config.xml

[aponert]

This IP could not be found in the config.xml. It is in fact my Gateway IP. This is the one I get from my ISP.
Now where I rebooted my sense, everything is working. Still have the same IP as my gateway but it does not come from within my configuration.

[fichtner]

Ok, will need these to before and after daily reset:

# netstat -nr | grep default

Maybe it lands on the wrong interface after reconnect. :/

[aponert]

Okay, I can provide this info when I'm back from work.

BTW after interface reset IPv4 internet works but I can't connect to openvpn. Maybe it is a firewall thing. But let's first check on the routing.

[aponert]

before:

default            62.214.63.93       UGS      pppoe0
default                           fe80::5651:1bff:fe83:bbbd%pppoe0 UGS   pppoe0

after:

default            62.214.63.93       UGS      pppoe0
default                           fe80::5651:1bff:fe83:bbbd%pppoe0 UGS   pppoe0

that's the same output.

for comparison I attach the same pppoe and system log during a normal boot, when everything is ok.
pppoe-reboot.log
syslog-reboot.log

[fichtner]

Thanks, I think this is deep inside netgraph that makes this fail as someone else reported lately. Was this test with the additional 64057c1 patch or without?

[aponert]

It was including 64057c1

[fichtner]

Ok, thanks for confirming. To me it seems something is completely broken and it's not our scripting... :(

[fichtner]

@theq86 18.1.7 replaced dhcp6c and a race condition in dhcp6c daemon creation. Does that make any difference here?

[aponert]

@fichtner
I just upgraded to 18.1.7 and will have a look on that. I was busy so no pre-testing was possible. I will report back in a few days. :-)

[aponert]

I deactivated the reboot cron. for 2 days now everything is working fine including ipv6.
However, I forgot to reactivate the periodic interface reset at 4 o' clock. I did it now and will report back. but so far so good

[aponert]

okay, tested for four days now. the issue is resolved.

[fichtner]

Thanks, great to hear. So 18.1.6 did not work but 18.1.7 does? In that case it was likely the SUGHUP addition for dhcp6c that prevents spawning of new processes which lock each other out. 😊