pf: delayed responses on ICMP6 neighbor solicitation packets since 24.7.1 (redux) #218

Crazyachmed · 2024-08-22T13:22:17Z

Important notices

Before you add a new report, we ask you kindly to acknowledge the following:

I have read the contributing guide lines at https://github.com/opnsense/core/blob/master/CONTRIBUTING.md
I am convinced that my issue is new after having checked both open and closed issues at https://github.com/opnsense/core/issues?q=is%3Aissue

Describe the bug

Android devices drop WiFi connectivity every couple of minutes and reconnect immediately. A capture showed that opnsense does not reply to IPv6 neighbor solicitation for its link-local address right before the disconnections.

Using multiple wired Linux devices and the tool ndisc6 I can replicate the issue sending about one solicitation per second, failure rate is above 90%.

In a packet capture on opnsense itself I can verify that all neighbor solicitation arrive but are not answered most of the time.
There is no overlap in timing when one device receives an answer.
Sometimes a device randomly works correctly for multiple minutes without any changes, while other devices are still affected
Under Firewall > Diagnostics > Statistics > interfaces > vtnet0 I can see a rising count for out6_block_packets when ndisc6 is running on other devices
I don't see any drops in the log or hits in any drop-rules (including floating, auto generated and default deny).
The problem goes away if check "Disable Firewall" in the advanced firewall settings and comes back immediately after apply when unchecked.

To Reproduce

while :
do
	ndisc6 -m -n -r 1 fe80::5864:8aff:fef1:7a5 eth0
done

Change IP to link-local address of firewall and change interface to the name of the interface on the sending station.

Expected behavior

Every neighbor solicitation should be answered, however a reasonably rate-limit may apply (much higher than here)

Describe alternatives you considered

I suspected my Proxmox, Switch or APs to be at fault, but for me the tcpdump on opnsense nails it down for me.

Relevant log files

root@sense:~ # tcpdump -i vtnet0 icmp6 and ip6 net fe80::/64
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on vtnet0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
14:58:16.948460 IP6 fe80::7b5d:be4e:512:ac91 > ff02::1:fff1:7a5: ICMP6, neighbor solicitation, who has fe80::5864:8aff:fef1:7a5, length 32
14:58:17.952548 IP6 fe80::7b5d:be4e:512:ac91 > ff02::1:fff1:7a5: ICMP6, neighbor solicitation, who has fe80::5864:8aff:fef1:7a5, length 32
14:58:18.957144 IP6 fe80::7b5d:be4e:512:ac91 > ff02::1:fff1:7a5: ICMP6, neighbor solicitation, who has fe80::5864:8aff:fef1:7a5, length 32
14:58:19.706487 IP6 fe80::5864:8aff:fef1:7a5 > fe80::7b5d:be4e:512:ac91: ICMP6, neighbor solicitation, who has fe80::7b5d:be4e:512:ac91, length 32
14:58:19.707107 IP6 fe80::7b5d:be4e:512:ac91 > fe80::5864:8aff:fef1:7a5: ICMP6, neighbor advertisement, tgt is fe80::7b5d:be4e:512:ac91, length 24
14:58:19.961609 IP6 fe80::7b5d:be4e:512:ac91 > ff02::1:fff1:7a5: ICMP6, neighbor solicitation, who has fe80::5864:8aff:fef1:7a5, length 32
14:58:20.965847 IP6 fe80::7b5d:be4e:512:ac91 > ff02::1:fff1:7a5: ICMP6, neighbor solicitation, who has fe80::5864:8aff:fef1:7a5, length 32
14:58:21.396036 IP6 fe80::5864:8aff:fef1:7a5 > fe80::61f5:3d5:b49d:1ab0: ICMP6, neighbor solicitation, who has fe80::61f5:3d5:b49d:1ab0, length 32
14:58:21.470895 IP6 fe80::61f5:3d5:b49d:1ab0 > fe80::5864:8aff:fef1:7a5: ICMP6, neighbor advertisement, tgt is fe80::61f5:3d5:b49d:1ab0, length 24
14:58:21.970370 IP6 fe80::7b5d:be4e:512:ac91 > ff02::1:fff1:7a5: ICMP6, neighbor solicitation, who has fe80::5864:8aff:fef1:7a5, length 32
14:58:22.974822 IP6 fe80::7b5d:be4e:512:ac91 > ff02::1:fff1:7a5: ICMP6, neighbor solicitation, who has fe80::5864:8aff:fef1:7a5, length 32
14:58:23.979156 IP6 fe80::7b5d:be4e:512:ac91 > ff02::1:fff1:7a5: ICMP6, neighbor solicitation, who has fe80::5864:8aff:fef1:7a5, length 32
14:58:24.983611 IP6 fe80::7b5d:be4e:512:ac91 > ff02::1:fff1:7a5: ICMP6, neighbor solicitation, who has fe80::5864:8aff:fef1:7a5, length 32
14:58:25.987861 IP6 fe80::7b5d:be4e:512:ac91 > ff02::1:fff1:7a5: ICMP6, neighbor solicitation, who has fe80::5864:8aff:fef1:7a5, length 32
14:58:26.992253 IP6 fe80::7b5d:be4e:512:ac91 > ff02::1:fff1:7a5: ICMP6, neighbor solicitation, who has fe80::5864:8aff:fef1:7a5, length 32
14:58:27.996596 IP6 fe80::7b5d:be4e:512:ac91 > ff02::1:fff1:7a5: ICMP6, neighbor solicitation, who has fe80::5864:8aff:fef1:7a5, length 32
14:58:29.001043 IP6 fe80::7b5d:be4e:512:ac91 > ff02::1:fff1:7a5: ICMP6, neighbor solicitation, who has fe80::5864:8aff:fef1:7a5, length 32
14:58:30.005406 IP6 fe80::7b5d:be4e:512:ac91 > ff02::1:fff1:7a5: ICMP6, neighbor solicitation, who has fe80::5864:8aff:fef1:7a5, length 32
14:58:30.216896 IP6 fe80::b2f2:8ff:fe18:974d > fe80::5864:8aff:fef1:7a5: ICMP6, neighbor solicitation, who has fe80::5864:8aff:fef1:7a5, length 32
14:58:30.216931 IP6 fe80::5864:8aff:fef1:7a5 > fe80::b2f2:8ff:fe18:974d: ICMP6, neighbor advertisement, tgt is fe80::5864:8aff:fef1:7a5, length 24
14:58:30.499881 IP6 fe80::5864:8aff:fef1:7a5 > ff02::1: ICMP6, router advertisement, length 104
14:58:31.009706 IP6 fe80::7b5d:be4e:512:ac91 > ff02::1:fff1:7a5: ICMP6, neighbor solicitation, who has fe80::5864:8aff:fef1:7a5, length 32
14:58:32.013353 IP6 fe80::7b5d:be4e:512:ac91 > ff02::1:fff1:7a5: ICMP6, neighbor solicitation, who has fe80::5864:8aff:fef1:7a5, length 32
14:58:33.017650 IP6 fe80::7b5d:be4e:512:ac91 > ff02::1:fff1:7a5: ICMP6, neighbor solicitation, who has fe80::5864:8aff:fef1:7a5, length 32
14:58:34.022072 IP6 fe80::7b5d:be4e:512:ac91 > ff02::1:fff1:7a5: ICMP6, neighbor solicitation, who has fe80::5864:8aff:fef1:7a5, length 32
14:58:35.026415 IP6 fe80::7b5d:be4e:512:ac91 > ff02::1:fff1:7a5: ICMP6, neighbor solicitation, who has fe80::5864:8aff:fef1:7a5, length 32
14:58:35.026452 IP6 fe80::5864:8aff:fef1:7a5 > fe80::7b5d:be4e:512:ac91: ICMP6, neighbor advertisement, tgt is fe80::5864:8aff:fef1:7a5, length 32
14:58:36.030782 IP6 fe80::7b5d:be4e:512:ac91 > ff02::1:fff1:7a5: ICMP6, neighbor solicitation, who has fe80::5864:8aff:fef1:7a5, length 32
14:58:37.035154 IP6 fe80::7b5d:be4e:512:ac91 > ff02::1:fff1:7a5: ICMP6, neighbor solicitation, who has fe80::5864:8aff:fef1:7a5, length 32
14:58:38.039424 IP6 fe80::7b5d:be4e:512:ac91 > ff02::1:fff1:7a5: ICMP6, neighbor solicitation, who has fe80::5864:8aff:fef1:7a5, length 32
14:58:39.043784 IP6 fe80::7b5d:be4e:512:ac91 > ff02::1:fff1:7a5: ICMP6, neighbor solicitation, who has fe80::5864:8aff:fef1:7a5, length 32
14:58:40.048012 IP6 fe80::7b5d:be4e:512:ac91 > ff02::1:fff1:7a5: ICMP6, neighbor solicitation, who has fe80::5864:8aff:fef1:7a5, length 32
14:58:41.052328 IP6 fe80::7b5d:be4e:512:ac91 > ff02::1:fff1:7a5: ICMP6, neighbor solicitation, who has fe80::5864:8aff:fef1:7a5, length 32
14:58:42.056492 IP6 fe80::7b5d:be4e:512:ac91 > ff02::1:fff1:7a5: ICMP6, neighbor solicitation, who has fe80::5864:8aff:fef1:7a5, length 32
14:58:43.060648 IP6 fe80::7b5d:be4e:512:ac91 > ff02::1:fff1:7a5: ICMP6, neighbor solicitation, who has fe80::5864:8aff:fef1:7a5, length 32
14:58:44.065127 IP6 fe80::7b5d:be4e:512:ac91 > ff02::1:fff1:7a5: ICMP6, neighbor solicitation, who has fe80::5864:8aff:fef1:7a5, length 32
14:58:44.081477 IP6 fe80::7b5d:be4e:512:ac91 > sense: ICMP6, neighbor solicitation, who has sense, length 32
14:58:45.069324 IP6 fe80::7b5d:be4e:512:ac91 > ff02::1:fff1:7a5: ICMP6, neighbor solicitation, who has fe80::5864:8aff:fef1:7a5, length 32
14:58:45.094855 IP6 fe80::7b5d:be4e:512:ac91 > sense: ICMP6, neighbor solicitation, who has sense, length 32
14:58:46.073780 IP6 fe80::7b5d:be4e:512:ac91 > ff02::1:fff1:7a5: ICMP6, neighbor solicitation, who has fe80::5864:8aff:fef1:7a5, length 32
14:58:46.108110 IP6 fe80::7b5d:be4e:512:ac91 > sense: ICMP6, neighbor solicitation, who has sense, length 32
14:58:47.077911 IP6 fe80::7b5d:be4e:512:ac91 > ff02::1:fff1:7a5: ICMP6, neighbor solicitation, who has fe80::5864:8aff:fef1:7a5, length 32
14:58:48.082187 IP6 fe80::7b5d:be4e:512:ac91 > ff02::1:fff1:7a5: ICMP6, neighbor solicitation, who has fe80::5864:8aff:fef1:7a5, length 32
14:58:49.086716 IP6 fe80::7b5d:be4e:512:ac91 > ff02::1:fff1:7a5: ICMP6, neighbor solicitation, who has fe80::5864:8aff:fef1:7a5, length 32
14:58:50.090925 IP6 fe80::7b5d:be4e:512:ac91 > ff02::1:fff1:7a5: ICMP6, neighbor solicitation, who has fe80::5864:8aff:fef1:7a5, length 32
14:58:51.095030 IP6 fe80::7b5d:be4e:512:ac91 > ff02::1:fff1:7a5: ICMP6, neighbor solicitation, who has fe80::5864:8aff:fef1:7a5, length 32
14:58:52.099541 IP6 fe80::7b5d:be4e:512:ac91 > ff02::1:fff1:7a5: ICMP6, neighbor solicitation, who has fe80::5864:8aff:fef1:7a5, length 32
14:58:53.103758 IP6 fe80::7b5d:be4e:512:ac91 > ff02::1:fff1:7a5: ICMP6, neighbor solicitation, who has fe80::5864:8aff:fef1:7a5, length 32
14:58:54.108259 IP6 fe80::7b5d:be4e:512:ac91 > ff02::1:fff1:7a5: ICMP6, neighbor solicitation, who has fe80::5864:8aff:fef1:7a5, length 32
14:58:55.112493 IP6 fe80::7b5d:be4e:512:ac91 > ff02::1:fff1:7a5: ICMP6, neighbor solicitation, who has fe80::5864:8aff:fef1:7a5, length 32
14:58:55.112522 IP6 fe80::5864:8aff:fef1:7a5 > fe80::7b5d:be4e:512:ac91: ICMP6, neighbor advertisement, tgt is fe80::5864:8aff:fef1:7a5, length 32
14:58:56.116455 IP6 fe80::7b5d:be4e:512:ac91 > ff02::1:fff1:7a5: ICMP6, neighbor solicitation, who has fe80::5864:8aff:fef1:7a5, length 32
14:58:57.120820 IP6 fe80::7b5d:be4e:512:ac91 > ff02::1:fff1:7a5: ICMP6, neighbor solicitation, who has fe80::5864:8aff:fef1:7a5, length 32
14:58:58.125251 IP6 fe80::7b5d:be4e:512:ac91 > ff02::1:fff1:7a5: ICMP6, neighbor solicitation, who has fe80::5864:8aff:fef1:7a5, length 32
14:58:58.299692 IP6 fe80::5864:8aff:fef1:7a5 > ff02::1: ICMP6, router advertisement, length 104
14:58:59.129443 IP6 fe80::7b5d:be4e:512:ac91 > ff02::1:fff1:7a5: ICMP6, neighbor solicitation, who has fe80::5864:8aff:fef1:7a5, length 32
14:58:59.770107 IP6 fe80::5864:8aff:fef1:7a5 > fe80::7b5d:be4e:512:ac91: ICMP6, neighbor solicitation, who has fe80::7b5d:be4e:512:ac91, length 32
14:58:59.770716 IP6 fe80::7b5d:be4e:512:ac91 > fe80::5864:8aff:fef1:7a5: ICMP6, neighbor advertisement, tgt is fe80::7b5d:be4e:512:ac91, length 24
14:59:00.133635 IP6 fe80::7b5d:be4e:512:ac91 > ff02::1:fff1:7a5: ICMP6, neighbor solicitation, who has fe80::5864:8aff:fef1:7a5, length 32
^C
59 packets captured
4416 packets received by filter
0 packets dropped by kernel
root@sense:~ #

Additional context

I think this behavior was introduced in the update to 24.7 or 24.7.1. No other config changes at the time except the update.

Environment

OPNsense 24.7.2-amd64 on Proxmox 8.2.4, Kernel 6.8.12-1-pve
Virtual NIC is virtio with 4 queues and MTU 9000

The text was updated successfully, but these errors were encountered:

fichtner · 2024-08-22T14:42:32Z

It was introduced in 24.7.1 by FreeBSD and there is a ticket for it. #217

fichtner · 2024-08-22T15:27:36Z

didn't mean to imply this is not useful... after reading the details this is very helpful indeed, thanks!

fichtner · 2024-08-22T15:35:16Z

How to scrape that info from pfctl verbatim:

# pfctl -vvsInterfaces | grep -e '^[a-z]' -e Out6/Block

Crazyachmed · 2024-08-22T16:20:16Z

I only looked for bugs in the core-repo, so I didn't find it. Did you want to have the output of your command? Let me know if I can check anything else.

root@sense:~ # pfctl -vvsInterfaces | grep -e '^[a-z]' -e Out6/Block
No ALTQ support in kernel
ALTQ related functions disabled
all
        Out6/Block:  [ Packets: 0                  Bytes: 0                  ]
enc
        Out6/Block:  [ Packets: 0                  Bytes: 0                  ]
enc0
        Out6/Block:  [ Packets: 0                  Bytes: 0                  ]
lo
        Out6/Block:  [ Packets: 0                  Bytes: 0                  ]
lo0
        Out6/Block:  [ Packets: 4                  Bytes: 278                ]
pflog
        Out6/Block:  [ Packets: 0                  Bytes: 0                  ]
pflog0
        Out6/Block:  [ Packets: 0                  Bytes: 0                  ]
pfsync
        Out6/Block:  [ Packets: 0                  Bytes: 0                  ]
pfsync0 (skip)
        Out6/Block:  [ Packets: 0                  Bytes: 0                  ]
pppoe0
        Out6/Block:  [ Packets: 7                  Bytes: 1806               ]
vtnet0
        Out6/Block:  [ Packets: 17081              Bytes: 1221160            ]
vtnet1
        Out6/Block:  [ Packets: 0                  Bytes: 0                  ]
vtnet2
        Out6/Block:  [ Packets: 119                Bytes: 8616               ]
wg
        Out6/Block:  [ Packets: 0                  Bytes: 0                  ]
wg0
        Out6/Block:  [ Packets: 0                  Bytes: 0                  ]
wireguard
        Out6/Block:  [ Packets: 0                  Bytes: 0                  ]
root@sense:~ #

fichtner · 2024-08-22T19:00:39Z

You mentioned out6_block_packets so I had to chase down the raw output source for the FreeBSD ticket. I’m seeing the same. Testing against original 24.7 kernel now where the problem wasn’t present to see if there are significantly less drops visible.

Crazyachmed · 2024-08-23T10:29:25Z

I see the discussion in the FreeBSD ticket has sadly gone fubar. I have reverted to the last known good kernel now. If anyone else has this issue:

opnsense-update -kr 24.7

fichtner · 2024-08-23T10:37:02Z

If I give you two test kernels would you test them and tell me which one doesn't work as expected regarding this particular issue?

Crazyachmed · 2024-08-23T11:01:52Z

Sure, but also how to install them, please.

fichtner · 2024-08-23T11:04:24Z

Ok first things first let me make this the active ticket to track it since you've been so helpful already. ❤️

sjm42 · 2024-08-23T11:14:45Z

I can also try out some test kernels, if it would be helpful.

fichtner · 2024-08-23T13:03:00Z

Thanks all! Here are two kernels to test. This is not a blackbox test by any means but I will just say for now that on the surface that at least one of these should work as expected. If you could tell which one works and which one doesn't for this particular issue that would be very helpful to locate the problematic code.

# opnsense-update -zkr 24.7.1_11

or

# opnsense-update -zkr 24.7.1_12

Cheers,
Franco

meyergru · 2024-08-23T13:36:42Z

The testing method is nice, since I can reproduce this on a standalone Proxmox VM instance without annoying my wife.

With the standard kernel on 24.7.2, the behaviour was the same as reported (i.e. most NDs time out).
With kernel 24.7.1_11, every single ND was answered,
With kernel 24.7.1_12, every single ND was answered as well. I re-checked to make sure I really rebooted it.

Was that a double-blind study? ;-)

fichtner · 2024-08-23T13:58:11Z

This is at least an interesting turn of events although within the defined test parameters.

I'll throw in two more kernels:

# opnsense-update -zkr 24.7.1_13

and

# opnsense-update -zkr 24.7.1_14

Was that a double-blind study? ;-)

It's a treasure hunt! :)

fichtner · 2024-08-23T14:03:57Z

To double check we need this one anyway:

# opnsense-update -zkr 24.7.1_15

Happy hunting and thanks in advance.

meyergru · 2024-08-23T14:06:58Z

24.7.1_13 works
24.7.1_14 does not work
24.7.1_15 does not work

fichtner · 2024-08-23T14:11:48Z

Ok that would mean our winner as determined by @meyergru _14 is: https://cgit.freebsd.org/src/commit/?id=46755f5224

OpenBSD commit for reference openbsd/src@ef4bccd7509e

Independent confirmation is appreciated.

Crazyachmed · 2024-08-23T14:20:20Z

_11 and _12 work for me as well, for some reason I can't download the other ones:

root@sense:~ # opnsense-update -zkr 24.7.1_13
Fetching kernel-24.7.1_13-amd64.txz: ..[fetch: https://mirror.fra10.de.leaseweb.net/opnsense/FreeBSD:14:amd64/snapshots/sets/kernel-24.7.1_13-amd64.txz.sig: Not Found] failed, no signature found
root@sense:~ # opnsense-update -zkr 24.7.1_14
Fetching kernel-24.7.1_14-amd64.txz: ..[fetch: https://mirror.fra10.de.leaseweb.net/opnsense/FreeBSD:14:amd64/snapshots/sets/kernel-24.7.1_14-amd64.txz.sig: Not Found] failed, no signature found
root@sense:~ # opnsense-update -zkr 24.7.1_15
Fetching kernel-24.7.1_15-amd64.txz: ..[fetch: https://mirror.fra10.de.leaseweb.net/opnsense/FreeBSD:14:amd64/snapshots/sets/kernel-24.7.1_15-amd64.txz.sig: Not Found] failed, no signature found
root@sense:~ # ping mirror.fra10.de.leaseweb.net
PING(56=40+8+8 bytes) 2a04:9740:b2:74fc:5864:8aff:fef1:7a5 --> 2a00:c98:2030:a034::21
16 bytes from 2a00:c98:2030:a034::21, icmp_seq=0 hlim=54 time=16.692 ms
16 bytes from 2a00:c98:2030:a034::21, icmp_seq=1 hlim=54 time=17.048 ms
16 bytes from 2a00:c98:2030:a034::21, icmp_seq=2 hlim=54 time=17.085 ms
^C
--- mirror.fra10.de.leaseweb.net ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 16.692/16.942/17.085/0.177 ms
root@sense:~ #

fichtner · 2024-08-23T14:20:44Z

@Crazyachmed not synced to leaseweb yet, try default mirror

Crazyachmed · 2024-08-23T14:25:29Z

24.7		ok
24.7.2		broken
24.7.1_11	ok
24.7.1_12	ok
24.7.1_13	ok
24.7.1_14	broken
24.7.1_15	broken

sjm42 · 2024-08-23T14:33:38Z

Uh oh, this has progressed even more than I imagined.

Anyway, I just tested 24.7.1_11 and 24.7.1_12 and had interesting results.

with 24.7.1_11 my provider's router is asking for ND only every 45 seconds or so, but with 24.7.1_12 the interval is dropped into 25 seconds or so. Do ND reply packets have some kind of lifetime indication in them? This looks funny.
with 24.7.1_11 every ND request is answered immediately by opnsense, but with 24.7.1_12 only usually -- sometimes the unicast requests get ignored by opnsense but multicast requests are answered normally.

So, some packet capures follow after my "analysis". I have grouped the request+response pairs with surrounding newlines for easier reading.

with 24.7.1_11 kernel:

17:11:15.132422 IP6 fe80::1afd:74ff:fec1:2acd > fe80::2e2:69ff:fe65:3a25: ICMP6, neighbor solicitation, who has fe80::2e2:69ff:fe65:3a25, length 32
17:11:15.132451 IP6 fe80::2e2:69ff:fe65:3a25 > fe80::1afd:74ff:fec1:2acd: ICMP6, neighbor advertisement, tgt is fe80::2e2:69ff:fe65:3a25, length 24

17:12:00.492380 IP6 fe80::1afd:74ff:fec1:2acd > fe80::2e2:69ff:fe65:3a25: ICMP6, neighbor solicitation, who has fe80::2e2:69ff:fe65:3a25, length 32
17:12:00.492427 IP6 fe80::2e2:69ff:fe65:3a25 > fe80::1afd:74ff:fec1:2acd: ICMP6, neighbor advertisement, tgt is fe80::2e2:69ff:fe65:3a25, length 24

17:12:44.722345 IP6 fe80::1afd:74ff:fec1:2acd > fe80::2e2:69ff:fe65:3a25: ICMP6, neighbor solicitation, who has fe80::2e2:69ff:fe65:3a25, length 32
17:12:44.722373 IP6 fe80::2e2:69ff:fe65:3a25 > fe80::1afd:74ff:fec1:2acd: ICMP6, neighbor advertisement, tgt is fe80::2e2:69ff:fe65:3a25, length 24

17:13:29.602297 IP6 fe80::1afd:74ff:fec1:2acd > fe80::2e2:69ff:fe65:3a25: ICMP6, neighbor solicitation, who has fe80::2e2:69ff:fe65:3a25, length 32
17:13:29.602337 IP6 fe80::2e2:69ff:fe65:3a25 > fe80::1afd:74ff:fec1:2acd: ICMP6, neighbor advertisement, tgt is fe80::2e2:69ff:fe65:3a25, length 24

17:14:16.322287 IP6 fe80::1afd:74ff:fec1:2acd > fe80::2e2:69ff:fe65:3a25: ICMP6, neighbor solicitation, who has fe80::2e2:69ff:fe65:3a25, length 32
17:14:16.322313 IP6 fe80::2e2:69ff:fe65:3a25 > fe80::1afd:74ff:fec1:2acd: ICMP6, neighbor advertisement, tgt is fe80::2e2:69ff:fe65:3a25, length 24

17:15:00.962258 IP6 fe80::1afd:74ff:fec1:2acd > fe80::2e2:69ff:fe65:3a25: ICMP6, neighbor solicitation, who has fe80::2e2:69ff:fe65:3a25, length 32
17:15:00.962293 IP6 fe80::2e2:69ff:fe65:3a25 > fe80::1afd:74ff:fec1:2acd: ICMP6, neighbor advertisement, tgt is fe80::2e2:69ff:fe65:3a25, length 24

with 24.7.1_12 kernel:

17:20:15.842334 IP6 fe80::1afd:74ff:fec1:2acd > fe80::2e2:69ff:fe65:3a25: ICMP6, neighbor solicitation, who has fe80::2e2:69ff:fe65:3a25, length 32
17:20:15.842367 IP6 fe80::2e2:69ff:fe65:3a25 > fe80::1afd:74ff:fec1:2acd: ICMP6, neighbor advertisement, tgt is fe80::2e2:69ff:fe65:3a25, length 24

17:20:39.042394 IP6 fe80::1afd:74ff:fec1:2acd > fe80::2e2:69ff:fe65:3a25: ICMP6, neighbor solicitation, who has fe80::2e2:69ff:fe65:3a25, length 32
17:20:40.082287 IP6 fe80::1afd:74ff:fec1:2acd > fe80::2e2:69ff:fe65:3a25: ICMP6, neighbor solicitation, who has fe80::2e2:69ff:fe65:3a25, length 32
17:20:41.122359 IP6 fe80::1afd:74ff:fec1:2acd > fe80::2e2:69ff:fe65:3a25: ICMP6, neighbor solicitation, who has fe80::2e2:69ff:fe65:3a25, length 32

17:20:42.260298 IP6 fe80::1afd:74ff:fec1:2acd > ff02::1:ff65:3a25: ICMP6, neighbor solicitation, who has fe80::2e2:69ff:fe65:3a25, length 32
17:20:42.260332 IP6 fe80::2e2:69ff:fe65:3a25 > fe80::1afd:74ff:fec1:2acd: ICMP6, neighbor advertisement, tgt is fe80::2e2:69ff:fe65:3a25, length 32

17:21:05.842621 IP6 fe80::1afd:74ff:fec1:2acd > fe80::2e2:69ff:fe65:3a25: ICMP6, neighbor solicitation, who has fe80::2e2:69ff:fe65:3a25, length 32
17:21:05.842644 IP6 fe80::2e2:69ff:fe65:3a25 > fe80::1afd:74ff:fec1:2acd: ICMP6, neighbor advertisement, tgt is fe80::2e2:69ff:fe65:3a25, length 24

17:21:30.722365 IP6 fe80::1afd:74ff:fec1:2acd > fe80::2e2:69ff:fe65:3a25: ICMP6, neighbor solicitation, who has fe80::2e2:69ff:fe65:3a25, length 32
17:21:30.722408 IP6 fe80::2e2:69ff:fe65:3a25 > fe80::1afd:74ff:fec1:2acd: ICMP6, neighbor advertisement, tgt is fe80::2e2:69ff:fe65:3a25, length 24

17:21:53.922384 IP6 fe80::1afd:74ff:fec1:2acd > fe80::2e2:69ff:fe65:3a25: ICMP6, neighbor solicitation, who has fe80::2e2:69ff:fe65:3a25, length 32
17:21:53.922425 IP6 fe80::2e2:69ff:fe65:3a25 > fe80::1afd:74ff:fec1:2acd: ICMP6, neighbor advertisement, tgt is fe80::2e2:69ff:fe65:3a25, length 24

17:22:18.242352 IP6 fe80::1afd:74ff:fec1:2acd > fe80::2e2:69ff:fe65:3a25: ICMP6, neighbor solicitation, who has fe80::2e2:69ff:fe65:3a25, length 32
17:22:18.242384 IP6 fe80::2e2:69ff:fe65:3a25 > fe80::1afd:74ff:fec1:2acd: ICMP6, neighbor advertisement, tgt is fe80::2e2:69ff:fe65:3a25, length 24

17:22:42.802400 IP6 fe80::1afd:74ff:fec1:2acd > fe80::2e2:69ff:fe65:3a25: ICMP6, neighbor solicitation, who has fe80::2e2:69ff:fe65:3a25, length 32
17:22:42.802438 IP6 fe80::2e2:69ff:fe65:3a25 > fe80::1afd:74ff:fec1:2acd: ICMP6, neighbor advertisement, tgt is fe80::2e2:69ff:fe65:3a25, length 24

17:23:06.322401 IP6 fe80::1afd:74ff:fec1:2acd > fe80::2e2:69ff:fe65:3a25: ICMP6, neighbor solicitation, who has fe80::2e2:69ff:fe65:3a25, length 32
17:23:06.322433 IP6 fe80::2e2:69ff:fe65:3a25 > fe80::1afd:74ff:fec1:2acd: ICMP6, neighbor advertisement, tgt is fe80::2e2:69ff:fe65:3a25, length 24

17:23:30.242364 IP6 fe80::1afd:74ff:fec1:2acd > fe80::2e2:69ff:fe65:3a25: ICMP6, neighbor solicitation, who has fe80::2e2:69ff:fe65:3a25, length 32
17:23:30.242408 IP6 fe80::2e2:69ff:fe65:3a25 > fe80::1afd:74ff:fec1:2acd: ICMP6, neighbor advertisement, tgt is fe80::2e2:69ff:fe65:3a25, length 24

17:23:53.602303 IP6 fe80::1afd:74ff:fec1:2acd > fe80::2e2:69ff:fe65:3a25: ICMP6, neighbor solicitation, who has fe80::2e2:69ff:fe65:3a25, length 32
17:23:53.602349 IP6 fe80::2e2:69ff:fe65:3a25 > fe80::1afd:74ff:fec1:2acd: ICMP6, neighbor advertisement, tgt is fe80::2e2:69ff:fe65:3a25, length 24

17:24:17.122308 IP6 fe80::1afd:74ff:fec1:2acd > fe80::2e2:69ff:fe65:3a25: ICMP6, neighbor solicitation, who has fe80::2e2:69ff:fe65:3a25, length 32
17:24:17.122343 IP6 fe80::2e2:69ff:fe65:3a25 > fe80::1afd:74ff:fec1:2acd: ICMP6, neighbor advertisement, tgt is fe80::2e2:69ff:fe65:3a25, length 24

17:24:41.682352 IP6 fe80::1afd:74ff:fec1:2acd > fe80::2e2:69ff:fe65:3a25: ICMP6, neighbor solicitation, who has fe80::2e2:69ff:fe65:3a25, length 32
17:24:42.723410 IP6 fe80::1afd:74ff:fec1:2acd > fe80::2e2:69ff:fe65:3a25: ICMP6, neighbor solicitation, who has fe80::2e2:69ff:fe65:3a25, length 32
17:24:43.762365 IP6 fe80::1afd:74ff:fec1:2acd > fe80::2e2:69ff:fe65:3a25: ICMP6, neighbor solicitation, who has fe80::2e2:69ff:fe65:3a25, length 32

17:24:45.381560 IP6 fe80::1afd:74ff:fec1:2acd > ff02::1:ff65:3a25: ICMP6, neighbor solicitation, who has fe80::2e2:69ff:fe65:3a25, length 32
17:24:45.381599 IP6 fe80::2e2:69ff:fe65:3a25 > fe80::1afd:74ff:fec1:2acd: ICMP6, neighbor advertisement, tgt is fe80::2e2:69ff:fe65:3a25, length 32

17:25:08.972302 IP6 fe80::1afd:74ff:fec1:2acd > fe80::2e2:69ff:fe65:3a25: ICMP6, neighbor solicitation, who has fe80::2e2:69ff:fe65:3a25, length 32
17:25:10.002357 IP6 fe80::1afd:74ff:fec1:2acd > fe80::2e2:69ff:fe65:3a25: ICMP6, neighbor solicitation, who has fe80::2e2:69ff:fe65:3a25, length 32
17:25:11.052382 IP6 fe80::1afd:74ff:fec1:2acd > fe80::2e2:69ff:fe65:3a25: ICMP6, neighbor solicitation, who has fe80::2e2:69ff:fe65:3a25, length 32

17:25:12.114988 IP6 fe80::1afd:74ff:fec1:2acd > ff02::1:ff65:3a25: ICMP6, neighbor solicitation, who has fe80::2e2:69ff:fe65:3a25, length 32
17:25:12.115019 IP6 fe80::2e2:69ff:fe65:3a25 > fe80::1afd:74ff:fec1:2acd: ICMP6, neighbor advertisement, tgt is fe80::2e2:69ff:fe65:3a25, length 32

sjm42 · 2024-08-23T15:15:25Z

Okay, now I tested with three more kernels, 24.7.1_13-15.

With kernel 24.7.1_13

sometimes it takes two ND requests for opnsense to respond, sometimes more
it looks like the operator's router will switch to multicast after 3 tries with unicast
the multicast request is always responded to

17:37:11.310941 IP6 fe80::1afd:74ff:fec1:2acd > fe80::2e2:69ff:fe65:3a25: ICMP6, neighbor solicitation, who has fe80::2e2:69ff:fe65:3a25, length 32
17:37:11.310977 IP6 fe80::2e2:69ff:fe65:3a25 > fe80::1afd:74ff:fec1:2acd: ICMP6, neighbor advertisement, tgt is fe80::2e2:69ff:fe65:3a25, length 24

17:37:56.670956 IP6 fe80::1afd:74ff:fec1:2acd > fe80::2e2:69ff:fe65:3a25: ICMP6, neighbor solicitation, who has fe80::2e2:69ff:fe65:3a25, length 32
17:37:57.710993 IP6 fe80::1afd:74ff:fec1:2acd > fe80::2e2:69ff:fe65:3a25: ICMP6, neighbor solicitation, who has fe80::2e2:69ff:fe65:3a25, length 32
17:37:57.711026 IP6 fe80::2e2:69ff:fe65:3a25 > fe80::1afd:74ff:fec1:2acd: ICMP6, neighbor advertisement, tgt is fe80::2e2:69ff:fe65:3a25, length 24

17:38:43.220843 IP6 fe80::1afd:74ff:fec1:2acd > fe80::2e2:69ff:fe65:3a25: ICMP6, neighbor solicitation, who has fe80::2e2:69ff:fe65:3a25, length 32
17:38:44.260887 IP6 fe80::1afd:74ff:fec1:2acd > fe80::2e2:69ff:fe65:3a25: ICMP6, neighbor solicitation, who has fe80::2e2:69ff:fe65:3a25, length 32
17:38:45.310845 IP6 fe80::1afd:74ff:fec1:2acd > fe80::2e2:69ff:fe65:3a25: ICMP6, neighbor solicitation, who has fe80::2e2:69ff:fe65:3a25, length 32

17:38:46.358983 IP6 fe80::1afd:74ff:fec1:2acd > ff02::1:ff65:3a25: ICMP6, neighbor solicitation, who has fe80::2e2:69ff:fe65:3a25, length 32
17:38:46.359026 IP6 fe80::2e2:69ff:fe65:3a25 > fe80::1afd:74ff:fec1:2acd: ICMP6, neighbor advertisement, tgt is fe80::2e2:69ff:fe65:3a25, length 32

17:39:31.620880 IP6 fe80::1afd:74ff:fec1:2acd > fe80::2e2:69ff:fe65:3a25: ICMP6, neighbor solicitation, who has fe80::2e2:69ff:fe65:3a25, length 32
17:39:31.620918 IP6 fe80::2e2:69ff:fe65:3a25 > fe80::1afd:74ff:fec1:2acd: ICMP6, neighbor advertisement, tgt is fe80::2e2:69ff:fe65:3a25, length 24

17:40:17.062090 IP6 fe80::1afd:74ff:fec1:2acd > fe80::2e2:69ff:fe65:3a25: ICMP6, neighbor solicitation, who has fe80::2e2:69ff:fe65:3a25, length 32
17:40:17.062117 IP6 fe80::2e2:69ff:fe65:3a25 > fe80::1afd:74ff:fec1:2acd: ICMP6, neighbor advertisement, tgt is fe80::2e2:69ff:fe65:3a25, length 24

17:41:02.590814 IP6 fe80::1afd:74ff:fec1:2acd > fe80::2e2:69ff:fe65:3a25: ICMP6, neighbor solicitation, who has fe80::2e2:69ff:fe65:3a25, length 32
17:41:02.590861 IP6 fe80::2e2:69ff:fe65:3a25 > fe80::1afd:74ff:fec1:2acd: ICMP6, neighbor advertisement, tgt is fe80::2e2:69ff:fe65:3a25, length 24

17:41:25.380777 IP6 fe80::1afd:74ff:fec1:2acd > fe80::2e2:69ff:fe65:3a25: ICMP6, neighbor solicitation, who has fe80::2e2:69ff:fe65:3a25, length 32
17:41:26.430774 IP6 fe80::1afd:74ff:fec1:2acd > fe80::2e2:69ff:fe65:3a25: ICMP6, neighbor solicitation, who has fe80::2e2:69ff:fe65:3a25, length 32
17:41:27.460755 IP6 fe80::1afd:74ff:fec1:2acd > fe80::2e2:69ff:fe65:3a25: ICMP6, neighbor solicitation, who has fe80::2e2:69ff:fe65:3a25, length 32

17:41:29.156937 IP6 fe80::1afd:74ff:fec1:2acd > ff02::1:ff65:3a25: ICMP6, neighbor solicitation, who has fe80::2e2:69ff:fe65:3a25, length 32
17:41:29.156977 IP6 fe80::2e2:69ff:fe65:3a25 > fe80::1afd:74ff:fec1:2acd: ICMP6, neighbor advertisement, tgt is fe80::2e2:69ff:fe65:3a25, length 32

17:41:51.860736 IP6 fe80::1afd:74ff:fec1:2acd > fe80::2e2:69ff:fe65:3a25: ICMP6, neighbor solicitation, who has fe80::2e2:69ff:fe65:3a25, length 32
17:41:51.860771 IP6 fe80::2e2:69ff:fe65:3a25 > fe80::1afd:74ff:fec1:2acd: ICMP6, neighbor advertisement, tgt is fe80::2e2:69ff:fe65:3a25, length 24

17:42:16.242512 IP6 fe80::1afd:74ff:fec1:2acd > fe80::2e2:69ff:fe65:3a25: ICMP6, neighbor solicitation, who has fe80::2e2:69ff:fe65:3a25, length 32
17:42:17.283965 IP6 fe80::1afd:74ff:fec1:2acd > fe80::2e2:69ff:fe65:3a25: ICMP6, neighbor solicitation, who has fe80::2e2:69ff:fe65:3a25, length 32
17:42:18.322711 IP6 fe80::1afd:74ff:fec1:2acd > fe80::2e2:69ff:fe65:3a25: ICMP6, neighbor solicitation, who has fe80::2e2:69ff:fe65:3a25, length 32

17:42:19.618006 IP6 fe80::1afd:74ff:fec1:2acd > ff02::1:ff65:3a25: ICMP6, neighbor solicitation, who has fe80::2e2:69ff:fe65:3a25, length 32
17:42:19.618070 IP6 fe80::2e2:69ff:fe65:3a25 > fe80::1afd:74ff:fec1:2acd: ICMP6, neighbor advertisement, tgt is fe80::2e2:69ff:fe65:3a25, length 32

17:42:42.732676 IP6 fe80::1afd:74ff:fec1:2acd > fe80::2e2:69ff:fe65:3a25: ICMP6, neighbor solicitation, who has fe80::2e2:69ff:fe65:3a25, length 32
17:42:43.772497 IP6 fe80::1afd:74ff:fec1:2acd > fe80::2e2:69ff:fe65:3a25: ICMP6, neighbor solicitation, who has fe80::2e2:69ff:fe65:3a25, length 32
17:42:44.812494 IP6 fe80::1afd:74ff:fec1:2acd > fe80::2e2:69ff:fe65:3a25: ICMP6, neighbor solicitation, who has fe80::2e2:69ff:fe65:3a25, length 32

17:42:46.060878 IP6 fe80::1afd:74ff:fec1:2acd > ff02::1:ff65:3a25: ICMP6, neighbor solicitation, who has fe80::2e2:69ff:fe65:3a25, length 32
17:42:46.060916 IP6 fe80::2e2:69ff:fe65:3a25 > fe80::1afd:74ff:fec1:2acd: ICMP6, neighbor advertisement, tgt is fe80::2e2:69ff:fe65:3a25, length 32

With kernel 24.7.1_14

not much different from _13 but sometimes opnsense is responding to the 3rd ND unicast request
might be just random variance compared to the previous

17:49:09.202828 IP6 fe80::1afd:74ff:fec1:2acd > fe80::2e2:69ff:fe65:3a25: ICMP6, neighbor solicitation, who has fe80::2e2:69ff:fe65:3a25, length 32
17:49:10.242930 IP6 fe80::1afd:74ff:fec1:2acd > fe80::2e2:69ff:fe65:3a25: ICMP6, neighbor solicitation, who has fe80::2e2:69ff:fe65:3a25, length 32
17:49:10.242989 IP6 fe80::2e2:69ff:fe65:3a25 > fe80::1afd:74ff:fec1:2acd: ICMP6, neighbor advertisement, tgt is fe80::2e2:69ff:fe65:3a25, length 24

17:49:50.092819 IP6 fe80::1afd:74ff:fec1:2acd > fe80::2e2:69ff:fe65:3a25: ICMP6, neighbor solicitation, who has fe80::2e2:69ff:fe65:3a25, length 32
17:49:51.122826 IP6 fe80::1afd:74ff:fec1:2acd > fe80::2e2:69ff:fe65:3a25: ICMP6, neighbor solicitation, who has fe80::2e2:69ff:fe65:3a25, length 32
17:49:52.162828 IP6 fe80::1afd:74ff:fec1:2acd > fe80::2e2:69ff:fe65:3a25: ICMP6, neighbor solicitation, who has fe80::2e2:69ff:fe65:3a25, length 32

17:49:53.214893 IP6 fe80::1afd:74ff:fec1:2acd > ff02::1:ff65:3a25: ICMP6, neighbor solicitation, who has fe80::2e2:69ff:fe65:3a25, length 32
17:49:53.214932 IP6 fe80::2e2:69ff:fe65:3a25 > fe80::1afd:74ff:fec1:2acd: ICMP6, neighbor advertisement, tgt is fe80::2e2:69ff:fe65:3a25, length 32

17:50:32.802831 IP6 fe80::1afd:74ff:fec1:2acd > fe80::2e2:69ff:fe65:3a25: ICMP6, neighbor solicitation, who has fe80::2e2:69ff:fe65:3a25, length 32
17:50:33.842799 IP6 fe80::1afd:74ff:fec1:2acd > fe80::2e2:69ff:fe65:3a25: ICMP6, neighbor solicitation, who has fe80::2e2:69ff:fe65:3a25, length 32
17:50:34.882821 IP6 fe80::1afd:74ff:fec1:2acd > fe80::2e2:69ff:fe65:3a25: ICMP6, neighbor solicitation, who has fe80::2e2:69ff:fe65:3a25, length 32
17:50:34.882853 IP6 fe80::2e2:69ff:fe65:3a25 > fe80::1afd:74ff:fec1:2acd: ICMP6, neighbor advertisement, tgt is fe80::2e2:69ff:fe65:3a25, length 24

17:51:14.642789 IP6 fe80::1afd:74ff:fec1:2acd > fe80::2e2:69ff:fe65:3a25: ICMP6, neighbor solicitation, who has fe80::2e2:69ff:fe65:3a25, length 32
17:51:15.682799 IP6 fe80::1afd:74ff:fec1:2acd > fe80::2e2:69ff:fe65:3a25: ICMP6, neighbor solicitation, who has fe80::2e2:69ff:fe65:3a25, length 32
17:51:16.722803 IP6 fe80::1afd:74ff:fec1:2acd > fe80::2e2:69ff:fe65:3a25: ICMP6, neighbor solicitation, who has fe80::2e2:69ff:fe65:3a25, length 32
17:51:16.722836 IP6 fe80::2e2:69ff:fe65:3a25 > fe80::1afd:74ff:fec1:2acd: ICMP6, neighbor advertisement, tgt is fe80::2e2:69ff:fe65:3a25, length 24

17:51:49.843565 IP6 fe80::1afd:74ff:fec1:2acd > fe80::2e2:69ff:fe65:3a25: ICMP6, neighbor solicitation, who has fe80::2e2:69ff:fe65:3a25, length 32
17:51:49.843610 IP6 fe80::2e2:69ff:fe65:3a25 > fe80::1afd:74ff:fec1:2acd: ICMP6, neighbor advertisement, tgt is fe80::2e2:69ff:fe65:3a25, length 24

17:52:23.922778 IP6 fe80::1afd:74ff:fec1:2acd > fe80::2e2:69ff:fe65:3a25: ICMP6, neighbor solicitation, who has fe80::2e2:69ff:fe65:3a25, length 32
17:52:23.922802 IP6 fe80::2e2:69ff:fe65:3a25 > fe80::1afd:74ff:fec1:2acd: ICMP6, neighbor advertisement, tgt is fe80::2e2:69ff:fe65:3a25, length 24

17:52:57.282713 IP6 fe80::1afd:74ff:fec1:2acd > fe80::2e2:69ff:fe65:3a25: ICMP6, neighbor solicitation, who has fe80::2e2:69ff:fe65:3a25, length 32
17:52:57.282744 IP6 fe80::2e2:69ff:fe65:3a25 > fe80::1afd:74ff:fec1:2acd: ICMP6, neighbor advertisement, tgt is fe80::2e2:69ff:fe65:3a25, length 24

17:53:30.172760 IP6 fe80::1afd:74ff:fec1:2acd > fe80::2e2:69ff:fe65:3a25: ICMP6, neighbor solicitation, who has fe80::2e2:69ff:fe65:3a25, length 32
17:53:30.172801 IP6 fe80::2e2:69ff:fe65:3a25 > fe80::1afd:74ff:fec1:2acd: ICMP6, neighbor advertisement, tgt is fe80::2e2:69ff:fe65:3a25, length 24

17:54:03.934096 IP6 fe80::1afd:74ff:fec1:2acd > fe80::2e2:69ff:fe65:3a25: ICMP6, neighbor solicitation, who has fe80::2e2:69ff:fe65:3a25, length 32
17:54:03.934129 IP6 fe80::2e2:69ff:fe65:3a25 > fe80::1afd:74ff:fec1:2acd: ICMP6, neighbor advertisement, tgt is fe80::2e2:69ff:fe65:3a25, length 24

With kernel 24.7.1_15

this one looks like the worst of the three versions tested
probably caused IPv6 outages -- did not check that, I was just keeping an eye on ICMPv6...
funnily enough, after ND problems in the beginning, then it began running smoothly for several minutes

17:57:33.202721 IP6 fe80::1afd:74ff:fec1:2acd > fe80::2e2:69ff:fe65:3a25: ICMP6, neighbor solicitation, who has fe80::2e2:69ff:fe65:3a25, length 32
17:57:34.242759 IP6 fe80::1afd:74ff:fec1:2acd > fe80::2e2:69ff:fe65:3a25: ICMP6, neighbor solicitation, who has fe80::2e2:69ff:fe65:3a25, length 32
17:57:35.282715 IP6 fe80::1afd:74ff:fec1:2acd > fe80::2e2:69ff:fe65:3a25: ICMP6, neighbor solicitation, who has fe80::2e2:69ff:fe65:3a25, length 32

17:57:36.564631 IP6 fe80::1afd:74ff:fec1:2acd > ff02::1:ff65:3a25: ICMP6, neighbor solicitation, who has fe80::2e2:69ff:fe65:3a25, length 32
17:57:37.603240 IP6 fe80::1afd:74ff:fec1:2acd > ff02::1:ff65:3a25: ICMP6, neighbor solicitation, who has fe80::2e2:69ff:fe65:3a25, length 32
17:57:38.643314 IP6 fe80::1afd:74ff:fec1:2acd > ff02::1:ff65:3a25: ICMP6, neighbor solicitation, who has fe80::2e2:69ff:fe65:3a25, length 32
17:57:39.953980 IP6 fe80::1afd:74ff:fec1:2acd > ff02::1:ff65:3a25: ICMP6, neighbor solicitation, who has fe80::2e2:69ff:fe65:3a25, length 32
17:57:40.964001 IP6 fe80::1afd:74ff:fec1:2acd > ff02::1:ff65:3a25: ICMP6, neighbor solicitation, who has fe80::2e2:69ff:fe65:3a25, length 32

17:57:47.043724 IP6 fe80::1afd:74ff:fec1:2acd > fe80::2e2:69ff:fe65:3a25: ICMP6, neighbor solicitation, who has fe80::2e2:69ff:fe65:3a25, length 32
17:57:48.092692 IP6 fe80::1afd:74ff:fec1:2acd > fe80::2e2:69ff:fe65:3a25: ICMP6, neighbor solicitation, who has fe80::2e2:69ff:fe65:3a25, length 32
17:57:49.132688 IP6 fe80::1afd:74ff:fec1:2acd > fe80::2e2:69ff:fe65:3a25: ICMP6, neighbor solicitation, who has fe80::2e2:69ff:fe65:3a25, length 32

17:57:50.237147 IP6 fe80::1afd:74ff:fec1:2acd > ff02::1:ff65:3a25: ICMP6, neighbor solicitation, who has fe80::2e2:69ff:fe65:3a25, length 32
17:57:51.283295 IP6 fe80::1afd:74ff:fec1:2acd > ff02::1:ff65:3a25: ICMP6, neighbor solicitation, who has fe80::2e2:69ff:fe65:3a25, length 32
17:57:52.333445 IP6 fe80::1afd:74ff:fec1:2acd > ff02::1:ff65:3a25: ICMP6, neighbor solicitation, who has fe80::2e2:69ff:fe65:3a25, length 32
17:57:53.438668 IP6 fe80::1afd:74ff:fec1:2acd > ff02::1:ff65:3a25: ICMP6, neighbor solicitation, who has fe80::2e2:69ff:fe65:3a25, length 32
17:57:54.483283 IP6 fe80::1afd:74ff:fec1:2acd > ff02::1:ff65:3a25: ICMP6, neighbor solicitation, who has fe80::2e2:69ff:fe65:3a25, length 32
17:57:55.533405 IP6 fe80::1afd:74ff:fec1:2acd > ff02::1:ff65:3a25: ICMP6, neighbor solicitation, who has fe80::2e2:69ff:fe65:3a25, length 32
17:57:56.589513 IP6 fe80::1afd:74ff:fec1:2acd > ff02::1:ff65:3a25: ICMP6, neighbor solicitation, who has fe80::2e2:69ff:fe65:3a25, length 32

17:57:57.603088 IP6 fe80::1afd:74ff:fec1:2acd > ff02::1:ff65:3a25: ICMP6, neighbor solicitation, who has fe80::2e2:69ff:fe65:3a25, length 32
17:57:57.603124 IP6 fe80::2e2:69ff:fe65:3a25 > fe80::1afd:74ff:fec1:2acd: ICMP6, neighbor advertisement, tgt is fe80::2e2:69ff:fe65:3a25, length 32

17:58:24.973820 IP6 fe80::1afd:74ff:fec1:2acd > fe80::2e2:69ff:fe65:3a25: ICMP6, neighbor solicitation, who has fe80::2e2:69ff:fe65:3a25, length 32
17:58:24.973864 IP6 fe80::2e2:69ff:fe65:3a25 > fe80::1afd:74ff:fec1:2acd: ICMP6, neighbor advertisement, tgt is fe80::2e2:69ff:fe65:3a25, length 24

17:58:51.772747 IP6 fe80::1afd:74ff:fec1:2acd > fe80::2e2:69ff:fe65:3a25: ICMP6, neighbor solicitation, who has fe80::2e2:69ff:fe65:3a25, length 32
17:58:51.772793 IP6 fe80::2e2:69ff:fe65:3a25 > fe80::1afd:74ff:fec1:2acd: ICMP6, neighbor advertisement, tgt is fe80::2e2:69ff:fe65:3a25, length 24

17:59:18.482734 IP6 fe80::1afd:74ff:fec1:2acd > fe80::2e2:69ff:fe65:3a25: ICMP6, neighbor solicitation, who has fe80::2e2:69ff:fe65:3a25, length 32
17:59:18.482775 IP6 fe80::2e2:69ff:fe65:3a25 > fe80::1afd:74ff:fec1:2acd: ICMP6, neighbor advertisement, tgt is fe80::2e2:69ff:fe65:3a25, length 24

17:59:45.212752 IP6 fe80::1afd:74ff:fec1:2acd > fe80::2e2:69ff:fe65:3a25: ICMP6, neighbor solicitation, who has fe80::2e2:69ff:fe65:3a25, length 32
17:59:45.212788 IP6 fe80::2e2:69ff:fe65:3a25 > fe80::1afd:74ff:fec1:2acd: ICMP6, neighbor advertisement, tgt is fe80::2e2:69ff:fe65:3a25, length 24

18:00:12.002773 IP6 fe80::1afd:74ff:fec1:2acd > fe80::2e2:69ff:fe65:3a25: ICMP6, neighbor solicitation, who has fe80::2e2:69ff:fe65:3a25, length 32
18:00:12.002820 IP6 fe80::2e2:69ff:fe65:3a25 > fe80::1afd:74ff:fec1:2acd: ICMP6, neighbor advertisement, tgt is fe80::2e2:69ff:fe65:3a25, length 24

18:00:39.292748 IP6 fe80::1afd:74ff:fec1:2acd > fe80::2e2:69ff:fe65:3a25: ICMP6, neighbor solicitation, who has fe80::2e2:69ff:fe65:3a25, length 32
18:00:39.292786 IP6 fe80::2e2:69ff:fe65:3a25 > fe80::1afd:74ff:fec1:2acd: ICMP6, neighbor advertisement, tgt is fe80::2e2:69ff:fe65:3a25, length 24

doktornotor · 2024-08-23T17:48:56Z

With everyone at home being very much p***ed off with the constant reboots, I'll make it short - mostly matches what @sjm42 posted anyway, Here's a completely unscientific ND_NEIGHBOR_SOLICIT / ND_NEIGHBOR_ADVERT experience rating, on a scale from 1 (garbage) to 10 (works perfectly fine}:

_11 + _12 - 10/10
_13 - 7/10
_14 - 5/10
_15 - 1/10

Will try to do something more extensive tomorrow. Now - 🍺 🍻 badly needed.

Crazyachmed · 2024-08-23T18:50:44Z

I've checked my WAN connection after looking at sjm42's analysis, but my internet is GPON fibre + PPP, so all of that fun stuff is negotiated beforehand and exactly once. The only thing I see are Prefix (Router) Advertisements every ~17 minutes.

I guess that also explains the limited impact on my setup.

I currently run the 24.7.1_13 kernel, is that "okay" or should I revert to one of the other versions or 24.7?

doktornotor · 2024-08-23T18:56:53Z

Well there's no need to revert if it's working for you. 😉

fichtner · 2024-08-23T19:18:28Z

Use a kernel that works from the _1x series for now. That's the closes to 24.7.2 without the issue.

fichtner · 2024-08-24T03:15:57Z

@sjm42 thanks, your analysis in particular is interesting suggesting this gets increasingly worse with each commit, but as I understand not all commits affect IPv6 directly. _11 is without the SA commits. _12 is the first commit and so on... I'll look at the code again with this in mind.

Note to self:

_11 -> no SA
_12 -> 7f1f57e
_13 -> c21004c
_14 -> ee1b712
_15 -> 9b2e3cf

fichtner · 2024-08-24T03:56:46Z

Ok so _12 introduces state matching for ND_NEIGHBOR_SOLICIT/ND_NEIGHBOR_ADVERT that wasn't there before. For some reason I also can't disable this at first glance from the pf ruleset. Maybe it's not an explicit icmpv6 rule doing that but what I want to try is to build a latest kernel with all the SA code and remove ND state creation. That should work?

Commit: ee7b012c5

Kernel to test:

# opnsense-update -zkr 24.7.2-nd

Looks like _11 to me on -nd.

Also I found this. 2012 greeting us. openbsd/src@2633ae8c4c8a64

And 2023? openbsd/src@49f39043a02d6

fichtner · 2024-08-24T04:21:46Z

^^^ updated the previous comment, but bumping for awareness: test kernel inside!

fichtner · 2024-08-24T04:31:04Z

Funny that openbsd/src@49f39043a02d6 adds a return (1) where I disabled state matching via return (1) in ee7b012c5

fichtner · 2024-08-24T05:06:45Z

I wonder what role this port omission plays in the grand scheme of things:

https://github.com/openbsd/src/blob/cea0d947241d3d26ca89a911e6b5de37412adb7c/sys/net/pf.c#L2699-L2704

src/sys/netpfil/pf/pf.c

Lines 1891 to 1896 in c61a3c2

    
           case ND_NEIGHBOR_SOLICIT: 
        
           	*icmp_dir = PF_IN; 
        
           case ND_NEIGHBOR_ADVERT: { 
        
           	*virtual_type = ND_NEIGHBOR_SOLICIT; 
        
           	*virtual_id = 0; 
        
           	break;

Since virtual_id is used to construct the state:

src/sys/netpfil/pf/pf.c

Lines 7177 to 7191 in c61a3c2

    
           pf_icmp_mapping(&pd2, iih->icmp6_type, 
        
               &icmp_dir, &multi, &virtual_id, &virtual_type); 
        
           ret = pf_icmp_state_lookup(&key, &pd2, state, m, 
        
               pd->dir, kif, virtual_id, virtual_type, 
        
               icmp_dir, &iidx, PF_ICMP_MULTI_NONE, 1); 
        
           if (ret >= 0) { 
        
           	if (ret == PF_DROP && pd->af == AF_INET6 && 
        
           	    icmp_dir == PF_OUT) { 
        
           		if (*state != NULL) 
        
           			PF_STATE_UNLOCK((*state)); 
        
           		ret = pf_icmp_state_lookup(&key, pd, 
        
           		    state, m, pd->dir, kif, 
        
           		    virtual_id, virtual_type, 
        
           		    icmp_dir, &iidx, multi, 1);

Wireheadbe · 2024-08-24T05:12:33Z

testing:

# opnsense-update -zkr 24.7.2-nd

Will update post when done

edit: seems to be behaving - will keep monitoring

Staticznld · 2024-08-24T05:20:55Z

opnsense-update -zkr 24.7.2-nd

Looks promising traceroute from "net analalyzer" android app s22 looks ok to me. Also my ripe atlas probe is right back online.

Right now not able to check nd sollicitant.

fichtner · 2024-08-24T05:22:22Z

@Staticznld traceroutes were supposedly fixed in 24.7.2 already but thanks for taking a peek so quickly!

Staticznld · 2024-08-24T05:28:06Z

I thought traceroute from Windows was working fine.
And that "MTR " was not. "MTR is working right now over ipv6"

`Asus-PN50 (2a02:a450:xxxx:xxxx::2000) -> google.com (2a00:1450:400e:811::200e) 2024-08-24T07:27:49+0200
Keys: Help Display mode Restart statistics Order of fields quit
Packets Pings
Host Loss% Snt Last Avg Best Wrst StDev

router.internal.SECRET.nl 0.0% 88 2.5 2.9 1.9 6.0 0.7
2001:67c:2502:f100::2:32 0.0% 88 4.4 5.1 3.7 9.0 1.1
2001:67c:24f0:cff0::9 10.2% 88 7.6 9.1 6.5 110.4 11.6
(waiting for reply)
2a00:1450:8106::1 15.9% 88 8.0 8.9 7.1 23.6 2.0
2001:4860:0:1::422c 2.3% 88 10.2 10.5 9.2 20.2 1.6
2001:4860:0:1::890e 67.8% 87 10.2 10.0 9.1 11.5 0.7
2001:4860::c:4000:d9aa 0.0% 87 9.0 9.2 6.9 46.7 4.5
(waiting for reply)
2001:4860:0:1::611b 0.0% 87 11.3 9.5 8.4 15.2 1.2
ams15s48-in-x0e.1e100.net 0.0% 87 10.8 9.4 8.2 11.6 0.8`

Wireheadbe · 2024-08-24T06:02:42Z

testing:
# opnsense-update -zkr 24.7.2-nd
Will update post when done

edit: seems to be behaving - will keep monitoring

working fine, no loss after reboot

cloudz · 2024-08-24T06:23:51Z

testing:

# opnsense-update -zkr 24.7.2-nd

Very fast online after applying DHCPv6 on the WAN interface. This would take a while and lock up the interface on the previous kernels. Previous config for internal DHCPv6 & RA was active instantly as well. 15 minutes in and no loss to report.

sjm42 · 2024-08-24T06:24:34Z

Well, for me 24.7.2-nd seems to behave well. I ran tcpdump for 20 minutes and it was all like this excerpt below.

09:19:52.413301 IP6 fe80::1afd:74ff:fec1:2acd > 2a0b:5c81:10:0:2e2:69ff:fe65:3a25: ICMP6, neighbor solicitation, who has 2a0b:5c81:10:0:2e2:69ff:fe65:3a25, length 32
09:19:52.413351 IP6 fe80::2e2:69ff:fe65:3a25 > fe80::1afd:74ff:fec1:2acd: ICMP6, neighbor advertisement, tgt is 2a0b:5c81:10:0:2e2:69ff:fe65:3a25, length 24
09:20:04.952965 IP6 fe80::2e2:69ff:fe65:3a25 > fe80::1afd:74ff:fec1:2acd: ICMP6, neighbor solicitation, who has fe80::1afd:74ff:fec1:2acd, length 32
09:20:04.960582 IP6 fe80::1afd:74ff:fec1:2acd > fe80::2e2:69ff:fe65:3a25: ICMP6, neighbor advertisement, tgt is fe80::1afd:74ff:fec1:2acd, length 24
09:20:15.453342 IP6 fe80::1afd:74ff:fec1:2acd > fe80::2e2:69ff:fe65:3a25: ICMP6, neighbor solicitation, who has fe80::2e2:69ff:fe65:3a25, length 32
09:20:15.453386 IP6 fe80::2e2:69ff:fe65:3a25 > fe80::1afd:74ff:fec1:2acd: ICMP6, neighbor advertisement, tgt is fe80::2e2:69ff:fe65:3a25, length 24
09:20:55.959949 IP6 fe80::2e2:69ff:fe65:3a25 > fe80::1afd:74ff:fec1:2acd: ICMP6, neighbor solicitation, who has fe80::1afd:74ff:fec1:2acd, length 32
09:20:55.966166 IP6 fe80::1afd:74ff:fec1:2acd > fe80::2e2:69ff:fe65:3a25: ICMP6, neighbor advertisement, tgt is fe80::1afd:74ff:fec1:2acd, length 24
09:20:56.403308 IP6 fe80::1afd:74ff:fec1:2acd > 2a0b:5c81:10:0:2e2:69ff:fe65:3a25: ICMP6, neighbor solicitation, who has 2a0b:5c81:10:0:2e2:69ff:fe65:3a25, length 32
09:20:56.403360 IP6 fe80::2e2:69ff:fe65:3a25 > fe80::1afd:74ff:fec1:2acd: ICMP6, neighbor advertisement, tgt is 2a0b:5c81:10:0:2e2:69ff:fe65:3a25, length 24
09:20:57.443292 IP6 fe80::1afd:74ff:fec1:2acd > fe80::2e2:69ff:fe65:3a25: ICMP6, neighbor solicitation, who has fe80::2e2:69ff:fe65:3a25, length 32
09:20:57.443340 IP6 fe80::2e2:69ff:fe65:3a25 > fe80::1afd:74ff:fec1:2acd: ICMP6, neighbor advertisement, tgt is fe80::2e2:69ff:fe65:3a25, length 24
09:21:39.373616 IP6 fe80::1afd:74ff:fec1:2acd > fe80::2e2:69ff:fe65:3a25: ICMP6, neighbor solicitation, who has fe80::2e2:69ff:fe65:3a25, length 32
09:21:39.373680 IP6 fe80::2e2:69ff:fe65:3a25 > fe80::1afd:74ff:fec1:2acd: ICMP6, neighbor advertisement, tgt is fe80::2e2:69ff:fe65:3a25, length 24
09:21:44.965962 IP6 fe80::2e2:69ff:fe65:3a25 > fe80::1afd:74ff:fec1:2acd: ICMP6, neighbor solicitation, who has fe80::1afd:74ff:fec1:2acd, length 32
09:21:44.973250 IP6 fe80::1afd:74ff:fec1:2acd > fe80::2e2:69ff:fe65:3a25: ICMP6, neighbor advertisement, tgt is fe80::1afd:74ff:fec1:2acd, length 24
09:22:00.403284 IP6 fe80::1afd:74ff:fec1:2acd > 2a0b:5c81:10:0:2e2:69ff:fe65:3a25: ICMP6, neighbor solicitation, who has 2a0b:5c81:10:0:2e2:69ff:fe65:3a25, length 32
09:22:00.403331 IP6 fe80::2e2:69ff:fe65:3a25 > fe80::1afd:74ff:fec1:2acd: ICMP6, neighbor advertisement, tgt is 2a0b:5c81:10:0:2e2:69ff:fe65:3a25, length 24
09:22:21.693336 IP6 fe80::1afd:74ff:fec1:2acd > fe80::2e2:69ff:fe65:3a25: ICMP6, neighbor solicitation, who has fe80::2e2:69ff:fe65:3a25, length 32
09:22:21.693393 IP6 fe80::2e2:69ff:fe65:3a25 > fe80::1afd:74ff:fec1:2acd: ICMP6, neighbor advertisement, tgt is fe80::2e2:69ff:fe65:3a25, length 24

Crazyachmed · 2024-08-24T08:04:54Z

Havn't had my coffee yet, but 24.7.2-nd looks good on my side as well.

@fichtner: One note from when I have the issue: From time to time a single host would randomly behave for a couple of minutes. That smells very much like a state is created for some reason.

doktornotor · 2024-08-24T08:12:36Z

Just chiming in here - 24.7.2-nd works as expected.

Maybe it still allows carefully crafted packets from attackers to trigger extremely dangerous echo reply even when it should be blocked by pf (haven't managed to find a POC code anywhere regarding the SA) - but I don't care frankly.

fichtner · 2024-08-24T08:13:45Z

No this is locked to ND_NEIGHBOR_SOLICIT/ND_NEIGHBOR_ADVERT

fichtner · 2024-08-24T08:26:34Z

Thanks to all chiming in BTW. We will keep this on top of the agenda regardless of how FreeBSD's stance is. ❤️

meyergru · 2024-08-24T08:39:05Z

24.7.2-nd looking good for the synthetic test on a VM instance for me as well. Also applied it to my bare-metal box to verify traceroute and mtr and still looks good there, too.

alan7000 · 2024-08-24T10:33:04Z

Hello, the 24.7.2-nd works perfectly. My network is almost ipv6-only and I have no more packet loss.

Thank you very much. 😁

Slashic · 2024-08-24T10:54:26Z

24.7.2-nd does work for me too (bare metal setup):

Before (with 24.7.2)

After (with 24.7.2-nd)

So far so good, i'll keep monitoring it and update if needed, thank you for the quick fix!

aque · 2024-08-24T18:03:17Z

I have been troubleshooting an issue where my Comcast router responds with Destination Unreachable (Address unreachable) and IPv6 packets would drop. I thought this was a Comcast problem, and it did not occur to me this is related to ND until I saw this. I confirmed with tcpdump and applied the 24.7.2-nd kernel. No issues in the last 45 minutes! Thank you all as I was dreading the call to Comcast. 🎉

This patch is likely going to fix a lot of intermittent IPv6 issues.

jdeluyck · 2024-08-25T10:45:55Z

24.7.2-nd also works without issues (that I've noticed or seen) on my apu2.

aque · 2024-08-25T15:33:10Z

Here is documentation on how significant this kernel was in my environment. dpinger was set to its default rates. You can tell when I applied the new kernel. 😄

doktornotor · 2024-08-25T17:23:38Z

@aque @Slashic - I've borrowed your pictures for the upstream bug report, hopefully it makes it more clear how bad things may be for some people with these bad upstream patches applied.

Things have not been nearly that bad for me in most of my setups, I'm probably lucky.

fichtner · 2024-08-25T18:37:02Z

I've added some more things there that have been on my mind for most of the weekend including the ping statistics which are a good indicator indeed. Let's focus on a shippable improvement in OPNsense for 24.7.3 tomorrow. The release will likely land later in this week so we still have time to test a bit more. :)

Cheers,
Franco

fichtner added upstream Third party issue support Community support and removed upstream Third party issue labels Aug 22, 2024

fichtner closed this as completed Aug 22, 2024

fichtner added upstream Third party issue and removed support Community support labels Aug 23, 2024

fichtner self-assigned this Aug 23, 2024

fichtner transferred this issue from opnsense/core Aug 23, 2024

fichtner reopened this Aug 23, 2024

fichtner changed the title ~~Intermittent answer to IPv6 neighbor solicitation~~ pf: delayed responses on ICMP6 neighbor solicitation packets since 24.7.1 (redux) Aug 23, 2024

fichtner mentioned this issue Aug 23, 2024

pf: delayed responses on ICMP6 neighbor solicitation packets since 24.7.1 #217

Closed

pf: delayed responses on ICMP6 neighbor solicitation packets since 24.7.1 (redux) #218

pf: delayed responses on ICMP6 neighbor solicitation packets since 24.7.1 (redux) #218

Comments

Crazyachmed commented Aug 22, 2024

fichtner commented Aug 22, 2024 • edited Loading

fichtner commented Aug 22, 2024

fichtner commented Aug 22, 2024

Crazyachmed commented Aug 22, 2024

fichtner commented Aug 22, 2024

Crazyachmed commented Aug 23, 2024

fichtner commented Aug 23, 2024

Crazyachmed commented Aug 23, 2024

fichtner commented Aug 23, 2024

sjm42 commented Aug 23, 2024

fichtner commented Aug 23, 2024 • edited Loading

meyergru commented Aug 23, 2024 • edited Loading

fichtner commented Aug 23, 2024

fichtner commented Aug 23, 2024

meyergru commented Aug 23, 2024

fichtner commented Aug 23, 2024 • edited Loading

Crazyachmed commented Aug 23, 2024

fichtner commented Aug 23, 2024

Crazyachmed commented Aug 23, 2024

sjm42 commented Aug 23, 2024

sjm42 commented Aug 23, 2024

doktornotor commented Aug 23, 2024

Crazyachmed commented Aug 23, 2024

doktornotor commented Aug 23, 2024

fichtner commented Aug 23, 2024

fichtner commented Aug 24, 2024 • edited Loading

fichtner commented Aug 24, 2024 • edited Loading

fichtner commented Aug 24, 2024

fichtner commented Aug 24, 2024

fichtner commented Aug 24, 2024 • edited Loading

Wireheadbe commented Aug 24, 2024 • edited Loading

Staticznld commented Aug 24, 2024 • edited Loading

fichtner commented Aug 24, 2024 • edited Loading

Staticznld commented Aug 24, 2024 • edited Loading

Wireheadbe commented Aug 24, 2024

cloudz commented Aug 24, 2024

sjm42 commented Aug 24, 2024

Crazyachmed commented Aug 24, 2024

doktornotor commented Aug 24, 2024

fichtner commented Aug 24, 2024

fichtner commented Aug 24, 2024

meyergru commented Aug 24, 2024 • edited Loading

alan7000 commented Aug 24, 2024

Slashic commented Aug 24, 2024

aque commented Aug 24, 2024

jdeluyck commented Aug 25, 2024

aque commented Aug 25, 2024

doktornotor commented Aug 25, 2024

fichtner commented Aug 25, 2024

fichtner commented Aug 22, 2024 •

edited

Loading

fichtner commented Aug 23, 2024 •

edited

Loading

meyergru commented Aug 23, 2024 •

edited

Loading

fichtner commented Aug 23, 2024 •

edited

Loading

fichtner commented Aug 24, 2024 •

edited

Loading

fichtner commented Aug 24, 2024 •

edited

Loading

fichtner commented Aug 24, 2024 •

edited

Loading

Wireheadbe commented Aug 24, 2024 •

edited

Loading

Staticznld commented Aug 24, 2024 •

edited

Loading

fichtner commented Aug 24, 2024 •

edited

Loading

Staticznld commented Aug 24, 2024 •

edited

Loading

meyergru commented Aug 24, 2024 •

edited

Loading