share_forward: NAT'd GRE (PPTP) egresses with stale IP header checksum after first packet

[Alfamonk]

Describe the bug

With net.pf.share_forward=1 (default; Firewall > Settings > Advanced > "Shared forwarding"), outbound-NAT'd GRE (IP proto 47) egresses with an invalid IP header checksum for every packet after the first of a flow.

The source address is rewritten, but the checksum is only adjusted by the TTL-decrement delta - the address change is never accounted for. Peers silently discard the packets. The first packet of the flow (which creates the state) is correct; every later packet on the established-state fast path is not.

Result: PPTP clients cannot complete PPP LCP. Control setup on TCP/1723 succeeds (SCCRQ/SCCRP, OCRQ/OCRP, SLI), GRE opens, the server acks GRE seq 0 and then never acks again, re-sending its own LCP Conf-Request until the client gives up (Windows error 619). TCP/UDP are unaffected, so ordinary traffic hides this completely.

Environment

• OPNsense 26.7.1_1, kernel FreeBSD 15.1-RELEASE-p1 stable/26.7-n283674-12334a596709 amd64
• WAN pppoe0 (MTU 1492) over VirtIO, LAN vtnet0, outbound NAT automatic
• Checksum / TSO / LRO offload all disabled - not a factor
• Single LAN client to a single PPTP server, so this is not the known multi-client GRE Call-ID limitation

To Reproduce

1. Automatic outbound NAT, net.pf.share_forward=1 (default)
2. From a LAN host, dial any PPTP server on the internet (stock Linux pptpd here)
3. tcpdump -n -v -i <wan> proto 47
4. First GRE packet is fine; every later outbound GRE packet shows bad cksum

Evidence

Consecutive outbound GRE packets of one call, captured pre-NAT (LAN) and post-NAT (WAN), checksums verified arithmetically. Addresses redacted:

packet	pre-NAT	post-NAT emitted	correct value
seq 0	valid	0xbdee	0xbdee	OK
seq 1	valid	0x1745	0xbde1	BAD
seq 2	valid	0x1753	0xbdef	BAD
seq 3	valid	0x1752	0xbdee	BAD

Signature: every bad value equals the pre-NAT checksum + 0x0100 - exactly and only the one's-complement delta for ttl 128 -> 127. The source-address rewrite contributes nothing to the emitted checksum.

All packets arrive from the LAN host with valid checksums, so the corruption is introduced by the firewall. With the workaround applied: 1230 outbound GRE packets, 0 bad, and PPP completes normally (LCP -> CHAP -> CCP -> IPCP).

Expected behavior

NAT'd GRE should egress with a correctly recomputed IP header checksum, as TCP and UDP do.

Workaround

net.pf.share_forward=0 (uncheck "Shared forwarding"). Immediate and complete; an already-established tunnel survives the change.

Possibly related

#20 (port forwarding with policy routing), #52 (panic with route-to), forum topic 4546 (captive portal redirection). The 26.7 notes mention "pf: do not mangle IP header before shared forwarding".

Full pcap pair (LAN + WAN) available on request.

[fichtner]

So the route-to standard code is quite crude and glosses over broken checksums by recomputing them. The packet filter shouldn’t try to forward packets to the hardware nor try to mangle the IP header. The reality is therefore difficult.

That being said there wasn’t any such issues in the past which indicates a certain degradation of design philosophy in pf.

What I find strange now is that there are multiple alleged spots where the TTL is altered but not recomputed? From the first bug fix there was only one spot where this code actually executes. Maybe you can throw your AI at the particular code being involved?

Cheers,
Franco

[Alfamonk]

Maybe you can throw your AI at the particular code being involved?

Cheers, Franco

Thanks Franco for your fast response, and sorry I had to ask the AI for help. But it was a desperate situation, and all of my VPN's stopped working. You are right, Claude helped me solve the problem (pretty fast actually), and then I asked if somehow we could help by letting OPNsense community about the problem. I don't have the knowledge to answer high technical questions, so I will let him/her/he/she/they explain:

Thanks Franco. I went through the source for the exact kernel this was captured on (stable/26.7-n283674-12334a596709).

On the TTL question — you're right, there is only one spot, and the TTL is a red herring. My table rows were four consecutive packets, not four code paths; sorry for the ambiguity. pf.c decrements TTL in exactly one place for IPv4 (pf_route(), line 9300) and one for IPv6 (line 9647), and both sit after the V_pf_share_forward early return at 9277-9297. So in the shared-forwarding path pf never touches the TTL at all. The +0x0100 I measured is ip_forward()'s own incremental fixup applied on top of an already-stale checksum.

The actual bug is in pf_change_ap() (pf.c:3403)

	pf_addrcpy(&ao, a, pd->af);
	if (pd->af == pd->naf)
		pf_addrcpy(a, an, pd->af);          /* 3418: address rewritten */

	if (pd->m->m_pkthdr.csum_flags & (CSUM_DELAY_DATA | CSUM_DELAY_DATA_IPV6))
		*pd->pcksum = ~*pd->pcksum;

	if (p == NULL)  /* no port -> done. no cksum to worry about. */
		return;                             /* 3424: early return */
	...
		*pd->ip_sum = pf_cksum_fixup(pf_cksum_fixup(*pd->ip_sum,
		    ao.addr16[0], an->addr16[0], 0),
		    ao.addr16[1], an->addr16[1], 0); /* 3433: never reached */

For GRE — and any port-less protocol — pd->sport / pd->dport are NULL, so p == NULL. The source address is rewritten at 3418 and the function returns at 3424 without ever fixing *pd->ip_sum.

The comment is only true of the L4 checksum: pd->pcksum is deliberately a dummy for these protocols (pd->pcksum = &pd->sctp_dummy_sum, pf.c:10787). The IP header checksum still has to be corrected, because the address changed six lines earlier.

Why only the first packet of a flow is correct: it goes through pf_translate_compat(), whose default: case (pf.c:6639-6658) uses pf_change_a(&pd->src->v4.s_addr, pd->ip_sum, ...), which does fix the header checksum. Every subsequent packet matches state and goes through pf_test_state() → pf_change_ap() at pf.c:7393.

Why share_forward exposes it: with it off, pf_route() takes the hardcoded send path and does ip->ip_sum = in_cksum(m0, ip->ip_hl << 2) at 9398-9400, recomputing the header checksum wholesale and masking the stale value — exactly the "glosses over broken checksums by recomputing them" you described. With share_forward on, the packet returns at 9291 and ip_forward() only applies its incremental TTL delta, so the bad checksum reaches the wire.

TCP/UDP are unaffected because they have ports and reach line 3433.

Suggested shape of a fix (untested)

Hoist the IPv4 header fixup above the early return:

	if (pd->af == AF_INET && pd->naf == AF_INET && pd->ip_sum)
		*pd->ip_sum = pf_cksum_fixup(pf_cksum_fixup(*pd->ip_sum,
		    ao.addr16[0], an->addr16[0], 0),
		    ao.addr16[1], an->addr16[1], 0);

	if (p == NULL)
		return;

with the existing fixup at 3433 dropped so it isn't applied twice.

Unrelated, same class, spotted on the way

pf_normalize_tcp_stateful() assigns h->ip_ttl at pf_norm.c:1623 with no pf_cksum_fixup, whereas min-ttl (2235), ip_off (2226), tos (2255) and ip_id (2275) in the same file all do one. It's TCP-only and gated on scrub ... reassemble tcp, so unrelated to this report — but it looks like it would corrupt the header checksum the same way.

Happy to test a patch on this box, and I still have the LAN+WAN pcap pair if it's useful.

[fichtner]

That’s pretty helpful actually. Thanks! Maybe @swhite2 can take a closer look now as I’m not at the office at the moment.

What is odd is that the TTL decrement in the netinet side of the stack does not fix the checksum by default anymore. I suspected this code was the issue earlier. It’s an easy cheap fix actually. Let me grab the commit.

[fichtner]

Ultimately we rolled this back c579a84 since the issue was confirmed on the pf side, but could be the one you’re seeing now despite the fact that pf might set the wrong flags too the stack on FreeBSD 14 may have handled this a bit more pragmatic.