Migration notes: everything that had to change moving from OpenWrt 24.10 to 25.12

[ispyisail]

This is a complete record of everything that had to be fixed or changed to move Gargoyle from OpenWrt 24.10 (kernel 6.6.143, GCC 13-era toolchain, nftables 1.1.1) to OpenWrt 25.12 (v25.12.5, kernel 6.12.94, GCC 14.3, nftables 1.1.6). All of it landed in PR #41 (merge commit 3e909f422). Final verified state: a from-scratch x86 build completes first-pass with no manual intervention, and the full 3-site vnet simulator suite passes 238/243 with zero real failures (the 5 remaining are long-standing TODOs unrelated to the bump).

The short version: the compile fixes were the easy 80% of the effort. The dangerous 20% was three runtime bugs that compiled completely clean and only revealed themselves on a booted system — one of them corrupted the root filesystem. If you take one thing from this writeup: after a major base bump, a green build means almost nothing until you have booted the image and exercised it.

---

1. Base tree and patch maintenance

• Pin: 16a0e12f4 moves the build to OpenWrt 25.12 (v25.12.5).
• Patches that needed re-deriving against the new tree (context drift, not logic changes): 004-fix_postinst_prerm_scripts, 027-wireguard_tools_bump, 101-gargoyle_ssid_override.
• Patches that had to be deleted: 023 and 024 — both were hand-backports of upstream kernel/netfilter features that 25.12's pinned versions now carry natively. Leaving them in causes patch failures; the check is to look at the live kernel source at the failure point, not just the reject file, before assuming a regression.
• Package manager: USE_APK=n forced in every diffconfig profile (45df18015) — Gargoyle keeps gpkg/opkg as its package manager; 25.12 defaults to apk if left alone.
• Diffconfigs: all 5 x86 profile diffconfigs re-derived against the 25.12 kernel/package tree (6d5db1780); kernel module lists shift between kernel versions (e.g. ip6_tables.ko needed explicit selection again, 71bd117c4).

2. GCC 14 compile fixes (the easy 80%)

GCC 14 promotes several long-tolerated patterns into hard errors or new warnings. Every one of these was a real fix, but all were mechanical once found:

Package	Failure class
libbbtargz	implicit-function-declaration
libnftbwctl	incompatible-pointer-type
zip	broken cross-compile configure probes
samba36	mkdir macro collision + missing include
gargoyle, qos-gargoyle	3× incompatible-pointer-type
cmake / mbedtls-clu / libffi	assorted new-toolchain breaks
netfilter-match-modules (all 6 modules + userspace)	implicit declarations, %lld format mismatches, a duplicated ipany union

3. Runtime bug #1: nft_bandwidth corrupted the root filesystem (kernel 6.12)

The worst bug of the migration, and the one with the most instructive failure mode (issue #40).

• Kernel 6.12 added a new check to nft_register_expr(): WARN_ON_ONCE(type->maxattr > NFT_EXPR_MAXATTR) with NFT_EXPR_MAXATTR = 16. This check does not exist in 6.6 — the module had always declared 17 netlink attributes and nothing ever cared.
• Registration failure alone would just mean "module doesn't load." The real damage: init() called nf_register_sockopt() before nft_register_expr(), and did not roll it back on failure. When a module's init returns nonzero, the kernel unloads the module's code without calling its exit function — leaving the global sockopt table pointing at freed memory. The next setsockopt/getsockopt on those command numbers (which Gargoyle's own bandwidth/quota userspace uses at boot) jumped into freed memory.
• On disk this manifested as ext4 metadata corruption within ~7 seconds of first boot (deleted inode referenced, doubly allocated) and an unbootable image — nothing about the symptom pointed at a kernel module. The image was byte-for-byte clean at rest (fsck clean); the corruption happened live.
• Diagnosis path that worked, in order: QEMU monitor screendump (serial was silent — output went to VGA), module-by-module isolation via loop-mounting the image and moving .ko files aside (bandwidth alone reproduced it; webmon/weburl alone were clean), then a live insmod on a booted system with a serial console attached, which finally surfaced the WARN with a stack trace.
• Fix (1ced1490f): unregister the sockopt on any init failure (makes future registration failures fail clean), and drop the NFTA_BANDWIDTH_MINUTESWEST attribute to bring the count to 16 — it was dead on the kernel side (declared in the parse policy, never read or written) but had live userspace libnftnl plumbing that had to be removed in sync to keep the wire-format enums identical on both sides.

4. Runtime bug #2: nft list ruleset asserted on every box (nftables 1.1.6)

• nftables 1.1.6 replaced its statement-ops mechanism: old versions read a handler pointer stored in the statement itself; 1.1.6 resolves through a central switch (stmt->type) in __stmt_ops_by_type() and BUG()s on unknown types.
• Gargoyle's four custom statement integrations (bandwidth, timerange, webmon, weburl) define their ops structs but had no cases in the new switch — because the switch didn't exist when they were written. Any ruleset containing one of these statements crashed nft list: Assertion failed: 0 (src/statement.c: stmt_ops: 1366). A stock Gargoyle firewall carries dozens of bandwidth statements, so this was universal.
• Sneaky aspect Fix WireGuard config file import to handle flexible formatting #1: rule loading was unaffected (fw4 goes through the parser, not the print path), so the firewall worked and everything looked fine until something listed the ruleset.
• Sneaky aspect chore: repo scaffolding for end users (issue/PR templates, security policy) #2: Gargoyle runtime scripts (gargoyle_firewall_util.sh quota/restriction checks, manage_groups.sh) grep the output of nft list table inet fw4 — they only kept "working" because nft prints progressively and the grepped chain names happened to appear in the partial output before the crash. Ordering-dependent luck, not correctness.
• Fix (401e28499): one case per module inserted into the switch via each module's integration meta.
• Debug gotcha worth recording: side-loading a rebuilt /usr/sbin/nft onto a test box changes nothing — nft is an ~18KB shim; the statement code lives in libnftables.so. Side-load the library and run with LD_LIBRARY_PATH.

5. Runtime bug #3: apk's target build silently missing a host dependency

• package/system/apk declares HOST_BUILD_DEPENDS:=lua/host, which only covers apk's own host build. The target package's meson configure also hard-requires lua5.1 on the host. In a tree where nothing builds apk/host — which is Gargoyle's situation, since gpkg is the package manager and apk-mbedtls is only selected as a target package — lua/host never enters the build graph and every from-scratch build fails deterministically at package/system/apk/compile.
• Fix: new patches-generic/032 adding PKG_BUILD_DEPENDS:=lua/host (2704ec924).

6. Wifi stack changes

• gargoyle_stamgr's roaming scan/disconnect detection broke under 25.12's wifi stack (247796a0b).
• APs come up broken when country is left at the 00 default (cf2b28c1a) — wifi-scripts behavior change.

7. Build tooling lessons (these cost more time than any single bug)

• build.sh/rebuild.sh swallowed make failures. Neither checked the main OpenWrt make's exit status; on failure they fell through to a bare silent exit. A failed build was indistinguishable from the script just stopping — the apk failure above hid behind this across multiple full builds. Fixed in eb7594a31 (loud error + exit status propagation). Debug technique that cut through it: bypass the wrappers entirely and run make V=s directly in <target>-src.
• Plain rebuilds do not pick up netfilter-match-modules source changes. integrate_netfilter_modules.sh (which copies module sources into the kernel tree) only runs from build.sh, i.e. only under FULL_BUILD=true. A plain make <target> rebuild will succeed and produce a real image containing stale module code, with no warning. Not yet fixed — treat any netfilter-match-modules change as requiring a full build, or copy the changed files into <target>-src's staged kernel tree manually.
• If you build inside a container with its own git checkout, verify the checkout's HEAD before trusting any build — a build banner stamped with the wrong commit hash is the tell (create_gargoyle_banner embeds it; check /etc/banner in the built image).

8. What testing caught what (methodology notes)

• The compile fixes were all caught by just building.
• The rootfs corruption was caught by booting the image in the vnet simulator — no amount of static image inspection found it (fsck was clean).
• The nft assert was caught by a single suite assertion (T-CFG-04: nft list ruleset exits 0) that exists precisely because "the firewall loads" and "the firewall is inspectable" are different properties.
• The apk dependency bug was only distinguishable from flakiness because it reproduced on a second from-scratch build after the first was (wrongly) blamed on build-order racing.
• Fresh-boot caveat for anyone re-verifying: a just-booted box has zero bandwidth statements in its ruleset for the first minutes until bwmon and the firewall settle — an early nft list ruleset exit-0 proves nothing about the stmt_ops fix. Wait until statements are present.

9. Still open after the merge

• Other targets (ath79, mediatek, ipq40xx) build against the same tree but have not been flashed or runtime-tested on 25.12 yet — the MT6000 (mediatek) is the natural next hardware test.
• The rebuild.sh-vs-integrate_netfilter_modules.sh staleness gap (section 7) still wants a proper fix.
• The 5 standing vnet TODOs (T-OVPN-13 CRL expiry, T-IPV6-10/11/12, T-WG-12 kill-switch) predate the bump and remain.
• 24.10 EOL is 2026-09-05, which is the deadline this migration was working against.

[ispyisail]

Update: the full bump landed as #43 (reland of the reverted #41), now verified building clean on all four primary targets (x86, mediatek, ath79, ipq40xx), not just x86.