OpenWrt 24.10 to 25.12 Migration Notes

OpenWrt 24.10 → 25.12 Migration Notes

Also posted as Discussion #42. This wiki copy is the living version — update it here as later targets get tested.

This is a complete record of everything that had to be fixed or changed to move Gargoyle from OpenWrt 24.10 (kernel 6.6.143, GCC 13-era toolchain, nftables 1.1.1) to OpenWrt 25.12 (v25.12.5, kernel 6.12.94, GCC 14.3, nftables 1.1.6).

PR status: the bump landed first as PR #41, which was reverted (revert commit) and relanded as a clean, scoped series in PR #43 — the bump only, with all four primary targets (x86, mediatek, ath79, ipq40xx) verified building clean before merge, unlike #41. The unrelated work that had ridden along in #41 was split into four independently reviewable follow-ups, all stacked on #43's branch: #44 gapk Phase 7, #45 netfilter warning cleanup, #46 NULL guards, #47 build-script loud-fail. Commit links below point at the final, relanded SHAs.

Final verified state: a from-scratch build completes first-pass with no manual intervention on all four primary targets, and the full 3-site vnet simulator suite passes 230/235 with zero real failures (the 5 remaining are long-standing TODOs unrelated to the bump).

The short version: the compile fixes were the easy 80% of the effort. The dangerous 20% was three runtime bugs that compiled completely clean and only revealed themselves on a booted system — one of them corrupted the root filesystem. If you take one thing from this writeup: after a major base bump, a green build means almost nothing until you have booted the image and exercised it.

---

1. Base tree and patch maintenance

• Pin: cfddc9a78 moves the build to OpenWrt 25.12 (v25.12.5).
• Patches that needed re-deriving against the new tree (context drift, not logic changes): 004-fix_postinst_prerm_scripts, 027-wireguard_tools_bump, 101-gargoyle_ssid_override, and (added during target verification) the mediatek uboot serial-rx-buffer patch — all squashed into 6342c5b04.
• Patches that had to be deleted: 023 and 024 — both were hand-backports of upstream kernel/netfilter features that 25.12's pinned versions now carry natively (a2c5f27). Leaving them in causes patch failures; the check is to look at the live kernel source at the failure point, not just the reject file, before assuming a regression.
• Package manager: USE_APK=n forced in every diffconfig profile (a0e2dab9) — Gargoyle keeps gpkg/opkg as its package manager; 25.12 defaults to apk if left alone.
• Diffconfigs: all 5 x86 profile diffconfigs re-derived against the 25.12 kernel/package tree (38ff3ae); kernel module lists shift between kernel versions (see the ip6_tables.ko fix in §5).

2. GCC 14 compile fixes (the easy 80%)

GCC 14 promotes several long-tolerated patterns into hard errors or new warnings. Every one of these was a real fix, but all were mechanical once found:

Package	Failure class	Commit
libbbtargz	implicit-function-declaration	029242f
libnftbwctl, gargoyle, qos-gargoyle	incompatible-pointer-type (3×)	40c7052
zip	broken cross-compile configure probes	154271b
samba36	mkdir macro collision + missing include	d8f1f67
CMake 4 (all packages using cmake_minimum_required < 3.5)	policy-version rejection	b562175
mbedtls-clu, https-dns-proxy, libffi	hash drift + autoreconf breakage	7de746a
netfilter-match-modules (all 6 modules + userspace)	implicit declarations, %lld format mismatches, a duplicated ipany union — cleanup, not a hard-error fix; see PR #45	6107f71

3. Runtime bug #1: nft_bandwidth corrupted the root filesystem (kernel 6.12)

The worst bug of the migration, and the one with the most instructive failure mode (issue #40).

• Kernel 6.12 added a new check to nft_register_expr(): WARN_ON_ONCE(type->maxattr > NFT_EXPR_MAXATTR) with NFT_EXPR_MAXATTR = 16. This check does not exist in 6.6 — the module had always declared 17 netlink attributes and nothing ever cared.
• Registration failure alone would just mean "module doesn't load." The real damage: init() called nf_register_sockopt() before nft_register_expr(), and did not roll it back on failure. When a module's init returns nonzero, the kernel unloads the module's code without calling its exit function — leaving the global sockopt table pointing at freed memory. The next setsockopt/getsockopt on those command numbers (which Gargoyle's own bandwidth/quota userspace uses at boot) jumped into freed memory.
• On disk this manifested as ext4 metadata corruption within ~7 seconds of first boot (deleted inode referenced, doubly allocated) and an unbootable image — nothing about the symptom pointed at a kernel module. The image was byte-for-byte clean at rest (fsck clean); the corruption happened live.
• Diagnosis path that worked, in order: QEMU monitor screendump (serial was silent — output went to VGA), module-by-module isolation via loop-mounting the image and moving .ko files aside (bandwidth alone reproduced it; webmon/weburl alone were clean), then a live insmod on a booted system with a serial console attached, which finally surfaced the WARN with a stack trace.
• Fix (327a8eb): unregister the sockopt on any init failure (makes future registration failures fail clean), and drop the NFTA_BANDWIDTH_MINUTESWEST attribute to bring the count to 16 — it was dead on the kernel side (declared in the parse policy, never read or written) but had live userspace libnftnl plumbing that had to be removed in sync to keep the wire-format enums identical on both sides.
• A defensive NULL-guard fix found while investigating this (real bug, same code family, but tested and confirmed NOT the cause of the corruption) is split out into its own PR #46.

4. Runtime bug #2: nft list ruleset asserted on every box (nftables 1.1.6)

• nftables 1.1.6 replaced its statement-ops mechanism: old versions read a handler pointer stored in the statement itself; 1.1.6 resolves through a central switch (stmt->type) in __stmt_ops_by_type() and BUG()s on unknown types.
• Gargoyle's four custom statement integrations (bandwidth, timerange, webmon, weburl) define their ops structs but had no cases in the new switch — because the switch didn't exist when they were written. Any ruleset containing one of these statements crashed nft list: Assertion failed: 0 (src/statement.c: stmt_ops: 1366). A stock Gargoyle firewall carries dozens of bandwidth statements, so this was universal.
• Sneaky aspect #1: rule loading was unaffected (fw4 goes through the parser, not the print path), so the firewall worked and everything looked fine until something listed the ruleset.
• Sneaky aspect #2: Gargoyle runtime scripts (gargoyle_firewall_util.sh quota/restriction checks, manage_groups.sh) grep the output of nft list table inet fw4 — they only kept "working" because nft prints progressively and the grepped chain names happened to appear in the partial output before the crash. Ordering-dependent luck, not correctness.
• Fix (0bbaf23): one case per module inserted into the switch via each module's integration meta.
• Debug gotcha worth recording: side-loading a rebuilt /usr/sbin/nft onto a test box changes nothing — nft is an ~18KB shim; the statement code lives in libnftables.so. Side-load the library and run with LD_LIBRARY_PATH.

5. Runtime bug #3: ip6_tables.ko silently failed to build on every target

• Kernel 6.12 gates ip6_tables.ko behind a hidden select-only symbol (IP6_NF_IPTABLES_LEGACY). Setting CONFIG_IP6_NF_IPTABLES=m alone — all that kmod-nf-ipt6 sets — no longer builds it.
• The IPv4 twin only worked by accident: kmod-ipt-core's CONFIG_IP_NF_FILTER=m happens to select the IPv4 legacy symbol, and that package is in Gargoyle's set; there was no IPv6 equivalent selected anywhere.
• First fix attempt scoped this to target/linux/x86/config-6.12 only — which worked for x86 but left mediatek/ath79/ipq40xx broken (this was the actual proximate cause of the #41 revert: master could only build x86). Corrected fix (1f2fd85) moves it to target/linux/generic/config-6.12, fixing every target at once. Verified via the mediatek gate: kmod-nf-ipt6 now packages cleanly on both the mt7622 and filogic (GL-iNet MT6000) subtargets.

6. Runtime bug #4: apk's target build silently missing a host dependency

• package/system/apk declares HOST_BUILD_DEPENDS:=lua/host, which only covers apk's own host build. The target package's meson configure also hard-requires lua5.1 on the host. In a tree where nothing builds apk/host — which is Gargoyle's situation, since gpkg is the package manager and apk-mbedtls is only selected as a target package — lua/host never enters the build graph and every from-scratch build fails deterministically at package/system/apk/compile.
• Fix: new patches-generic/032 adding PKG_BUILD_DEPENDS:=lua/host. Landed alongside the rest of the apk-backend work in PR #44 (2751241), since it only matters once apk-mbedtls is selected.

7. Wifi stack changes

• gargoyle_stamgr's roaming scan/disconnect detection broke under 25.12's wifi stack (b4ceb2f).
• APs come up broken when country is left at the 00 default (8c48564) — wifi-scripts behavior change.

8. Build tooling lessons (these cost more time than any single bug)

• build.sh/rebuild.sh swallowed make failures. Neither checked the main OpenWrt make's exit status; on failure they fell through to a bare silent exit. A failed build was indistinguishable from the script just stopping — the apk failure above hid behind this across multiple full builds. Fixed in PR #47 (0175d9c — loud error + exit status propagation). Debug technique that cut through it: bypass the wrappers entirely and run make V=s directly in <target>-src.
• Plain rebuilds do not pick up netfilter-match-modules source changes. integrate_netfilter_modules.sh (which copies module sources into the kernel tree) only runs from build.sh, i.e. only under FULL_BUILD=true. A plain make <target> rebuild will succeed and produce a real image containing stale module code, with no warning. Not yet fixed — treat any netfilter-match-modules change as requiring a full build, or copy the changed files into <target>-src's staged kernel tree manually.
• A container's git checkout can silently be on the wrong commit. Nothing about a local git push reaches a build container automatically. Before trusting a container build, check git log --oneline -1 inside it against the commit you intend to build — a build banner stamped with the wrong hash is the tell (create_gargoyle_banner embeds it; check /etc/banner in the built image). This exact mistake produced a wasted ~40-minute build during this migration.

9. What testing caught what (methodology notes)

• The compile fixes were all caught by just building.
• The rootfs corruption was caught by booting the image in the vnet simulator — no amount of static image inspection found it (fsck was clean).
• The nft assert was caught by a single suite assertion (T-CFG-04: nft list ruleset exits 0) that exists precisely because "the firewall loads" and "the firewall is inspectable" are different properties.
• The ip6_tables gap was caught by extending target verification beyond x86 — it was invisible on x86 alone since the IPv4 legacy symbol happened to already be selected there for unrelated reasons.
• The apk dependency bug was only distinguishable from flakiness because it reproduced on a second from-scratch build after the first was (wrongly) blamed on build-order racing.
• Fresh-boot caveat for anyone re-verifying: a just-booted box has zero bandwidth statements in its ruleset for the first minutes until bwmon and the firewall settle — an early nft list ruleset exit-0 proves nothing about the stmt_ops fix. Wait until statements are present.

10. Verification summary and what's still open

All four primary targets verified with a genuine FULL_BUILD=true (from-scratch, zero manual intervention) on the final relanded branch:

Target	Result
x86	Clean build + full 3-site vnet suite: 230/235, zero real failures
mediatek	Clean build, both subtargets (mt7622 + filogic); GL-iNet MT6000 sysupgrade image produced; kmod-nf-ipt6 verified packaging on both
ath79	Clean build, both profiles (generic + nand)
ipq40xx	Clean build

Still open:

• Other targets (bcm27xx, bcm47xx, ipq806x, mvebu, ramips, rockchip) were not touched in this bump and keep their existing 24.10-generation full configs, per the project's existing convention of only converting a target once a buildroot exists to verify against.
• The rebuild.sh-vs-integrate_netfilter_modules.sh staleness gap (§8) still wants a proper fix.
• The 5 standing vnet TODOs (T-OVPN-13 CRL expiry, T-IPV6-10/11/12, T-WG-12 kill-switch) predate the bump and remain.
• MT6000 hardware flash (the actual physical device) — the sysupgrade image is built and verified in the simulator; flashing real hardware is the natural next step.
• 24.10 EOL is 2026-09-05, which is the deadline this migration was working against.