RFC: Signed OTA firmware upgrade (usign/fwtool/sysupgrade + signed manifest)

[ispyisail]

Status: Ready for review · Updated: paired with #97 (Settings Profile) for config survival, hardened for fleet reality (hardware floors, EOL policy, allowlist rollout), covering the bootstrap problem (how routers on old firmware reach the OTA era safely), channels and custom images (stable/testing manifests, manifest anti-replay, self-built firmware with private OTA channels, custom plugin sets), and now downgrades — validated history entries, lossless downgrade→re-upgrade round-trips via the preserved profile.

Summary

Gargoyle has no verified one-click firmware upgrade today — you download the right sysupgrade.bin by hand, upload it, and hope you grabbed the correct file. No update check, no authenticity guarantee, no guided progress. This RFC proposes a signed OTA upgrade built entirely from open OpenWrt primitives (usign, fwtool, sysupgrade) plus a small Gargoyle-hosted signed manifest. No proprietary components, no third-party servers, no telemetry.

Prior art: the GL.iNet stock one_click_upgrade was studied on owned hardware to understand the method (not to copy code). Its entire verify/apply path is already stock OpenWrt — fwtool -t -s to extract the appended signature, usign -V against a build pubkey, then sysupgrade -n/-k. Everything GL layers on top (their update API, mirror CDN, cloud telemetry, Vue UI) is exactly what we replace with our own.

Mechanism (three parts)

1. Build-time signing — the build generates a usign keypair once; the private key stays on the build host, the public key is baked into every image at /etc/gargoyle/ota.pub. Each *-sysupgrade.bin is signed (usign -S) as a detached <image>.sig published alongside the image. (Revised from “appended via fwtool” — implementation revealed the buildroot already appends its own signature: when BUILD_KEY exists, include/image-commands.mk signs every image and appends a ucert chain via fwtool -S. Appending a second trailer would bury the native one in exactly the slot stock fwtool_check_signature pops expecting a ucert — a no-op on today’s images (no /usr/bin/ucert shipped, REQUIRE_IMAGE_SIGNATURE unset) but broken the day enforcement lands — and it mutates the artifact so its sha256 no longer matches the build’s own sha256sums. Detached signing keeps every published byte identical to build output. See the dated comment below for the evidence trail.)

2. A signed manifest.json (static file / GitHub Release asset — no dynamic server) keyed by OpenWrt board_name. Each entry carries version / url / sha256 / size / changelog, hardware floors, and config-compat info:

   "glinet_gl-mt6000": {
     "version": "1.16.0", "date": "2026-08-01",
     "url": "https://<host>/...-sysupgrade.bin",
     "sig_url": "https://<host>/...-sysupgrade.bin.sig",
     "sha256": "ae8181...", "size": 16814932,
     "min_ram_kb": 131072, "min_flash_kb": 32768,
     "min_profile_version": 1,
     "changelog": "https://<host>/changelog/1.16.0"
   },
   "tplink_tl-wr841n-v9": {
     "eol": true, "final_version": "1.12.0",
     "note": "4/32 device: final supported release. Newer firmware exceeds this hardware."
   }

   The manifest is itself signed so a tampered one can't redirect the router to a malicious image, and it is machine-generated from build output — hundreds of models means hundreds of generated entries, not hand-maintained ones. The top level also carries a monotonic generated timestamp: the router remembers the newest it has seen and refuses older manifests, closing the replay attack where a validly-signed stale manifest holds routers on a known-vulnerable version. (Deliberate local downgrades — a channel switch, a manual flash — remain allowed: those are explicit user actions, not manifest-driven.)

3. On-router ota_upgrade.sh — check → download → verify → export settings profile → apply, writing a pollable status file for the progress UI.

The security-critical core is ~8 lines of stock tooling:

[ "$(sha256sum /tmp/ota.img | cut -d" " -f1)" = "$SHA" ]            || fail hash
usign -V -q -p /etc/gargoyle/ota.pub -x /tmp/ota.sig -m /tmp/ota.img || fail bad-sig
sysupgrade -f /tmp/profile-stash.tar.gz /tmp/ota.img

(/tmp/ota.sig is downloaded from the manifest’s sig_url next to the image
— one fewer verify step than the original appended design, and fwtool
drops out of the client’s trust path entirely; it remains only in the
pre-flight device-match, where it reads image metadata.)

Note the deliberate absence of -n on the apply line: in sysupgrade's own
source, -f forces SAVE_CONFIG=1 and overrides -n (verified in
base-files sbin/sysupgrade: the CONF_IMAGE branch re-exports
SAVE_CONFIG=1). The stash archive itself defines exactly what is
preserved — and since it contains only the profile and dropbear host keys,
-f <stash> alone is the clean-flash-plus-stash semantics. Passing -n
as well would be misleading dead weight.

One pipeline, three entry points

Gargoyle's existing manual upload page (update.sh/update.js) already
performs real pre-flight today: fwtool device-match, compat_version
check, image-size vs storage check, and an explicit force checkbox. The OTA
orchestrator reuses that validation path rather than duplicating it —
manual upload, OTA, and CLI are three entry points into one verify/apply
pipeline. The one gap in the manual path is that its "preserve settings"
checkbox is raw keep-config even across a major version jump — exactly the
breakage #97 exists to prevent — so once the profile lands, the update page
routes a cross-major "preserve settings" through the same profile-stash
export the OTA path uses.

Coping with the real fleet (models, memory sizes, EOL hardware)

Gargoyle's install base spans hundreds of models across wildly different flash/RAM classes, and some hardware is simply past the point where newer firmware fits. The design handles this with one structural inversion and two honest states:

• Presence in the manifest is an allowlist, absence is the default. A board gets OTA only after it has been validated; a device whose board_name has no entry simply reports "no update available." We never have to anticipate every bad model/version combination — an unlisted device cannot be offered an image, so it cannot be bricked by one.
• Hardware floors gate before anything happens. Per-entry min_ram_kb/min_flash_kb plus on-router pre-flight: the image must fit in /tmp (RAM) with running headroom, and must fit the target flash partition. Any failure means the upgrade is never offered or never starts — a router cannot crash on an upgrade it never downloads.
• EOL is a first-class, truthful terminal state. OpenWrt itself ended 4/32 support at 19.07 and 8/64 is marginal; pretending otherwise bricks routers. An eol: true + final_version entry makes the update page say "this device has reached its final supported version" — honest, permanent, and safe.

This yields a natural three-tier fleet policy:

Tier	Hardware	Gets
A	Validated boards with comfortable specs	Full OTA + settings survival (#97)
B	Buildable but not yet OTA-validated	Settings survival only — manual flash, profile carries the config
C	EOL / below hardware floor	Frozen at final_version, told so honestly

Tier B matters: #97's profile export/import works with zero OTA infrastructure, so settings survival ships to the whole buildable fleet even before a board earns its OTA allowlist entry.

Channels and custom images

Stable / testing channels — one signed manifest per channel, not channel fields inside one manifest:

https://<host>/ota/manifest-stable.json      (+ .sig)
https://<host>/ota/manifest-testing.json     (+ .sig)

• Router setting gargoyle.ota.channel='stable' (default); UI dropdown with an honest warning on testing ("pre-release; report problems on GitHub — there is no telemetry, so we only know what you tell us").
• Same signing key for both — signing is about authenticity, not stability; only the channel pointer differs. Separate files keep failure modes clean (a botched testing manifest cannot touch stable users) and the publisher tooling is identical.
• Channel switching, testing→stable: the stable manifest will usually offer an older version than the tester runs. Raw keep-config backward is worse than forward — but the profile (RFC: Settings Profile — configuration that survives major upgrades (export intent, flash clean, regenerate) #97) already solves it: profile readers ignore unknown keys by design, so a downgrade is "flash + profile import; newer keys dropped with a logged notice." The UI says exactly that: "Return to stable — settings carry over; features from the testing build are dropped."
• Staged rollouts (graduate testing→stable to N% first) are noted as future work; overkill at current scale.

Custom images — two different cases:

• Self-built firmware (running gargoyle-build.sh yourself): the build generates its own keypair, so a self-built image carries the builder's pubkey. Two rules follow. First, manifest_url and the trusted-key set become configuration (gargoyle.ota.manifest_url + /etc/gargoyle/ota-keys.d/), defaulting to the official URL and key — and since the phase-2 manifest publisher is a script in the repo, self-builders get the entire server side for free: build, run the publisher, host two static files anywhere, and you have a private OTA channel for your own fleet. Second, the anti-clobber rule: the router knows it runs a custom build (its baked key fingerprint ≠ official) and says so — "Custom build detected — official OTA disabled unless you opt in." An official image must never silently replace someone's custom firmware; the check falls out of the key comparison for free.
• Official base + custom plugin set: on squashfs, sysupgrade wipes installed packages. Near-term honest answer: the installed-plugin list already recorded by the backup manifest (RFC: Backup coverage overhaul — declarative registry, backup manifest, and safe restore #98) and profile (RFC: Settings Profile — configuration that survives major upgrades (export intent, flash clean, regenerate) #97) drives a post-OTA "reinstall your plugins" prompt against the release's plugin feed. Long-term correct answer: this is exactly what Attended Sysupgrade (Option A) exists for — a server that bakes your package set into the image on demand. This retroactively strengthens the existing decision to keep the manifest schema ASU-compatible: "custom package sets" becomes the natural migration trigger from Option B to Option A, not a rearchitecting event.

Explicitly out of scope for v1: per-user image builds on project infrastructure (that's ASU, later, if ever); channels more granular than stable/testing (nightlies demand automation and triage capacity that doesn't exist); any unsigned "dev mode" bypass — self-builders sign with their own key, and the verify path stays fail-closed always.

None of this needs new mechanisms: channels are a second manifest file, self-built support is two hardcoded values made configurable, and custom package sets ride the plugin list #98 already records plus the ASU escape hatch already reserved.

Downgrades

The #1 real-world downgrade isn't a tester returning to stable — it's "the new release broke my wifi." Driver regressions are routine in OpenWrt, and a user whose radio got flaky after an upgrade needs a safe path back or they're stuck choosing between a broken router and TFTP recovery. An upgrade system that can only go forward isn't credible. Design:

• The manifest gains a per-board history array — the last 2–3 stable releases, same url/sha256/size/sig fields as the head entry. Consistent with the allowlist principle: history lists only validated downgrade targets — never below a per-board floor, never across a compat_version break, and a release pulled for a security hole simply doesn't appear. (This composes cleanly with anti-replay: the generated check refuses stale manifests, not older versions listed in a fresh manifest — vetted history entries are exactly the legitimate path.)
• Config carries by profile, with target-awareness. A downgrade is the mirror of an upgrade but worse for raw carry (newer raw config on firmware that has never heard of the syntax). Profile readers ignore unknown keys, and since the manifest states the target's profile capability, the UI can say before flashing exactly what won't carry ("quotas v2 rules will be dropped; everything else survives").
• Downgrade→re-upgrade is a lossless round-trip. The downgrade imports what the older version can read but preserves the newer profile file intact at /etc/gargoyle/. Downgrade to escape a bad driver, wait a release, upgrade again — everything comes back, including the settings the old version couldn't read. This one property removes most of the fear from both directions.
• UI: an "Install a previous version" disclosure under Advanced — a documented escape hatch with the profile-carry statement shown before confirming, not a prominent button.
• Honest limits: no automatic rollback on a bad flash (single-partition hardware has no A/B slot — unchanged); no OTA downgrades across a compat era (Finder + factory image, stated plainly); no downgrade below the security floor (absent from history by construction).

Bootstrap: reaching the existing fleet (old firmware, no OTA client aboard)

Everything above assumes the router runs firmware that contains the OTA client. The installed base doesn't — it runs 1.10/1.12/1.14 across several OpenWrt eras. Three facts (verified in source) shape the answer:

1. No probing is ever needed — the model is pull, not push. Every fact the upgrade decision requires (board_name, version, RAM, flash, plugin set) is already known by the router itself. New firmware self-checks against the manifest; nothing external ever inspects a router. (Server-side SSH probing was considered and rejected: it's a credential-handling anti-pattern, doesn't scale, and — decisively — can learn nothing the router doesn't already know about itself.)
2. Old sysupgrade judges the new image using metadata the new image carries. When a 1.12 router flashes a 1.16 image, it's 1.12's sysupgrade validating — against the supported_devices / compat_version lists embedded in the new image (lib/upgrade/fwtool.sh). So old-firmware acceptance is controlled from the build side: keep legacy board names in SUPPORTED_DEVICES where the jump is safe; bump DEVICE_COMPAT_VERSION with a message where it isn't, and old sysupgrade refuses by itself. The compatibility gate reaches backward in time, carried by the image — no code on the old router required. (Caveat for the honest floor: pre-17.01-era firmware (≈1.9.x) has no metadata checks at all, and 1.10's do_upgrade.sh validated almost nothing — for those, safety must come from handing the user the right file.)
3. The version landscape is finite and knowable. Each Gargoyle major maps to one OpenWrt base (1.12→19.07, 1.13→22.03, 1.14→23.05, 1.15→24.10 — the base_on_openwrt_* branches literally enumerate it), each with known board-naming and known sysupgrade behavior. This is five or six well-defined source eras, not hundreds of unknowable firmwares.

Three rungs, cheapest-first:

• Rung 1 — the manifest learns history (build hygiene, non-optional). Per-board upgradable_from routing plus legacy_board_names:

  "tplink_tl-wr1043nd-v2": {
    "version": "1.16.0", "url": "...",
    "upgradable_from": { ">=1.13": "direct", "1.10-1.12": "direct-legacy-name", "<1.10": "manual-factory" },
    "legacy_board_names": ["tl-wr1043nd-v2"]
  }

  Images keep legacy SUPPORTED_DEVICES names where safe, declare bumped compat where not. Old routers physically cannot flash an unsafe image.

• Rung 2 — a Firmware Finder for routers that can't ask. Old firmware can't fetch a manifest, but its owner can: a static page (GitHub Pages, generated from the same signed manifest — zero new infrastructure) where you pick model + version, or simply drag in your existing backup tarball (it contains enough of /etc/config to identify board and version), and it answers with exactly one file and one instruction: "flash this through your System→Update page; settings carry via the profile/legacy converter (RFC: Settings Profile — configuration that survives major upgrades (export intent, flash clean, regenerate) #97)." One guided manual flash onto the first OTA-capable release — every upgrade after that is automatic forever. For Tier C hardware the Finder tells the truth: final supported version, here's your last safe image.

• Rung 3 (phase-2 candidate, validate first) — a bootstrap plugin instead of a manual flash. Gargoyle's Plugins page (plugins.sh + gpkg) exists back to at least 1.6. A script-only, architecture-independent gargoyle-ota-bootstrap package — POSIX sh over tools old images already ship (wget, sha256sum; fwtool/usign present since ~17.01) — installable through the old firmware's own plugin UI, retrofitting check→download→verify→flash onto 1.10+. The one manual step shrinks from "flash a firmware" to "install a plugin." Scoped 17.01+ (usign present); pre-1.10 stays Finder-only. To be validated end-to-end on an old image in the vnet before it's promised (T-OTA-07).

Config survival across major versions (companion RFC: #97)

Raw keep-config is exactly what breaks on major OpenWrt bumps (swconfig→DSA, fw3→fw4, ipset→nft...), forcing users to reconfigure from scratch. That gap is owned by #97 (Settings Profile): ota_upgrade.sh exports a versioned, board-agnostic intent profile before flashing, flashes with clean config, and the new firmware regenerates its own native config from the profile at first boot — per-feature, fail-soft, never a brick and never a blank slate.

What that changes in this RFC:

• The pipeline gains an export step between verify and apply; the stash tarball passed to sysupgrade contains only the profile + dropbear host keys, nothing else.
• The manifest's min_profile_version lets the upgrade page state before flashing exactly what will carry over — and refuse combinations it cannot honor instead of discovering them at boot.
• The raw -k keep-config path remains available for same-major upgrades where it is known-safe; the profile path is the default for cross-major jumps.
• For installs running firmware that predates the profile system, RFC: Settings Profile — configuration that survives major upgrades (export intent, flash clean, regenerate) #97's legacy converter turns an old raw backup tarball into a best-effort profile, so even the one unavoidable manual reflash recovers most settings.

Anti-brick (the 20% that matters)

Fail-closed verification (any size/hash/sig mismatch aborts before sysupgrade), a device-match guard (refuse an image whose fwtool board name ≠ system board — the #1 bricking mistake), the hardware floors + allowlist gating above, free-space/power checks, config survival via #97, and honest scope: single-partition targets (e.g. MT6000) can't auto-rollback a bad flash, so pre-flight verification is the mitigation — we won't promise a revert we can't deliver.

Two paths

• A — adopt OpenWrt Attended Sysupgrade (auc/owut + ASU server): upstream-maintained, but needs us to host an ASU instance that knows Gargoyle's feed.
• B — native thin wrapper (recommended for v1): manifest + ota_upgrade.sh + a haserl progress page, ~150 lines, no build-server infra, matches how we already publish images. Manifest schema kept ASU-compatible for a later migration.

Plan

5 phases (build signing → manifest/publisher → orchestrator → web page → opt-in auto-check with no telemetry), rolled out board-by-board via the allowlist — starting with the boards we can validate on real hardware and in the vnet simulator. The bootstrap rungs slot in as: Rung 1 with phase 1 (it's build/manifest hygiene), Rung 2 with phase 2 (the Finder is generated from the manifest publisher), Rung 3 after vnet validation. Channels and the downgrade history array land with phase 2 (they are manifest/publisher features: a second signed file and extra generated entries); the anti-replay generated check and the custom-build/anti-clobber detection land with phase 3 (orchestrator logic). New vnet phase NN-ota_upgrade.test.sh (T-OTA-01…05: manifest verify, tampered-manifest reject, bad-signature reject, wrong-device reject, real older→newer flash happy-path) plus gating tests (below-floor device refused, EOL device shown terminal state, unlisted board offered nothing) and T-OTA-07: boot an old-era image in the vnet, install the bootstrap plugin through its own Plugins page, and drive a full legacy→current OTA. T-OTA-08 covers channels and replay: a router on testing switches back to stable (downgrade offered with profile-carry messaging, not raw keep-config), a stale-but-validly-signed manifest with an older generated is refused, and a self-built image (non-official key) reports custom-build status and does not offer official images without opt-in. T-OTA-09 covers downgrades: a history entry is offered under Advanced with correct what-won't-carry messaging; after downgrading, the newer profile file is preserved intact; re-upgrading restores the full configuration (the lossless round-trip); a below-floor or cross-compat version is absent from history. The end-to-end settings-survival test lives in #97's plan (T-PROF-03) and exercises this RFC's pipeline as its transport.

Full design doc (with schema, phases, risks, legality note): docs/signed-ota-upgrade-plan.md in ispyisail/gargoyle-tools; the config-survival companion doc is docs/settings-profile-upgrade-plan.md.

Feedback welcome — especially on Option A vs B, where the signed manifest should live (GitHub Releases vs a Gargoyle-owned host), which boards should seed the Tier A allowlist, the signing-key lifecycle (rotation cadence, and how a key compromise would be handled given the pubkey is baked into shipped images — likely a two-key manifest transition scheme, but this deserves review before v1), and whether the Rung-3 bootstrap plugin is worth its validation cost versus telling legacy users "one last manual flash via the Finder".

[ispyisail]

Server-side implementation scope posted: #99 (GitHub firmware + plugin hosting). It builds this RFC's publishing infrastructure ahead of the on-router client: release-hosted images, the signed manifests, and the Rung-2 Firmware Finder as a searchable GitHub Pages site, plus per-arch plugin feeds and a free-tier CI pipeline (self-hosted runner option keeps the usign key on the build host, per this RFC's key policy).

[ispyisail]

Implementation update (2026-07-17): image signing is detached, not appended — and why

The server side of this RFC (build → sign → publish → signed manifests, phases 1–2) is now live in gargoyle-firmware, and building it surfaced a fact that changes one design detail above. The RFC originally specified signing each sysupgrade image with usign -S and appending the sig via fwtool -S, with the router extracting it via fwtool -t -s — "OpenWrt's native scheme."

It turns out that slot is already taken. While testing verify tooling against real build output, fwtool -t -s on a freshly built, never-OTA-signed ath79 image happily popped a signature block — the buildroot itself signs every image: when $(BUILD_KEY) exists (and our trees auto-generate one, "Local build key"), include/image-commands.mk signs the image and appends a ucert chain via fwtool -S. So every image already ends in [metadata trailer][build-key ucert signature].

Appending our OTA signature on top would mean:

• Stock verification breaks the day it's enforced. lib/upgrade/fwtool.sh's fwtool_check_signature pops the last trailer and hands it to ucert -V. Our bare usign sig in that position fails ucert validation. Today this is a no-op — shipped images carry no /usr/bin/ucert and REQUIRE_IMAGE_SIGNATURE is unset (both verified against a real built rootfs manifest and the target configs) — but it plants a landmine under any future move to stock signature enforcement.
• The artifact mutates. An appended trailer changes the file, so its sha256 no longer matches the build's own sha256sums, hurting reproducibility and making "is this the file the build produced?" needlessly hard to answer.
• Confusing failure modes. fwtool -t -s on a not-OTA-signed image extracts the build's ucert block, and usign -V then dies with a misleading "Failed to decode signature" rather than "no signature" — observed live while building the verify script.

What shipped instead: detached <image>.sig sidecars, published next to each sysupgrade image, with the manifest carrying a sig_url per entry (and the manifest generator refusing to list any sysupgrade image lacking one — same fail-closed rule as a missing sha256). The on-router core gets simpler: download image + sig from manifest URLs, then size → sha256 → usign -V — fwtool drops out of the client's trust path entirely (it stays in pre-flight, reading image metadata for the device-match guard). Published bytes are exactly what the build produced, native build signing stays untouched, and the full chain — Pages manifest fetch → manifest sig verify against the baked key → entry lookup → image+sig download → size/sha256/usign verify → device-match metadata read — has been exercised end-to-end against the live test release from public URLs only.

Long-term recommendation (a phase-1 build-side decision, not urgent): the architecturally cleanest endgame is to make the official OTA key the buildroot's BUILD_KEY for release builds. Then images come out natively signed with the official key in OpenWrt's own ucert format, and on-router enforcement becomes stock: ship ucert, drop the official pubkey in /etc/opkg/keys, set REQUIRE_IMAGE_SIGNATURE=1 — no custom verify code at all, and the detached sidecars become a belt-and-suspenders extra rather than the primary mechanism. That's a bigger change (key custody wired into the build trees, self-builder implications already discussed above), so the detached scheme is the right v1: it works now, breaks nothing, and loses no work if BUILD_KEY adoption lands later.

RFC body updated accordingly (mechanism §1, the manifest schema example, and the verify core).