RFC: GitHub firmware + plugin hosting — searchable Finder, per-arch plugin feeds, free-tier CI

[ispyisail]

Status: Ready for review · implements the server side of #62 (signed OTA) and its Rung-2 Firmware Finder · full doc: docs/firmware-hosting-plan.md in ispyisail/gargoyle-tools

Goal

Host every Gargoyle firmware image and every plugin package on GitHub, with a searchable download page (hundreds of device/version combinations), a plugin feed the firmware points at out of the box, and CI that builds and publishes all of it — on the free tier.

One assumption to correct first

Plugins are not identical across firmwares: binary plugin packages are per-architecture (aarch64_cortex-a53 for the MT6000, mips_24kc for most ath79, ...) and per-Gargoyle-major. What is true: one arch's plugin set serves every device of that arch, and script-only plugins (arch all) are shared. So the feed layout is per-release + per-arch — a dozen directories, not hundreds.

The load-bearing GitHub facts

Surface	Limits	Use for
Releases (assets)	2 GiB/file, no meaningful total cap, CDN-backed, free egress	firmware images, plugin .ipks
Pages	1 GB site, ~100 GB/month soft bandwidth	Finder search page, manifest-*.json, feed index files
Actions (public repo)	unlimited free minutes, 4 vCPU/16 GB, ~14 GB free disk (~45–60 GB after cleanup), 6 h/job, 10 GB cache	build + publish pipeline
Git LFS	1 GB storage / 1 GB bandwidth free	not usable — avoid

Two conclusions: big binaries live in Releases (never git, never Pages), and GitHub itself cannot be customized for search — release pages aren't searchable by device. The search experience is a static Pages site we control: CI generates an index.json (device, board_name, aliases, target, version, channel, sha256, URL) and a single page filters it client-side as you type. This page is the Firmware Finder from #62 (Rung 2) — same artifact, one build.

Repository layout

A new dedicated repo (suggest gargoyle-firmware), separate from source so release assets, Pages and publish workflows never tangle with source PRs:

gargoyle-firmware/
├── .github/workflows/build-release.yml    # matrix build → release assets
├── .github/workflows/publish-site.yml     # index.json + manifests + Pages
├── site/                                  # the Finder (static, client-side search)
├── scripts/make-index.sh | make-manifest.sh | make-feed.sh
└── devices/                               # per-board metadata: floors, EOL, aliases, upgradable_from

The devices/ directory makes fleet-gating metadata versioned, reviewable data — marking a device EOL becomes a PR, not a script edit.

The plugin feed

Correction (Phase 2 implementation): the original draft above assumed opkg would resolve an absolute URL in a package's Filename: field, letting the index live on Pages while packages rode the Release CDN. Reading opkg's actual source (libopkg/opkg_download.c's opkg_download_pkg()) shows that's wrong: opkg concatenates the package list URL and every package's Filename: onto the exact same src/gz <name> <url> base with a plain "%s/%s", no absolute-URL case. Index and packages must live at the same base URL — so both now live together in the same release, no Pages involvement for this part:

https://github.com/<user>/gargoyle-firmware/releases/download/plugins-<major>-<arch>/Packages.gz
https://github.com/<user>/gargoyle-firmware/releases/download/plugins-<major>-<arch>/<pkg>.ipk

scripts/make-feed.sh builds Packages/Packages.gz by downloading every .ipk in a plugins-<major>-<arch> release (small enough — a few KB to low hundreds of KB — that pulling real control-file fields and hashing actual bytes directly is cheap and more correct than guessing from filenames) and uploads the index back into that same release. The apk migration keeps the same same-base shape (APKINDEX + signed packages; the gapk signing runbook already covers keys).

Firmware change (small, one place): the default src/gz gargoyle_core <url> line in the opkg.conf template that gpkg-uci-defaults installs — pointed directly at the plugins-<major>-<arch> release's own download base for the matching major+arch at build time. Users keep add/remove sources in the Plugins page exactly as today.

CI on the free tier — honest assessment

• Minutes: solved — public repos get unlimited standard-runner minutes.
• Disk is the real constraint: full target builds want 30–50 GB; runners give ~14 GB, recoverable to ~45–60 GB with the standard cleanup pattern. Workable, tight for the biggest targets.
• Time is the second: 6 h/job on 4 vCPU; warm dl/ + ccache caches (10 GB/repo) keep rebuilds well inside it, from-scratch toolchains may flirt with the limit.
• Pragmatic hybrid (recommended): register the existing build box as a self-hosted runner (free) for heavy matrix jobs; hosted runners for light jobs (indexing, signing, manifests, Pages deploy). This also settles key custody the right way: the usign private key never leaves the build host (RFC: Signed OTA firmware upgrade (usign/fwtool/sysupgrade + signed manifest) #62's stated policy) instead of living in an Actions secret.

Pipeline per release

tag push (v1.16.0)
  → matrix build per target (self-hosted heavy / hosted light)
  → collect images + plugin ipks
  → usign-sign images (fwtool append) on the key-holding runner
  → upload assets to the release
  → make-index / make-manifest / make-feed
  → deploy site + JSON + feed indexes to Pages

Phases

1. P1 — hosting + Finder (~2–3 days, no firmware changes): repo, release layout, make-index.sh, searchable Pages site. Immediately useful.
2. P2 — plugin feed (~2 days + a firmware PR): per-major/per-arch feeds, opkg.conf default; verified on a real router + in the vnet.
3. P3 — CI builds (~2–3 days): matrix workflow, self-hosted runner, caches, first end-to-end tag→release→Pages run.
4. P4 — OTA integration: make-manifest.sh emits RFC: Signed OTA firmware upgrade (usign/fwtool/sysupgrade + signed manifest) #62's signed manifest-{stable,testing}.json with floors/EOL/history from devices/; the Finder gains the drag-in-a-backup-tarball identifier (Rung 2 complete). Lands alongside RFC: Signed OTA firmware upgrade (usign/fwtool/sysupgrade + signed manifest) #62 phase 2.

Risks / open questions

• Pages bandwidth (~100 GB/month soft): mitigated by keeping all binaries on the release CDN — indexes only on Pages.
• 6 h / 2 GiB edges: validate the largest targets early on the self-hosted runner; no single Gargoyle asset approaches 2 GiB.
• Key custody: self-hosted runner recommended (key stays home); flag if the project prefers Actions secrets for fully-cloud builds.
• Backfill: import existing published versions as back-dated releases so the Finder is complete from day one — mechanical via gh release create.

Feedback welcome — especially on the dedicated-repo split, self-hosted vs fully-hosted builds (key custody), and whether the Finder should launch with the backfilled historical versions or current releases only.

[ispyisail]

Phase 1 shipped and live: https://github.com/ispyisail/gargoyle-firmware

• Finder site: https://ispyisail.github.io/gargoyle-firmware/ (currently empty — 0 releases published yet, which is expected; the pipeline correctly renders the empty state rather than erroring)
• make-index.sh joins release assets against devices/*.json, grouping multi-image releases (sysupgrade + factory variants) under one entry per board/version
• Two gatherers with identical output shape: gather-local-assets.sh (offline testing against a staging directory) and gather-github-assets.sh (production, resolves sha256 via .sha256 sidecar assets rather than downloading full images)
• devices/ registry seeded with 3 real devices (GL-MT6000, Archer C7 v2, an EOL TL-WR841N/ND v9 example) — schema already carries the floors/EOL/upgradable_from fields Phase 4's OTA manifest will need
• .github/workflows/publish-site.yml auto-regenerates and deploys on release publish, devices/ changes, or manual dispatch

Everything above was tested against real behavior before going live: a 13-assertion headless-browser test of the Finder's search/filter/EOL-badge/copy-to-clipboard UX, the gatherer tested against a mocked gh + local sidecar server, and the README's own quickstart dogfooded in a clean checkout. One real bug was caught and fixed in the process (an empty devices/ directory broke make-index.sh via a jq glob-fallback edge case — exactly the kind of day-one state a fresh repo starts in).

Next up: Phase 2 (plugin feeds) whenever you're ready.

[ispyisail]

Phase 2 shipped and live: plugin feed generation via scripts/make-feed.sh.

A real correction surfaced while building this, now fixed everywhere it was stated: the original plan assumed opkg would accept an absolute URL in a package's Filename: field, letting the feed index live on Pages while the .ipks rode the Release CDN. Reading opkg's actual source (libopkg/opkg_download.c) shows that's wrong — opkg concatenates the package list URL and every package's Filename: onto the exact same src/gz <name> <url> base, with no absolute-URL case. So the index and the packages it lists now live together in the same plugins-<major>-<arch> release; publish-site.yml gained a build-feed job that generates Packages/Packages.gz and uploads them back into that release. RFC #99's body and the design doc are corrected to match.

make-feed.sh downloads each .ipk directly (they're small — a few KB to low hundreds of KB, unlike firmware images) to read its real control-file fields (Depends, Provides, Description) rather than guessing from the filename, and hashes the actual bytes. Verified byte-for-byte against a real OpenWrt-built Packages index for three real Gargoyle packages, including one with a multi-line Description (the harder case, since that field's continuation-line formatting has to match exactly).

Also found and fixed, only because I tested under the actual /bin/sh (dash) rather than an interactive bash shell: dash's builtin echo interprets backslash escapes by default, so echo "$json_var" | jq ... silently turned an escaped \n inside a JSON string into a raw newline byte, which the next jq stage then rejected as invalid JSON. Every such site (across all three shell scripts, including the already-shipped gather-github-assets.sh) now uses printf '%s\n' instead. Confirmed live: the corrected build-feed job ran cleanly on the real GitHub Actions runner (Ubuntu 24.04), correctly reporting "nothing to do" since no plugin releases exist yet.

Next up: Phase 3 (firmware build workflow) whenever you're ready.