ci/nightly: manifest aggregator + 90-build retention sweep#92
Merged
Conversation
PR-Bld-B in the mirror of OpenIPC/firmware's nightly redesign. Adds
the two workflows that turn dated nightly-* releases (from PR-Bld-A)
into a queryable index served via GitHub Pages.
`manifest.yml` — triggers on successful `Build` workflow_run completion
(or manual dispatch). Runs `.github/scripts/enrich_manifest.py` to
enumerate dated releases, keep the 90 newest, parse the body
(sha/short/built_at from PR-Bld-A) and asset list, then emit:
- `manifest.json` — rich JSON for hosts/agents/CI.
- `manifest.flat` — whitespace-delimited columns
`build_id platform flash size url` plus `@channel` records,
parseable by pure busybox `awk` (no jq, no jsonfilter) for
on-device sysupgrade in Phase 2.
Both committed to the `gh-pages` branch (already bootstrapped).
`cleanup.yml` — Mondays 06:00 UTC (offset by +1h from firmware's
05:00 UTC cleanup). Deletes dated nightly releases beyond the 90
newest via `gh release delete --cleanup-tag`, then re-triggers
manifest.yml.
`enrich_manifest.py` differs from firmware's only in the asset parser:
builder produces TWO filename forms (per `master.yml`'s COMMON awk
trick that counts underscores in the matrix entry name):
- **Compound** `<soc>_<variant>_<vendor>-<model>-nor.tgz` →
platform key = the full string. This is the common case
(per-device builds).
- **Simple** `openipc.<soc>-<flash>-<variant>.tgz` (firmware-style,
when matrix name has one underscore) → platform key =
`<soc>_<variant>`.
The platform key gets carried into manifest.flat as a single column.
sysupgrade's existing `awk '$2==p'` lookup works unchanged; Phase 3
(model-aware BUILD_PLATFORM field in os-release) will let cameras
filter against the full per-model key.
URLs after merge:
https://openipc.github.io/builder/manifest.json
https://openipc.github.io/builder/manifest.flat
(`gh-pages` branch + Pages config already set up; mirrors firmware-
side setup.)
Concurrency-grouped `gh-pages-manifest` so manifest.yml and
cleanup.yml can't race the gh-pages push.
Includes the retry hardening from firmware OpenIPC/firmware#2129
(4-attempt backoff for transient GitHub API 401/5xx, fast-fail on
permanent 404).
First-run behaviour validated locally: empty index emits explicit
placeholder. Parser unit-tested across 7 synthetic filenames covering
compound, simple, and edge cases.
See ~/.claude/plans/mirror-nightly-redesign-to-builder.md.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
PR-Bld-B of three in the mirror of OpenIPC/firmware's nightly redesign. Adds the two workflows that turn dated
nightly-*releases (from #91) into a queryable index served via GitHub Pages.manifest.yml—workflow_rundownstream ofBuild(master.yml). Runs.github/scripts/enrich_manifest.pyto enumerate dated releases, keep the 90 newest, parsesha=/built_at=from the release body (set by ci/nightly: SHA-gate cron, publish dated nightly-* releases, BUILD_ID env #91), and emit:manifest.json— rich JSON for hosts / agents / CI.manifest.flat— whitespace-delimited (build_id platform flash size urlplus@channelrecords), parseable by pure busyboxawkfor on-devicesysupgradein Phase 2.cleanup.yml— Mondays 06:00 UTC (offset +1h from firmware's 05:00). Prunes dated releases beyond the 90 newest; re-triggers manifest.gh-pages-manifestso they can't race the gh-pages push.Bimodal asset parser
master.ymlproduces two filename forms depending on the underscore count of the matrix entry:ssc338q_fpvopenipc.ssc338q-nor-fpv.tgzssc338q_fpvssc338q_fpv_caddx-flyssc338q_fpv_caddx-fly-nor.tgzssc338q_fpv_caddx-flyt31_lite_xiaomi-mjsxj03hl-jxq03t31_lite_xiaomi-mjsxj03hl-jxq03-nor.tgzt31_lite_xiaomi-mjsxj03hl-jxq03The full per-device key is what flows into
manifest.flat. Phase 3 (model-awareBUILD_PLATFORMfield on the camera) will letsysupgrade --list-buildsfilter against that precise key.Retry hardening
gh()wrapper has the same 4-attempt backoff (0/5/15/40s) for transient GitHub API failures as OpenIPC/firmware#2129. Fast-fail on permanent 404.URLs after merge
(
gh-pagesbranch already bootstrapped; Pages auto-enabled.)First-run state
When no dated
nightly-*releases exist yet (i.e. right after merge, before the next cron),enrich_manifest.pywrites an explicit empty manifest with a "first cron will populate" comment. Already validated locally against the live repo (returnsmanifest: 0 builds (empty index)).Test plan
manifest.yml; expect green run and emptymanifest.{json,flat}committed togh-pages.Buildcron completes,workflow_runfiresmanifest.ymlautomatically and the index populates.curl https://openipc.github.io/builder/manifest.json | jq '.channels.nightly'returns the just-publishedbuild_id.curl https://openipc.github.io/builder/manifest.flat | awk '\$2==\"gk7205v200_fpv_caddx-fly\" && \$3==\"nor\" {print \$NF}'returns the per-device asset URL.See also
~/.claude/plans/mirror-nightly-redesign-to-builder.md.🤖 Generated with Claude Code