Skip to content

ci/nightly: manifest aggregator + 90-build retention sweep#92

Merged
widgetii merged 1 commit into
masterfrom
ci/nightly-manifest-and-cleanup
May 23, 2026
Merged

ci/nightly: manifest aggregator + 90-build retention sweep#92
widgetii merged 1 commit into
masterfrom
ci/nightly-manifest-and-cleanup

Conversation

@widgetii
Copy link
Copy Markdown
Member

Summary

PR-Bld-B of three in the mirror of OpenIPC/firmware's nightly redesign. Adds the two workflows that turn dated nightly-* releases (from #91) into a queryable index served via GitHub Pages.

  • manifest.ymlworkflow_run downstream of Build (master.yml). Runs .github/scripts/enrich_manifest.py to enumerate dated releases, keep the 90 newest, parse sha=/built_at= from the release body (set by ci/nightly: SHA-gate cron, publish dated nightly-* releases, BUILD_ID env #91), and emit:
    • manifest.json — rich JSON for hosts / agents / CI.
    • manifest.flat — whitespace-delimited (build_id platform flash size url plus @channel records), parseable by pure busybox awk for on-device sysupgrade in Phase 2.
  • cleanup.yml — Mondays 06:00 UTC (offset +1h from firmware's 05:00). Prunes dated releases beyond the 90 newest; re-triggers manifest.
  • Both concurrency-grouped on gh-pages-manifest so they can't race the gh-pages push.

Bimodal asset parser

master.yml produces two filename forms depending on the underscore count of the matrix entry:

Matrix entry COMMON Filename Manifest platform key
ssc338q_fpv 1 openipc.ssc338q-nor-fpv.tgz ssc338q_fpv
ssc338q_fpv_caddx-fly 2 ssc338q_fpv_caddx-fly-nor.tgz ssc338q_fpv_caddx-fly
t31_lite_xiaomi-mjsxj03hl-jxq03 2 t31_lite_xiaomi-mjsxj03hl-jxq03-nor.tgz t31_lite_xiaomi-mjsxj03hl-jxq03

The full per-device key is what flows into manifest.flat. Phase 3 (model-aware BUILD_PLATFORM field on the camera) will let sysupgrade --list-builds filter against that precise key.

Retry hardening

gh() wrapper has the same 4-attempt backoff (0/5/15/40s) for transient GitHub API failures as OpenIPC/firmware#2129. Fast-fail on permanent 404.

URLs after merge

(gh-pages branch already bootstrapped; Pages auto-enabled.)

First-run state

When no dated nightly-* releases exist yet (i.e. right after merge, before the next cron), enrich_manifest.py writes an explicit empty manifest with a "first cron will populate" comment. Already validated locally against the live repo (returns manifest: 0 builds (empty index)).

Test plan

  • CI on this PR (no buildable code; only YAML parse + script compile).
  • After merge, manually dispatch manifest.yml; expect green run and empty manifest.{json,flat} committed to gh-pages.
  • After PR-Bld-A merges and the next Build cron completes, workflow_run fires manifest.yml automatically and the index populates.
  • curl https://openipc.github.io/builder/manifest.json | jq '.channels.nightly' returns the just-published build_id.
  • curl https://openipc.github.io/builder/manifest.flat | awk '\$2==\"gk7205v200_fpv_caddx-fly\" && \$3==\"nor\" {print \$NF}' returns the per-device asset URL.

See also

🤖 Generated with Claude Code

PR-Bld-B in the mirror of OpenIPC/firmware's nightly redesign. Adds
the two workflows that turn dated nightly-* releases (from PR-Bld-A)
into a queryable index served via GitHub Pages.

`manifest.yml` — triggers on successful `Build` workflow_run completion
(or manual dispatch). Runs `.github/scripts/enrich_manifest.py` to
enumerate dated releases, keep the 90 newest, parse the body
(sha/short/built_at from PR-Bld-A) and asset list, then emit:
  - `manifest.json` — rich JSON for hosts/agents/CI.
  - `manifest.flat` — whitespace-delimited columns
    `build_id platform flash size url` plus `@channel` records,
    parseable by pure busybox `awk` (no jq, no jsonfilter) for
    on-device sysupgrade in Phase 2.
Both committed to the `gh-pages` branch (already bootstrapped).

`cleanup.yml` — Mondays 06:00 UTC (offset by +1h from firmware's
05:00 UTC cleanup). Deletes dated nightly releases beyond the 90
newest via `gh release delete --cleanup-tag`, then re-triggers
manifest.yml.

`enrich_manifest.py` differs from firmware's only in the asset parser:
builder produces TWO filename forms (per `master.yml`'s COMMON awk
trick that counts underscores in the matrix entry name):
  - **Compound** `<soc>_<variant>_<vendor>-<model>-nor.tgz` →
    platform key = the full string. This is the common case
    (per-device builds).
  - **Simple** `openipc.<soc>-<flash>-<variant>.tgz` (firmware-style,
    when matrix name has one underscore) → platform key =
    `<soc>_<variant>`.

The platform key gets carried into manifest.flat as a single column.
sysupgrade's existing `awk '$2==p'` lookup works unchanged; Phase 3
(model-aware BUILD_PLATFORM field in os-release) will let cameras
filter against the full per-model key.

URLs after merge:
  https://openipc.github.io/builder/manifest.json
  https://openipc.github.io/builder/manifest.flat

(`gh-pages` branch + Pages config already set up; mirrors firmware-
side setup.)

Concurrency-grouped `gh-pages-manifest` so manifest.yml and
cleanup.yml can't race the gh-pages push.

Includes the retry hardening from firmware OpenIPC/firmware#2129
(4-attempt backoff for transient GitHub API 401/5xx, fast-fail on
permanent 404).

First-run behaviour validated locally: empty index emits explicit
placeholder. Parser unit-tested across 7 synthetic filenames covering
compound, simple, and edge cases.

See ~/.claude/plans/mirror-nightly-redesign-to-builder.md.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@widgetii widgetii merged commit 6399426 into master May 23, 2026
@widgetii widgetii deleted the ci/nightly-manifest-and-cleanup branch May 23, 2026 20:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant