Skip to content

ci: build the SDK in one workflow, core -> media via needs - #22

Merged
jcelerier merged 1 commit into
masterfrom
ci/single-sdk-workflow
Aug 3, 2026
Merged

ci: build the SDK in one workflow, core -> media via needs#22
jcelerier merged 1 commit into
masterfrom
ci/single-sdk-workflow

Conversation

@jcelerier

Copy link
Copy Markdown
Member

Prototype: one workflow instead of the core.yml + media.yml pair, with core-<platform> -> media-<platform> wired by needs.

Why needs alone isn't enough

needs orders the two halves inside a run. The rolling continuous prerelease is what shares one core build across runs — every later push and every PR downloads a core built hours earlier by a different run. So the release handoff stays exactly as it is; only the cascade goes away.

What the split cost that this doesn't

  • The workflow_run cascade. It fires on completed, not succeeded, so a cancelled core still started one. That needed a conclusion guard (35c90c0) plus its own concurrency group so a doomed cascade wouldn't evict a live run.
  • Two entry points on master racing in one concurrency group — six cancelled checks on every master commit (f8e4a28).
  • resolve-core.sh's skip mode, which existed only to defer to the cascade. Every call here passes is_master=false, so it goes unused (the script keeps it for the manual build.yml escape hatch).
  • Building the core twice on a PR. A PR can't publish a release, so core.yml built the core and media.yml rebuilt it inline. On PR faust: pin the version and carry our fixes as PR picks #19, core-linux-x86_64 spent 53min and media-linux-x86_64 then spent 304min on a STAGE=full core+media build (Extract core into prefix skipped -> mode=inline). The core job now hands its tarball to media as a run-scoped artifact.

Use cases

PUBLISH_CORE PUBLISH_SDK SDK_RELEASE core jobs
pull_request false false build if hash unpublished, hand over via artifact
push master true true continuous (rolling) short-circuit or build+publish
push sdk* tag false true ${{ github.ref_name }} (not rolling) short-circuit on the published core
workflow_dispatch true true continuous force_core_rebuild input available

Two deliberate changes:

  • Core jobs now run on tag pushes. core.yml never triggered on tags, so a tag build took media's inline path and rebuilt the whole core inside media. Now it reuses the published core, so tag builds get faster and can't diverge from what master published.
  • PUBLISH_CORE excludes tags, so tagging an older commit can't clobber the rolling core with an older one. Everything else still publishes it, as before.

Failure semantics — finer, not coarser

case before after
a linux/macos core leg fails core workflow fails -> cascade guard skips all six media legs only that platform's media is skipped
core-windows fails continue-on-error -> workflow still succeeds -> media ran continue-on-error kept, media-windows: if: !cancelled() -> still runs, falls back to the published core
two merges in quick succession second cancels the first; the cancelled core's cascade cancelled the live media run, then skipped one sdk-<ref> group; the second push simply supersedes the first

Verification

Job/step structure was diffed against the originals with a yaml-cpp parser: all six core/media jobs match the old ones step-for-step, in order, with the same if: conditions. The only additions are Hand core to media / Core handed over by ..., and the mode != 'skip' guards dropped as unreachable.

This PR run exercises the pull_request path end to end. The master and tag paths are expression-level changes on the same steps and scripts; the plan job prints the resolved values for whatever event it runs under, so the master/tag rows can be confirmed from the first run after merge (or via workflow_dispatch) rather than inferred.

Since core.yml/media.yml are deleted here, this PR does not double-build: only sdk.yml runs. Reverting is a single git revert.

Open question

media-<plat> needs the whole core-<plat> matrix, so media-linux-x86_64 waits for core-linux-aarch64 too. That's still finer than today (the cascade waits for all six core legs), and irrelevant when core short-circuits. Splitting into per-arch job pairs, or reusable workflows, would remove it if the wait ever matters.

Replaces core.yml + media.yml with a single sdk.yml holding a
core-<platform> -> media-<platform> pair per platform, wired with `needs`.

`needs` orders the two halves inside a run; it does not replace the rolling
`continuous` prerelease, which is what shares one core build ACROSS runs. Both
are still there, they just solve different problems: the release still carries
the hashed core, and a leg whose hash is already published still short-circuits
in seconds.

What the split cost that this does not:

  - the workflow_run cascade. It fires on *completed*, not *succeeded*, so a
    cancelled core still started one; that needed a conclusion guard (35c90c0)
    and its own concurrency group to stop a doomed cascade evicting a live run.
  - two entry points on master racing in one concurrency group, which left six
    cancelled checks on every master commit (f8e4a28).
  - resolve-core.sh's `skip` mode, which only existed to defer to the cascade.
    Every call here passes is_master=false, so it goes unused.
  - building the core TWICE on a pull request. A PR cannot publish a release,
    so core.yml built the core and media.yml then rebuilt it inline: on PR #19,
    53min in core-linux-x86_64 plus a 304min core+media build in media. The
    core job now hands its tarball over as a run-scoped artifact.

Also new: core jobs run on tag pushes (core.yml did not), so a tag reuses the
published core instead of taking media's inline path. PUBLISH_CORE excludes
tags so tagging an older commit cannot clobber the rolling core.

Failure semantics get finer, not coarser: a failed core leg used to fail the
core workflow and skip all six media legs; now it only skips its own platform.
core-windows keeps continue-on-error, and media-windows gets `if: !cancelled()`
so it still runs and falls back to the published core, as it did before.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@jcelerier
jcelerier merged commit d5bad12 into master Aug 3, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant