Publish nightlies to GHCR

[SeanTAllen]

Today, nightly tool binaries go to our Cloudsmith nightlies repository and nowhere else. We're adding GHCR as a second destination, storing nightlies as ORAS artifacts under a nightly/ path segment, and trimming the GHCR copy at 90 days. Cloudsmith publishing stays as it is.

Unlike the release-channel migration, nightlies don't land in per-tool GitHub Releases — that would clutter each tool's release list with daily noise. GHCR is already where each tool stores its container images and CI builder images. Nightlies fit there too.

Divergences from today

• Today: nightly binaries go only to Cloudsmith. After: nightly binaries go to both Cloudsmith and GHCR. Nothing about Cloudsmith publishing changes.
• Today: GHCR holds no nightly artifacts, so there's nothing to retain. After: GHCR nightly artifacts are trimmed at 90 days.

Settled

• Store nightlies as ORAS artifacts in GHCR, one OCI image per (tool, platform) under a nightly/ path segment. Tag matches the YYYYMMDD date string each tool's nightly CI already passes to its Cloudsmith push. ponyc's x86-64 alpine3.23 nightly for 2026-04-12 lives at ghcr.io/ponylang/nightly/ponyc-x86-64-unknown-linux-alpine3.23:20260412.
• The nightly/ path segment is deliberate. It mirrors the Cloudsmith split between ponylang/nightlies and ponylang/releases, and it reserves room for releases to move to GHCR-as-ORAS later (under ghcr.io/ponylang/release/...) as a purely additive change. No rename of existing image names ever required.
• Dual-publish. Each tool's nightly CI pushes the same archive — already on disk from the existing Cloudsmith step — to both destinations. No extra build time, one extra upload per platform. The YYYYMMDD date string is computed once per platform job and passed to both the Cloudsmith and GHCR push steps, so a midnight-UTC rollover can't split the two destinations across different dates.
• Push mechanism: a Python script against the OCI distribution spec, stdlib only. Same posture as the github_release.py script releases use. Python3 availability in ponyup's builder image was verified during the releases work; other tools' builder images need the same probe before their PR lands (see pre-implementation verification). The script uploads the archive blob, uploads a config blob declaring the artifact type, PUTs the manifest, and sets the GHCR package to public. Every HTTP call checks for 2xx; non-2xx prints status and response body to stderr and exits non-zero. The visibility call is idempotent, and any failure — visibility or otherwise — fails the push. There is no code path that produces a private nightly. Authenticates against GHCR with GITHUB_TOKEN and packages: write on the job.
• Script location: each tool's .ci-scripts/, per-tool copy. Matches the github_release.py precedent.
• Media types:
  • Artifact type: application/vnd.ponylang.nightly.v1.
  • Config: application/vnd.ponylang.nightly.config.v1+json, body carries {tool, platform, version, sha512, built_at}. Ponyup can read metadata straight off the manifest response without fetching the archive.
  • Layer: application/vnd.ponylang.nightly.archive.v1.tar+gzip for Linux/Darwin, application/vnd.ponylang.nightly.archive.v1.zip for Windows.
• Hashes: GHCR accepts only sha256:-prefixed digests on manifests and blobs in practice, so the OCI manifest carries the SHA-256 digest of the archive as registry-enforced integrity. SHA-512 of the archive goes in the config blob, continuing the "SHA-512 everywhere" convention from the releases plan for any ponyup code path that wants to cross-check.
• Trim runs as the final job of the nightly workflow, gated on every platform publish job completing. Uses snok/container-retention-policy (pinned to a commit SHA, matching the pinning convention ponyc already uses for this action) with tag-selection: tagged and a 90-day cutoff — ninety days covers the tail of users who lag on picking up nightlies; older than that is debugging archaeology and can fall back to Cloudsmith. image-names either enumerates the per-(tool, platform) names or uses a glob against the nightly/ path segment, depending on what the action accepts (see pre-implementation verification). A YAML comment on the prune job names the invariant: every platform publish job must appear in the prune job's needs: list before any artifact is eligible for deletion. That comment lives on the prune job, not on each platform job — the prune job's behavior is what breaks if someone adds a platform without updating needs:.
• Discovery from GHCR is primarily ponyup's job. The OCI endpoint is also usable by hand for debugging, but that's not a first-class UX concern for this plan.
• Scope: ponyc, corral, ponyup, changelog-tool.

General shape, per tool

Each tool's nightlies.yml has its own platform matrix — ponyc has the most (roughly a dozen jobs across Linux distros, macOS, and Windows); ponyup and corral have six apiece; changelog-tool fewer. The shape of the change is the same regardless of how the matrix is structured:

1. Python push script invoked in each platform job. Runs alongside the existing cloudsmith push raw step, reusing the archive the Cloudsmith push already produced. Same version string. Pushes to ghcr.io/ponylang/nightly/<tool>-<triple>:<date>. Workflow gains packages: write.
2. Final prune job. Depends on all platform publish jobs, deletes GHCR artifacts older than 90 days. Pattern is the existing snok/container-retention-policy usage in ponyc/.github/workflows/build-nightly-image.yml, adapted to the nightly/<tool>-<triple> image names and a 90-day cutoff instead of 1 day.

Details — how auth is set up, the exact config-blob field set, exactly how the prune job names its images — may vary per tool.

Pre-implementation verification

Before the first implementing PR lands, verify on an experimental image name:

• Python3 is available in each non-ponyup tool's nightly builder image: corral's shared builder, ponyc's alpine and ubuntu builders, changelog-tool's builder. Ponyup's builder was verified during the releases work.
• snok/container-retention-policy accepts multi-segment image names (nightly/<name>). Ponyc's current usage is on single-segment names, so this is new ground. Confirm cut-off: 90d is syntactically valid (the existing usage is 1d) and determine whether image-names supports a glob across path segments or requires enumeration.
• GHCR's package-visibility REST API accepts URL-encoded multi-segment package names.
• OCI push against multi-segment package names on GHCR works under our auth setup. Multi-segment is a known-supported GHCR feature, used by Homebrew and others, but worth confirming end-to-end with the exact HTTP shape our Python script will use.

If any of these fail in ways we can't work around, fall back to channel-in-package-name (ghcr.io/ponylang/nightly-<tool>-<triple>). The path segment is a preference, not load-bearing.

Out of scope

• Ponyup's read-side — fetching nightlies from GHCR, parsing OCI manifests, verifying digests. Separate plan.
• Whether releases actually move to GHCR-as-ORAS. This plan reserves the namespace; it doesn't commit to that direction.

[SeanTAllen]

ponyup is live — a model for the other tools

ponyup now dual-publishes its nightlies to GHCR alongside Cloudsmith. The work landed in #433, and the first scheduled nightly after it merged confirmed the whole thing in production: all six platforms published, the prune job ran clean, and the artifacts pull anonymously with their hashes intact. I'm writing the shape down so corral, ponyc, and changelog-tool can be cut from the same pattern.

The whole change is additive. Cloudsmith publishing is untouched. Every platform's nightly build, right after its existing Cloudsmith push, ships the same archive a second time to ghcr.io/ponylang/nightly/ponyup-<triple>:<YYYYMMDD>.

The pieces to copy

A stdlib-only push script, one per tool. Same posture as the github_release.py we already carry for the releases work: no pip install on any runner, so it runs everywhere from the musl container to the Windows host. It does the OCI distribution dance by hand — grab a pull/push token, upload the archive as a blob and a small JSON config blob, then PUT the manifest. The manifest carries artifactType: application/vnd.ponylang.nightly.v1; the config blob (application/vnd.ponylang.nightly.config.v1+json) holds {tool, platform, version, sha512, built_at}; the archive is the single layer, typed …archive.v1.tar+gzip on Linux/macOS and …archive.v1.zip on Windows. GHCR enforces the sha256 on the manifest; we keep the sha512 in the config blob for cross-checking. The script is tool-agnostic — copy .ci-scripts/release/ghcr_nightly.py from #433 verbatim, no edits, and pass the tool as an argument: ghcr_nightly.py push <tool> <triple> <version> <archive>. It reads GITHUB_TOKEN and GITHUB_REPOSITORY from the environment (and GITHUB_ACTOR if present).

Call it from the existing nightly script, after the Cloudsmith push. It reuses the date and the archive the Cloudsmith step already produced, so both destinations get the same bytes under the same date string. Ordering is deliberate: GHCR runs second. If GHCR ever falls over, it can't take the primary download source down with it.

Workflow plumbing. Bump the workflow to permissions: packages: write, and put GITHUB_TOKEN on every job — it's a secret, not one of the default env vars, so it has to be passed in by hand. The one wrinkle is the arm64 musl job that runs through a bare docker run instead of a container:. That one doesn't inherit the runner environment, so it needs -e GITHUB_TOKEN and -e GITHUB_ACTOR handed in explicitly. The container and host-runner jobs get GITHUB_REPOSITORY and GITHUB_ACTOR for free.

A prune job. Ages out GHCR tags older than 90 days with snok/container-retention-policy@3b0972b2276b171b212f8c4efbca59ebba26eceb (the same pin ponyc uses for image retention), with account: ponylang, token: ${{ secrets.DELETE_PACKAGES_TOKEN }} (an org-level secret), tag-selection: tagged, cut-off: 90d, and image-names: "nightly/<tool>-*". It must list every platform publish job in needs:, or it could delete a tag before anyone's pulled it. That invariant lives in a comment on the prune job, where someone adding a platform would trip over it.

What the experiment settled, and the first nightly confirmed

Before touching the real pipeline I proved the whole thing out on a throwaway ponyup-experiment package; the first production nightly then corroborated every bit of it on the real packages, including the three paths I couldn't run by hand.

• Multi-segment names work. nightly/<tool>-<triple>, slash and all, is fine for both the OCI push and the retention-policy glob. No need to flatten to a single segment. The nightly's prune matched all six nightly/ponyup-* packages and correctly deleted nothing.
• Packages come out public on their own. This was the open question that ate the most time. There's no documented API to flip an org container package public, the undocumented PATCH just 404s, and I was bracing to flip every new package by hand forever. A controlled run put it to rest, and the nightly confirmed it at scale: all six real packages landed anonymously pullable with no visibility call anywhere. So the script carries no visibility code at all. The one thing I never pinned down is why it's public — org default versus inheriting from the public repo. So when corral and ponyc roll out, glance at whether the first package lands public. If one doesn't, that's the thread to pull.
• The artifacts are real. Pulled the Windows .zip package back anonymously, re-hashed the bytes, sha256 and sha512 both matched what we recorded, and it unzips to bin/ponyup.exe.
• The hand-untestable paths all came through. Windows .zip published via python (not python3), the native macOS jobs, and the arm64 bare-docker run token plumbing all published on the first try.

Check these per tool before cutting the PR

• python3 in the builder image. ponyup's was a given — the release path already runs Python there. ponyc has separate alpine and ubuntu builders; corral has its shared builder; changelog-tool its own. Probe each one rather than assuming.
• Python linting on the new script. ponyup lints every PR with ruff check .ci-scripts (.github/workflows/lint-python.yml, the pinned astral-sh/ruff image, ruff defaults, no config file), so the script is covered automatically and just has to be ruff-clean — I run that same image locally before pushing. Check whether the target repo already lints .ci-scripts; if it doesn't, copy ponyup's lint-python.yml over in the same PR so the script doesn't rot. Either way, make sure the script is ruff-clean before you push.
• The platform matrix differs. ponyc has the most jobs, and its triples carry a distro segment like x86-64-unknown-linux-alpine3.23, so its image names and prune glob follow that. ponyup's are plain <arch>-<vendor>-<os>.
• Windows calls python, not python3.
• The job names say "to Cloudsmith". Each tool's nightly jobs are named like Build and upload <triple>-nightly to Cloudsmith. Once they also push to GHCR that's stale, so fix it in the same PR — drop the destination entirely (just Build and upload <triple>-nightly) rather than relabel to "Cloudsmith and GHCR", which only rots again when the read side moves. ponyup did this in Drop the publish destination from the nightly job names #434.
• No release notes. This is publishing infrastructure. A tool user notices nothing until the read side moves to GHCR, so it isn't user-facing yet.

Rolling each one out

Prove it once in isolation if a tool's matrix makes you nervous, then integrate and let the next scheduled nightly exercise the platforms you can't run by hand. Because GHCR sits behind Cloudsmith, a per-platform stumble surfaces in the usual Zulip alert without costing anyone a nightly. Fix forward from there.

Validating after the first nightly

A green board is necessary but not sufficient — confirm the packages are actually there, public, and intact. This is the same pass I ran for ponyup; repeat it per tool, swapping in the tool name, a triple, and the date.

1. Every platform job and the prune job is green. A GHCR push failure fails its job loudly (it runs in the same step, right after Cloudsmith), so a fully green board means every platform published.
2. The prune job matched the right packages. Its log prints N/M package names matched the package-name filters — N should equal the platform count, and on a fresh run it should report selecting 0 versions for deletion (nothing is 90 days old yet). If N is short, the glob or a package name is off.
3. A package is anonymously pullable and its bytes are intact. This is the real proof: it confirms born-public and a well-formed artifact, with no credentials, exactly how a future read side will fetch it.

REPO=ponylang/nightly/<tool>-<triple>
TAG=<YYYYMMDD>

# Anonymous pull token — note there are NO credentials in this request.
TOKEN=$(curl -s "https://ghcr.io/token?service=ghcr.io&scope=repository:${REPO}:pull" \
  | python3 -c "import sys,json;print(json.load(sys.stdin)['token'])")

# Manifest: a 200 here with no creds means the package is PUBLIC. Check the
# artifactType and the layer mediaType match what the script wrote.
curl -s -H "Authorization: Bearer ${TOKEN}" \
  -H "Accept: application/vnd.oci.image.manifest.v1+json" \
  "https://ghcr.io/v2/${REPO}/manifests/${TAG}"

# Pull the layer blob by the digest from the manifest and re-hash it:
#   sha256 of the bytes must equal the manifest's layer digest
#   sha512 of the bytes must equal the sha512 field in the config blob
# (fetch the config blob the same way, by its digest, to read that field).

Reading the result:

• 200, no credentials → public and present. That's the born-public confirmation for this repo.
• 401 (or a denial) to an anonymous client → the package landed private. That's the signal the why-is-it-public question matters here, and the thread to pull for this tool.
• 404 MANIFEST_UNKNOWN → almost always the wrong tag, not private. Mind the date: a nightly that crosses 00:00 UTC mid-run tags with the later day, so check the actual tag the job logged before concluding anything.

The exact diff to copy is in #433.

[SeanTAllen]

corral and changelog-tool have been implemented.

[SeanTAllen]

ponyc has been implemented as well.