Skip to content

ci: stop pushing a nightly image on every commit to main - #8

Merged
TheMeinerLP merged 1 commit into
mainfrom
ci/nightly-registry-pressure
Aug 8, 2026
Merged

ci: stop pushing a nightly image on every commit to main#8
TheMeinerLP merged 1 commit into
mainfrom
ci/nightly-registry-pressure

Conversation

@TheMeinerLP

Copy link
Copy Markdown

Why

On 2026-08-07 the Container image job failed like this:

ERROR: failed to push ghcr.io/onelitefeathernet/ingot:nightly:
denied: permission_denied: Error from intermediary with HTTP status code 403 "Forbidden"
{ "message": "You have exceeded a secondary rate limit. ..." }

Worth knowing before anyone debugs this again: GHCR reports a throttled manifest
PUT as 403 permission_denied. It looks like packages: write is missing. It is
not.

What ran into the limit was volume. Every commit on main pushed both images for
two architectures, with an SBOM and provenance: mode=max on each. Four of those
ran between 12:04 and 12:23 that day, every one of them triggered by a
documentation commit, every one overwriting the same nightly tag.

What changed

A nightly is only pushed when the push changed something the image contains.
The new changes job diffs the pushed range against a deny list. Deny rather
than allow, because Dockerfile:24 copies the whole repository into the build
stage: anything the list has not heard of counts as relevant. A redundant nightly
costs minutes, a missing one costs somebody an afternoon. Releases are never
filtered, and a force push or rewritten history falls through to a build.

Checked against real history on this repository:

Commit Result
779e9ebe docs: write CLAUDE.md in English skip
6b51a2f9 docs: write down the drop-in compatibility rule skip
05bfbb5d refactor(ci): give the container checks a shared library skip
1fe847d0 fix(ci): read file ownership through a container build
47bd5a0b build(docker): build both architectures without emulating build

All four pushes from the 12:0x burst would have been skipped.

The workflow level concurrency group is gone. It had cancel-in-progress: false, which serialised every run of the workflow. That sounds harmless and is
not: a queued nightly still runs its full push once the run ahead of it finishes,
so a burst of commits becomes a burst of registry writes spread over an hour.
Because runs never overlapped, a job level cancel could never have fired either.
Release Please now carries that serialised group itself, where it belongs, and
the container job has its own: nightlies share container-nightly-<ref> and
cancel each other, releases are keyed by version and cancel nothing.

SBOM and provenance are release only. Each attestation is a further manifest
pushed per platform, and on a tag overwritten several times a day it is replaced
before anyone reads it. Releases keep the full set.

Both pushes retry once, after waiting 180 seconds. Buildx has no backoff, so
one refused manifest PUT used to lose an entire two platform build. The retry is
cheap because every layer is still in the cache and only the push is replayed.

CLAUDE.md is updated to match: it previously stated that every push carries an
SBOM and provenance.

Reviewing this

The two with: blocks per image are duplicated between the attempt and the
retry, and there is a comment saying to keep them in sync. A composite action
would have removed the duplication, but whether steps.<id>.outcome is readable
inside one is undocumented, and a retry that silently never fires is worse than
the redundancy.

actionlint including its shellcheck pass is clean on both workflow files.

Not in this PR

Three further findings from the same investigation, none of which have caused a
failure yet:

  • The Actions cache sits at 10.95 GB across 203 entries, above the 10 GB limit,
    so it evicts constantly and cache-from: type=gha misses more often than not.
  • Docker Hub is pulled anonymously everywhere. Roughly 15 to 20 pulls per CI run
    across three jobs, against a limit of 100 per 6 hours per shared runner IP.
  • Trivy fetches its database (about 1 GB) from GHCR anonymously, five times per
    CI run. TRIVY_USERNAME and TRIVY_PASSWORD are unset.

Unrelated, found on the way: the Codecov upload fails with
{"message":"Repository not found"} and is hidden by fail_ci_if_error: false.
Coverage reporting has not worked at all.

GHCR refused a nightly push on 2026-08-07 with `403 permission_denied`
and a secondary rate limit body, which reads like a permissions problem
and is not one. Four full nightly pushes had run between 12:04 and
12:23, all triggered by documentation commits and all overwriting the
same tag.

A nightly is now pushed only when the push changed something the image
contains. The new `changes` job decides that with a deny list rather
than an allow list, because the Dockerfile copies the whole repository
into its build stage: anything the list has not heard of counts as
relevant, and a redundant nightly is cheaper than a stale one. Releases
are never filtered.

Three things reduce the pressure further:

The workflow level concurrency group is gone. It serialised every run,
so a queued nightly still ran its full push once the run ahead of it
finished, turning a burst of commits into a burst of registry writes
spread over an hour. Release Please keeps its own serialised group, and
the container job cancels superseded nightlies while leaving releases
alone.

SBOM and provenance are release only. Each attestation is a further
manifest pushed per platform, and on a tag overwritten several times a
day the evidence is replaced before anyone reads it.

Both pushes retry once after waiting out the registry. Buildx has no
backoff of its own, so a single throttled manifest PUT used to lose an
entire multi platform build; the retry finds every layer in the cache.
@github-actions

github-actions Bot commented Aug 8, 2026

Copy link
Copy Markdown

Test results

 68 files   68 suites   9m 31s ⏱️
325 tests 324 ✅ 1 💤 0 ❌
327 runs  326 ✅ 1 💤 0 ❌

Results for commit f4b5d06.

@TheMeinerLP
TheMeinerLP merged commit 7fc6ea7 into main Aug 8, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant