Add supply-chain tiering: Renovate config, pin-freshness check, re-pin runbook - #29
Merged
Conversation
Tier A of the supply-chain strategy needs refresh automation to exist BEFORE anything is pinned by digest. quay.io/fedora/fedora-bootc:44 was pinned without it, Fedora pruned the superseded digest within days, and every build broke at once (Actions run 30702410393) until d27cefa un-pinned it. The lesson is not "never pin" but "land the automation first". Keeps Containerfile base images pinned by digest and GitHub Actions pinned to full SHAs with their version comment rewritten in place. Digest refreshes deliberately skip the weekly window: quay prunes by the day, so queueing a refresh until Monday is itself the failure mode. Renovate is a GitHub App, not a workflow. This file does nothing at all until someone with admin rights installs the App on the repository, so the fedora-bootc re-pin stays blocked until that happens. Validated with renovate-config-validator. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Reports, for every image reference in os/Containerfile (and read-only, os/scripts/build-iso.sh), whether the pinned digest is still served by the registry and how old that content is. An unresolvable pin is exactly the fedora-bootc breakage, caught before the next build rather than during it; unpinned references are reported too, along with the digest they resolve to right now, so the re-pin runbook is copy-paste. Prefers skopeo and falls back to the Docker Registry v2 HTTP API over curl + jq, which ubuntu-latest preinstalls -- so the check runs in the one-minute CI job instead of needing the 48-minute one. Wired into ci.yml as continue-on-error on purpose. The step reports on upstream registries, whose answers change without anyone touching this repository; gating merges on it would turn an unrelated PR red for a problem its author neither caused nor can fix there. Callers that want a gate pass --strict. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Ties the pieces together: the A/B/C/D tiering, the full input inventory with per-input status, the re-pin runbook, and how to read the freshness check. Records two investigations the review asked for. ghcr.io/osbuild/image-builder-cli is NOT the next fedora-bootc. GitHub Packages documents deletion as manual only (30-day restore window) with no automatic expiry, the currently pinned digest still resolves while :latest has moved on, and osbuild tags every build sha-<commit> so historical versions never become untagged. Keeping that pin is correct; build-iso.sh is owned by opt/e2e-p1-cache-split and stays untouched. Fedora publishes no durable dated snapshot for a supported release, so Tier B is documented rather than implemented. kojipkgs dated composes exist and serve valid repodata, but the index holds only ~10 days (F44: 20260724-20260802, measured) -- pointing the Containerfile at one would rot faster than the quay digest that already broke the build, trading a reproducibility problem for an availability one. The frozen GA compose is durable but ships zero security updates. The content- addressed payload cache from P1 #6 gets the same reproducibility with no external endpoint to rot, and is the recommended path instead. fedora-bootc stays tag-tracked. Re-pinning before Renovate is installed would just re-break the build within days. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
oratis
force-pushed
the
opt/supply-chain-tiering
branch
from
August 2, 2026 06:02
1f6beaf to
551916c
Compare
The first CI run of this check reported "age: unparseable timestamp" and 0 findings, while the same script locally reported the rust pin as 501 days stale. Two mistakes, both only visible on a real runner: ubuntu-latest DOES ship skopeo, so CI took the skopeo backend, not the curl fallback the ci.yml comment claimed. And skopeo's Go template renders time.Time in Go's default layout (2025-03-18 20:40:17 +0000 UTC) rather than the RFC 3339 the registry API returns, which no date(1) invocation here could parse. Net effect: the age half of the check was dead in the only place it runs automatically. Normalise both forms to YYYY-MM-DDTHH:MM:SSZ before parsing; sub-second precision is irrelevant to a budget measured in days. Verified against all four shapes (both layouts, with and without fractional seconds) and that garbage is still rejected. ci.yml comment corrected to match reality. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Owner
Author
Review(深度审查,对
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements P1 #8 (supply-chain tiering) from
docs/reviews/e2e-pipeline-review.md.Stacked on
opt/e2e-p0-hardening(#22) because that PR already touches.github/workflows/.The lesson this PR encodes
quay.io/fedora/fedora-bootc:44was pinned by@sha256. Fedora prunes supersededfedora-bootc digests from quay.io within days, the pin became
manifest unknown, andevery build broke at once (run 30702410393).
d27cefaun-pinned it — correct first aid,wrong place to stop.
The lesson is not "never pin". It is land the refresh automation first, then re-pin.
So this PR lands the automation and the safety net, and deliberately does not re-pin.
What's here
.github/renovate.jsonos/scripts/check-pin-freshness.sh.github/workflows/ci.ymldocs/development/supply-chain.mdos/Containerfile.github/renovate.jsondoes nothing on its own. No PRs, no dashboard, no digest refreshuntil someone with admin rights installs the Renovate App on
oratis/Andromeda(https://github.com/apps/renovate). I cannot do this.
Until it is installed, do not re-pin
fedora-bootc— the pin would rot within days andre-break every build. The runbook in the doc gates on three verifiable signals before re-pinning.
Nice property:
pinDigests: truemeans Renovate itself opens the re-pin PR on first run.That PR existing is the proof that refresh automation works — a proof hand-pinning can't give.
Digest refreshes deliberately skip the weekly window (
schedule: ["at any time"],prPriority: 10):quay prunes by the day, so queueing a refresh until Monday is itself the failure mode.
Why the CI step is
continue-on-error: trueIt reports on the world outside this repo — whether an upstream registry still serves a digest.
That answer changes without anyone touching the repository. Gating merges on it would turn an
unrelated PR red for a problem its author neither caused nor can fix there — which is exactly how
run 30702410393 was experienced. The signal belongs on a dashboard, not in the merge gate.
--strictgives gate semantics to callers that want it.Finding:
ghcr.io/osbuild/image-builder-cliis NOT the next fedora-bootcThe review flagged it as the same risk class. Investigated — it is not. Keep the pin as-is.
automatic expiry or GC (docs).
The ecosystem of third-party
ghcr-cleanup-actiontools exists because GitHub doesn't prune.sha256:67f1c248…returns HTTP 200 while:latesthasalready moved to
sha256:469a4269…— the superseded digest is still served.sha-<commit>besideslatest.The difference from quay is policy, not technology — Fedora prunes to control storage, GitHub
doesn't. Residual risk is low but real (upstream could enable a cleanup action); the freshness
check already covers
build-iso.shread-only and would warn first.os/scripts/build-iso.shis owned byopt/e2e-p1-cache-splitand is not touched here —and doesn't need to be, since the verdict is "keep as-is".
Finding: no usable Fedora dated snapshot — Tier B documented, not implemented
Probed four candidate endpoints:
kojipkgs…/compose/updates/Fedora-44-updates-<date>.N/20260724–20260802, measured 2026-08-02)dl.fedoraproject.org/…/releases/44/…(GA)dl.fedoraproject.org/…/updates/44/…dl.fedoraproject.org/pub/archive/…Decisive: a kojipkgs dated compose 404s in ~10 days — it would rot faster than the quay digest
that already broke the build, trading a reproducibility problem for an availability one. (The ~10-day
window is measured, not a published policy — itself a reason not to depend on it.)
So: not implemented. The doc records what the Containerfile change would look like, its costs,
and recommends the content-addressed payload cache from P1 #6 instead — same reproducibility, no
external endpoint to rot, and it removes 11.1 min of dnf time.
Validation
shellcheck os/scripts/check-pin-freshness.sh— cleanrenovate-config-validator—Config validated successfully against 1 file(s)yaml.safe_load(ci.yml)— parses--strictexits 1,--max-age-days 600exits 0The rust finding is a true positive: the pin can't rot (Docker Hub retains), but it's 17 months
of unmerged bookworm security updates. Renovate would file that PR automatically.
🤖 Generated with Claude Code
Verified on a real runner (and it caught a bug)
The first CI run reported
age: unparseable timestampand 0 findings, while the same scriptlocally reported the rust pin as 501 days stale. Two mistakes only visible on a real runner:
ubuntu-latestdoes ship skopeo, so CI took the skopeo backend — not the curl fallback mycomment claimed.
time.Timeas2025-03-18 20:40:17 +0000 UTC, not the RFC 3339the registry API returns. No
date(1)invocation could parse it.Net effect: the age half of the check was dead in the only place it runs automatically. Fixed by
normalising both layouts (verified against all four shapes, garbage still rejected).
Confirmed on run 30735321078:
Job conclusion: success — the finding is reported without failing the run, which is the
non-blocking behaviour this step is supposed to have.