Skip to content

Add supply-chain tiering: Renovate config, pin-freshness check, re-pin runbook - #29

Merged
oratis merged 4 commits into
mainfrom
opt/supply-chain-tiering
Aug 2, 2026
Merged

Add supply-chain tiering: Renovate config, pin-freshness check, re-pin runbook#29
oratis merged 4 commits into
mainfrom
opt/supply-chain-tiering

Conversation

@oratis

@oratis oratis commented Aug 2, 2026

Copy link
Copy Markdown
Owner

Implements P1 #8 (supply-chain tiering) from docs/reviews/e2e-pipeline-review.md.
Stacked on opt/e2e-p0-hardening (#22) because that PR already touches .github/workflows/.

The lesson this PR encodes

quay.io/fedora/fedora-bootc:44 was pinned by @sha256. Fedora prunes superseded
fedora-bootc digests from quay.io within days, the pin became manifest unknown, and
every build broke at once (run 30702410393). d27cefa un-pinned it — correct first aid,
wrong place to stop.

The lesson is not "never pin". It is land the refresh automation first, then re-pin.
So this PR lands the automation and the safety net, and deliberately does not re-pin.

What's here

.github/renovate.json Digest pinning + refresh for Containerfile base images and Actions
os/scripts/check-pin-freshness.sh Early warning before a pin goes unresolvable
.github/workflows/ci.yml Runs the check, non-blocking
docs/development/supply-chain.md Tiering, inventory, re-pin runbook, both investigations
os/Containerfile Comment-only: marks the reference as re-pin-pending, points at the runbook

⚠️ Renovate is a GitHub App — a human must install it

.github/renovate.json does nothing on its own. No PRs, no dashboard, no digest refresh
until someone with admin rights installs the Renovate App on oratis/Andromeda
(https://github.com/apps/renovate). I cannot do this.

Until it is installed, do not re-pin fedora-bootc — the pin would rot within days and
re-break every build. The runbook in the doc gates on three verifiable signals before re-pinning.

Nice property: pinDigests: true means Renovate itself opens the re-pin PR on first run.
That PR existing is the proof that refresh automation works — a proof hand-pinning can't give.

Digest refreshes deliberately skip the weekly window (schedule: ["at any time"], prPriority: 10):
quay prunes by the day, so queueing a refresh until Monday is itself the failure mode.

Why the CI step is continue-on-error: true

It reports on the world outside this repo — whether an upstream registry still serves a digest.
That answer changes without anyone touching the repository. Gating merges on it would turn an
unrelated PR red for a problem its author neither caused nor can fix there — which is exactly how
run 30702410393 was experienced. The signal belongs on a dashboard, not in the merge gate.
--strict gives gate semantics to callers that want it.

Finding: ghcr.io/osbuild/image-builder-cli is NOT the next fedora-bootc

The review flagged it as the same risk class. Investigated — it is not. Keep the pin as-is.

  1. GitHub Packages documents deletion as manual only, with a 30-day restore window and no
    automatic expiry or GC (docs).
    The ecosystem of third-party ghcr-cleanup-action tools exists because GitHub doesn't prune.
  2. Probed the registry: the pinned sha256:67f1c248… returns HTTP 200 while :latest has
    already moved to sha256:469a4269… — the superseded digest is still served.
  3. osbuild tags every build: 334 tags, all sha-<commit> besides latest.

The difference from quay is policy, not technology — Fedora prunes to control storage, GitHub
doesn't. Residual risk is low but real (upstream could enable a cleanup action); the freshness
check already covers build-iso.sh read-only and would warn first.

os/scripts/build-iso.sh is owned by opt/e2e-p1-cache-split and is not touched here
and doesn't need to be, since the verdict is "keep as-is".

Finding: no usable Fedora dated snapshot — Tier B documented, not implemented

Probed four candidate endpoints:

Candidate Result
kojipkgs…/compose/updates/Fedora-44-updates-<date>.N/ repodata 200, but index holds only ~10 days (F44: 2026072420260802, measured 2026-08-02)
dl.fedoraproject.org/…/releases/44/… (GA) 200, durable, but frozen at GA — zero security updates
dl.fedoraproject.org/…/updates/44/… 200, but rolling — no date dimension (this is the current drift source)
dl.fedoraproject.org/pub/archive/… 200, but EOL releases only

Decisive: a kojipkgs dated compose 404s in ~10 days — it would rot faster than the quay digest
that already broke the build, trading a reproducibility problem for an availability one. (The ~10-day
window is measured, not a published policy — itself a reason not to depend on it.)

So: not implemented. The doc records what the Containerfile change would look like, its costs,
and recommends the content-addressed payload cache from P1 #6 instead — same reproducibility, no
external endpoint to rot, and it removes 11.1 min of dnf time.

Validation

  • shellcheck os/scripts/check-pin-freshness.sh — clean
  • renovate-config-validatorConfig validated successfully against 1 file(s)
  • yaml.safe_load(ci.yml) — parses
  • Script run locally (see below); --strict exits 1, --max-age-days 600 exits 0
docker.io/library/rust@sha256:e51d0265...
  resolvable: yes
  !! STALE: built 2025-03-18T20:40:17Z (501 days ago, budget 90).

quay.io/fedora/fedora-bootc:44
  TAG-TRACKED (no digest pin)
  current digest: sha256:413aa29c...
  tag content built 2026-08-01T11:06:26Z (0 days ago)
  to re-pin, the reference becomes:
    quay.io/fedora/fedora-bootc:44@sha256:413aa29c...

ghcr.io/osbuild/image-builder-cli@sha256:67f1c248...
  resolvable: yes
  age: 7 days, within the 90 day budget

The rust finding is a true positive: the pin can't rot (Docker Hub retains), but it's 17 months
of unmerged bookworm security updates. Renovate would file that PR automatically.

🤖 Generated with Claude Code


Verified on a real runner (and it caught a bug)

The first CI run reported age: unparseable timestamp and 0 findings, while the same script
locally reported the rust pin as 501 days stale. Two mistakes only visible on a real runner:

  • ubuntu-latest does ship skopeo, so CI took the skopeo backend — not the curl fallback my
    comment claimed.
  • skopeo's Go template renders time.Time as 2025-03-18 20:40:17 +0000 UTC, not the RFC 3339
    the registry API returns. No date(1) invocation could parse it.

Net effect: the age half of the check was dead in the only place it runs automatically. Fixed by
normalising both layouts (verified against all four shapes, garbage still rejected).

Confirmed on run 30735321078:

  backend:      skopeo
  !! STALE: built 2025-03-18 20:40:17 +0000 UTC (501 days ago, budget 90).
  ----
  3 reference(s) checked, 1 finding(s)
  ANDROMEDA_PIN_FRESHNESS_WARN (advisory mode, not failing)

Job conclusion: success — the finding is reported without failing the run, which is the
non-blocking behaviour this step is supposed to have.

oratis and others added 3 commits August 2, 2026 13:59
Tier A of the supply-chain strategy needs refresh automation to exist
BEFORE anything is pinned by digest. quay.io/fedora/fedora-bootc:44 was
pinned without it, Fedora pruned the superseded digest within days, and
every build broke at once (Actions run 30702410393) until d27cefa
un-pinned it. The lesson is not "never pin" but "land the automation
first".

Keeps Containerfile base images pinned by digest and GitHub Actions
pinned to full SHAs with their version comment rewritten in place.
Digest refreshes deliberately skip the weekly window: quay prunes by
the day, so queueing a refresh until Monday is itself the failure mode.

Renovate is a GitHub App, not a workflow. This file does nothing at all
until someone with admin rights installs the App on the repository, so
the fedora-bootc re-pin stays blocked until that happens.

Validated with renovate-config-validator.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Reports, for every image reference in os/Containerfile (and read-only,
os/scripts/build-iso.sh), whether the pinned digest is still served by
the registry and how old that content is. An unresolvable pin is exactly
the fedora-bootc breakage, caught before the next build rather than
during it; unpinned references are reported too, along with the digest
they resolve to right now, so the re-pin runbook is copy-paste.

Prefers skopeo and falls back to the Docker Registry v2 HTTP API over
curl + jq, which ubuntu-latest preinstalls -- so the check runs in the
one-minute CI job instead of needing the 48-minute one.

Wired into ci.yml as continue-on-error on purpose. The step reports on
upstream registries, whose answers change without anyone touching this
repository; gating merges on it would turn an unrelated PR red for a
problem its author neither caused nor can fix there. Callers that want
a gate pass --strict.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Ties the pieces together: the A/B/C/D tiering, the full input inventory
with per-input status, the re-pin runbook, and how to read the freshness
check. Records two investigations the review asked for.

ghcr.io/osbuild/image-builder-cli is NOT the next fedora-bootc. GitHub
Packages documents deletion as manual only (30-day restore window) with
no automatic expiry, the currently pinned digest still resolves while
:latest has moved on, and osbuild tags every build sha-<commit> so
historical versions never become untagged. Keeping that pin is correct;
build-iso.sh is owned by opt/e2e-p1-cache-split and stays untouched.

Fedora publishes no durable dated snapshot for a supported release, so
Tier B is documented rather than implemented. kojipkgs dated composes
exist and serve valid repodata, but the index holds only ~10 days
(F44: 20260724-20260802, measured) -- pointing the Containerfile at one
would rot faster than the quay digest that already broke the build,
trading a reproducibility problem for an availability one. The frozen GA
compose is durable but ships zero security updates. The content-
addressed payload cache from P1 #6 gets the same reproducibility with no
external endpoint to rot, and is the recommended path instead.

fedora-bootc stays tag-tracked. Re-pinning before Renovate is installed
would just re-break the build within days.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@oratis
oratis force-pushed the opt/supply-chain-tiering branch from 1f6beaf to 551916c Compare August 2, 2026 06:02
The first CI run of this check reported "age: unparseable timestamp" and
0 findings, while the same script locally reported the rust pin as 501
days stale. Two mistakes, both only visible on a real runner:

ubuntu-latest DOES ship skopeo, so CI took the skopeo backend, not the
curl fallback the ci.yml comment claimed. And skopeo's Go template
renders time.Time in Go's default layout (2025-03-18 20:40:17 +0000 UTC)
rather than the RFC 3339 the registry API returns, which no date(1)
invocation here could parse. Net effect: the age half of the check was
dead in the only place it runs automatically.

Normalise both forms to YYYY-MM-DDTHH:MM:SSZ before parsing; sub-second
precision is irrelevant to a budget measured in days. Verified against
all four shapes (both layouts, with and without fractional seconds) and
that garbage is still rejected. ci.yml comment corrected to match
reality.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@oratis
oratis changed the base branch from opt/e2e-p0-hardening to main August 2, 2026 06:09
@oratis

oratis commented Aug 2, 2026

Copy link
Copy Markdown
Owner Author

Review(深度审查,对 opt/e2e-p0-hardening...opt/supply-chain-tiering 真实增量:4 commits,5 文件,+919/−6)

结论:自动化设计论证充分、脚本是真材实料的 shell(heredoc 保计数器、锚定 HOST_RE、端口/tag 消歧、三级 date 回退在本机逐一验证通过;对真实 Containerfile 复跑输出与 PR 正文完全一致);标题里的 "re-pin" 实际零重钉——"先自动化后钉版"的顺序论证是本 PR 最好的部分。问题集中在接缝处:Renovate 规则与既定策略打架、所有退化路径静默、runbook 依赖的门今天无法通过。修复后合并。

Blockers

  1. Renovate 会对第一方 action 提 SHA 钉版 PR,违反仓库明文策略renovate.json:15,60,71-72):helpers:pinGitHubActionDigests + pinDigests: true 对全部 action 生效;第四条 actions/** 规则只设了 groupName,按序合并后继承 pinDigests: true → Renovate 会把 actions/checkout@v4 改写成 SHA。而 ci.yml:24-26 白纸黑字:"first-party actions/* are pinned to major tags"。规则自己的 description 都承认这一点却没编码它。需 "pinDigests": false 或同 PR 内显式改策略。
  2. 扫描零引用报成功,--strict 下也如此check-pin-freshness.sh:483-492,505-507,已实测复现):文件改名 / SCAN_FILES 笔误 / Containerfile 重构都会把整张安全网降级为报 OK 的 no-op,叠加 CI 的 continue-on-error: true 永远无人发现。checked==0 必须是 finding 或 exit 2。
  3. runbook 第 4 步的 --strict 门今天必失败(实测 exit 1):rust pin 已 501 天——PR 自己都声明它安全(Docker Hub 不清理)。--strict 把 UNRESOLVABLE(真断裂)和 STALE(年龄劝告)当同一种失败,文档化的门在无关的 rust bump 落地前不可执行。前置清单要求"无 UNRESOLVABLE"却没有任何 exit code 能表达它。需拆分退出码或加 --fail-on

主要次要项

  • continue-on-error: true 对其声称的目的冗余(劝告模式本来就 exit 0),真正吞掉的是 exit 2 和 set -e 中止——本 PR 自己的历史就是证据:1921496 修的"unparseable timestamp + 0 findings"半死状态当时 CI 是绿的。结构性根因(pin_created 一切失败路径 || true; return 0、零 finding)未变。
  • 无定时执行:检查只在有人开 PR 时跑;它独有的"腐烂前"年龄预警在安静的一周永远不响。文档自己的"横切:定时探测"节点名了这个失败模式然后搁置了它。~20 秒纯网络检查是 nightly 最便宜的追加。
  • digest 刷新与版本 bump 共用 groupName,"跳过周窗口"的核心紧急性主张取决于分支级 schedule 合并时谁赢——给 rule 2 单独 groupName。
  • build-iso.sh:7 的 ghcr pin 无任何刷新路径(shell 赋值,enabledManagers 都不解析)——文档表格说"Renovate 接管刷新"是不成立的;day 91 起永久 STALE。需 ~8 行 customManagers regex。
  • pin_resolvable 无法区分 404 与 401/429/5xx/DNS(匿名 Docker Hub 在共享 Actions IP 上 429 是常态),无重试——strict 下是假阴性发布阻塞。
  • skopeo 路径客户端算 digest($(...) 吞尾换行)且 CI 恰好走这条未验证路径,本地/PR 正文引用的输出全来自 curl 回退;算错的值会被 runbook 教人贴进 Containerfile → 秒变 manifest unknown
  • 12 个函数零 local、mktemp 无 trap、新 job 无 timeout-minutesosvVulnerabilityAlerts 在 dockerfile+actions 管理器下是 no-op(真正能用漏洞告警的 cargo 被排除,声明夸大)、supply-chain.md:40 引用行号被本 PR 自己加的注释块挤偏、Containerfile 的"故意不钉"注释块在 Renovate 首选路径下会变成谎言(runbook 只在手动回退路径删它)。

已验证无误

JSON 合法且各键正确;引用的文档/提交全部存在且实现了 e2e-pipeline-review P1 #8 声明的顺序;layer-budget 脚本数 RUN 层不数行,注释块无害;新脚本被 shellcheck glob 与 paths 过滤覆盖;iso_to_epoch 修复后对两种时间格式均正确。

处理方式

轮到本 PR 时(#30 之后)将应用全部三个 blocker 修复 + --fail-on 拆分 + nightly 接入 + customManagers + HTTP 状态区分 + local/trap/timeout 清理 + 文档行号与标题措辞修正。

@oratis
oratis merged commit dc22a7b into main Aug 2, 2026
6 checks passed
@oratis
oratis deleted the opt/supply-chain-tiering branch August 2, 2026 08:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant