Skip to content

Harden OS/installer/CI: bootc observability, kernel-level taskd loopback, supply-chain pinning - #13

Merged
oratis merged 5 commits into
mainfrom
opt/os-ci-infra
Aug 1, 2026
Merged

Harden OS/installer/CI: bootc observability, kernel-level taskd loopback, supply-chain pinning#13
oratis merged 5 commits into
mainfrom
opt/os-ci-infra

Conversation

@oratis

@oratis oratis commented Aug 1, 2026

Copy link
Copy Markdown
Owner

PR3 of a 4-PR optimization pass. Scope is strictly os/** and .github/**; no Rust, schemas, or docs are touched (sibling PRs own those).

What this does

bootc install observability (OS review #1, #6 — highest priority)

  • Failure-path evidence (test-install.sh): the ESP/root diagnostics harvest now runs before the install-status / qemu-img / partition-probe exit checks (best-effort, EXIT trap unmounts on early exit). A failed install now uploads the same disk-side evidence as a successful one.
  • First-class bootc stderr (collect-anaconda-diagnostics.sh): captures the Anaconda 44 Payloads-module journal (journalctl -u 'org.fedoraproject.Anaconda.Modules.Payloads*') to its own ESP file, and emits program.log in full instead of tail -n 2000.
  • Interactive %onerror (interactive-defaults.ks): human-facing installs now leave diagnostics on the ESP too.

CI/harness robustness

Security hardening

  • Kernel-level loopback + sandbox (andromeda-taskd.service, security #2c): IPAddressAllow=localhost + IPAddressDeny=any plus SystemCallFilter=@system-service, LockPersonality, MemoryDenyWriteExecute, RestrictSUIDSGID, ProtectProc=invisible. Verified no conflict with the existing RestrictAddressFamilies.
  • Base-image digest pinning (os/Containerfile, security feat: add hardware preflight CI and project README #4): rust:1.85-bookworm and fedora-bootc:44 pinned by @sha256, plus a documented signing-policy follow-up next to --target-imgref in both kickstarts.

Validation

  • shellcheck 0.11.0 clean on all four touched scripts (exit 0).
  • bash -n clean on all four scripts.
  • test-installer-platform-guard.shcases=14 OK.
  • test-containerfile-layer-budget.sh → OK (payload_dnf_layers=23, size-check skip-with-log); size assertion also exercised against synthetic under/over-budget history JSON.
  • Both workflow YAMLs parse (yaml.safe_load).
  • QEMU/nested-KVM and ISO build were not runnable in this environment; changes are conservative and syntactically verified. os-e2e will exercise them on this PR.

🤖 Generated with Claude Code

oratis and others added 5 commits August 1, 2026 16:57
Three related fixes to the QEMU-based install/lifecycle harness:

- Failure-path observability (OS review #1a): move the ESP/root diagnostics
  harvest ahead of the install_status/qemu-img/partition-probe exit checks so
  a failed install uploads the same disk-side evidence (EFI/Andromeda/
  diagnostics, root var/log/anaconda) as a successful one. The harvest is now
  best-effort; the EXIT trap unmounts on early exit. This is what turns an
  opaque "bootc ... exited with status 1" into a root-causable file.

- Marker robustness (OS review #4): the boot-phase trigger greps now run on a
  NUL/CR/ANSI-stripped copy of the serial log, mirroring the Python sequence
  validator and test-gcp-nested.sh, so a marker split by serial cursor-control
  residue can no longer cause a false 2700s timeout.

- Fail-fast KVM gate (OS review #2): test-install.sh and test-hardware-matrix.sh
  now assert /dev/kvm is present (as test-gcp-nested.sh does) and exit with a
  clear message instead of silently falling back to TCG, whose timeout budget
  can exceed the os-e2e job timeout and surface as confusing timeouts. Gated
  behind ANDROMEDA_ALLOW_TCG=1 for local debugging.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- First-class bootc stderr capture (OS review #1b): Anaconda 44's native bootc
  payload runs under the Payloads DBus module, whose journal is where its
  stderr actually lands. collect-anaconda-diagnostics.sh now writes
  `journalctl -u 'org.fedoraproject.Anaconda.Modules.Payloads*'` to its own
  file on the ESP (anaconda-payloads-journal.log) and echoes it to serial, and
  emits program.log in full instead of tail -n 2000 so a late payload-phase
  failure is not truncated away.

- Interactive diagnostics (OS review #6): interactive-defaults.ks gains the same
  %onerror hook the CI kickstart already has, so a failed human-facing install
  also leaves diagnostics on the ESP for post-mortem.

Also documents (security review #4) that the mutable, unsigned --target-imgref
needs a bootc signing policy before remote upgrades can be trusted.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Supply-chain hardening (security review #4):

- Pin the build base images by @sha256 digest so a retagged or compromised
  upstream tag cannot silently change build inputs:
  * docker.io/library/rust:1.85-bookworm
      @sha256:e51d0265072d2d9d5d320f6a44dde6b9ef13653b035098febd68cce8fa7c0bc4
  * quay.io/fedora/fedora-bootc:44
      @sha256:69944835958dda565f8d613587535a3d90e3256cf0a0f0e29bca5ca3d2360d06
  The human-readable tag is kept for context; the digest is authoritative.

- Document near the CI kickstart's --target-imgref that a bootc signing policy
  (sigstore/containers-policy.json) is a required follow-up before trusting
  remote bootc switch/upgrade from the mutable, unsigned edge tag.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Security review #2c / defence-in-depth for the unauthenticated task control
plane:

- IPAddressAllow=localhost + IPAddressDeny=any give kernel-level loopback
  enforcement, so taskd can never serve non-loopback regardless of
  ANDROMEDA_LISTEN. This is orthogonal to (and does not conflict with) the
  existing RestrictAddressFamilies, which filters socket() families rather than
  routes.

- Additional systemd sandboxing: SystemCallFilter=@System-service,
  LockPersonality, MemoryDenyWriteExecute, RestrictSUIDSGID, and
  ProtectProc=invisible. (SystemCallArchitectures=native, RestrictRealtime, and
  ProtectHostname were already present.)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Action pinning (OS review #3): ci.yml pinned dtolnay/rust-toolchain and
  Swatinem/rust-cache to full commit SHAs with trailing version comments,
  matching os-e2e.yml's stated policy (third-party actions pinned to a commit
  SHA). The rust-toolchain commit hardcodes toolchain 1.85.0 in its action.yml,
  so no explicit `with: toolchain` is needed.
  * dtolnay/rust-toolchain @8641a17e25bf5b40c118d48fe0f81e8655731839 # 1.85.0
  * Swatinem/rust-cache    @c19371144df3bb44fab255c43d04cbc2ab54d1c4 # v2.9.1

- Layer-size guard (OS review #5): test-containerfile-layer-budget.sh now, in
  addition to the min-DNF-layer-count proxy, asserts the largest single OCI
  layer stays under budget using andromeda-v1-history.json when build-iso.sh
  has produced it. This directly guards the real risk (a single oversized blob
  overflowing the installer /var/tmp overlay, the cause of the recent bootc
  failure). In the unit-test CI context no image is built, so the check
  skips-with-log rather than failing.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@oratis
oratis merged commit ded31f1 into main Aug 1, 2026
5 checks passed
@oratis
oratis deleted the opt/os-ci-infra branch August 1, 2026 10:52
oratis added a commit that referenced this pull request Aug 1, 2026
PR #13 pinned quay.io/fedora/fedora-bootc:44 by @sha256 digest for supply-chain
integrity. fedora-bootc:44 is a frequently-rebuilt rolling tag whose old digests
Fedora garbage-collects from quay.io within days, so the pinned digest became
"manifest unknown" and every subsequent ISO build failed at the payload FROM —
a latent regression that surfaced on the next os-e2e run.

Track the rolling :44 tag for the payload base until digest-refresh automation
(Renovate/Dependabot) is in place; keep the rust builder pinned by digest (Docker
Hub retains historical digests). Re-pinning fedora-bootc by digest + refresh
automation is the tracked follow-up. See docs/reviews/os-infrastructure-review.md
and security-review.md finding #4.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant