fix: Phase 2 Linux repo prep — BE pkg naming, RPM signing, empty-pool guard#2759
Conversation
… guard End-to-end smoke test of the apt + yum bucket-repo templates (with a throwaway GPG key + Docker Ubuntu producer / Ubuntu+Fedora consumer containers) against real v4.0.0 artifacts surfaced five bugs that would block Phase 2 from working in production. Fixes them together because they're all in service of issue #2605 standup. 1. release.yml: derive CHANNEL from version, not hardcoded `stable`. Every snapshot build was producing `wheels_<v>_amd64.deb` regardless of branch. The Phase 2 bleeding-edge receiver fetches `wheels-be_<v>_amd64.deb` from wheels-snapshots → 404 every time. Now: `contains(env.WHEELS_VERSION, '-snapshot')` selects `bleeding-edge` (which routes build-linux-packages.sh to nfpm-wheels-be.yaml). 2. release.yml: upload globs `wheels_*_amd64.deb` (underscore) didn't match `wheels-be_*_amd64.deb` (dash). Loosened to `wheels*_amd64.deb` + `wheels*.x86_64.rpm` so both channels' artifacts upload. 3. apt-repo/scripts/regenerate-apt-metadata.sh: first publish to a brand-new channel crashed — apt-ftparchive aborts on a missing pool scan path. Yum side already guarded; apt didn't. Now mkdir -p pool/<dist> at the top of the dist loop so empty-channel first-runs produce empty-but-signed metadata. 4. yum-repo/scripts/regenerate-yum-metadata.sh: nfpm produces unsigned RPMs but wheels.repo sets gpgcheck=1, so dnf rejects with "Package is not signed: GPG check FAILED". Now signs each .rpm via `rpm --addsign` before createrepo_c walks the dir. Also pins %__gpg to $(command -v gpg) — Ubuntu runners ship `gpg`, not the `gpg2` rpm's macro defaults to. 5. Both receiver workflows: `gpg --import-ownertrust` was being fed the 16-char key_id from `sec:` line field 5, but the format wants the 40-char fingerprint from `fpr:` line field 10. Emitted a "gpg: error in '[stdin]': invalid fingerprint" warning every run. Non-fatal but noisy; fixed by extracting the right field. Plus: README + placeholder files updated to point at `op://Wheels/wheels-linux-repo-signing/` (Wheels project vault on my.1password.com) rather than `op://Infrastructure/` (PAI work tenant). Verified end-to-end inside Docker against real v4.0.0 artifacts: - Ubuntu 24.04 receiver produces apt + yum metadata, signs Release/ InRelease/repomd.xml + each RPM - Fresh Ubuntu 24.04 client: apt-get download wheels → success (MD5 matches Packages.gz) - Fresh Fedora 40 client: dnf install --downloadonly wheels → "Complete!" with all signatures verifying Once merged, the next develop snapshot will ship a properly named wheels-be_<v>_amd64.deb / wheels-be-<v>.x86_64.rpm pair, and the Phase 2 bucket repos (when created) will be able to consume them without code changes. See #2605 for the operational standup checklist. Signed-off-by: Peter Amiri <peter@alurium.com>
There was a problem hiding this comment.
Wheels Bot — Reviewer A
TL;DR: Five real bugs, well-diagnosed and well-explained. The CHANNEL derivation, fingerprint fix, empty-pool guard, and vault-path updates are all clean. One correctness issue blocks merge: rpm --addsign is called before the gpg-agent has the passphrase cached, so any passphrase-protected production key will cause silent or visible signing failure in CI. Two minor nits below that level.
Correctness
rpm --addsign receives no passphrase — will fail in production CI
tools/distribution-drafts/yum-repo/scripts/regenerate-yum-metadata.sh, lines 62–66:
for rpm_file in "${PKG_DIR}"/*.rpm; do
[ -f "$rpm_file" ] || continue
rpm --addsign "$rpm_file" >/dev/null
echo " ✓ signed $(basename "$rpm_file")"
doneThe comment at line 39 says "Passphrase via env (rpm reads $GNUPGHOME/gpg.conf which sets pinentry-mode loopback)." That is incorrect: pinentry-mode loopback redirects the passphrase prompt from a TTY to the calling application, it does not source the passphrase from an env var. The agent only has a passphrase cached if a prior signing operation primed it.
The ordering in the script is: sign RPMs → createrepo_c → sign repomd.xml (line 91–97). The explicit --passphrase "${GPG_PASSPHRASE:-}" call that would prime the agent does not happen until after the RPM signing loop. On a fresh CI runner with a freshly imported key the agent cache is empty at RPM-sign time. Result: rpm --addsign either hangs waiting for a TTY prompt (no TTY → non-zero exit, caught by set -euo pipefail) or, on some rpm versions, exits 0 with a warning and produces unsigned RPMs.
The local Docker test likely used a key without a passphrase, which explains the green test checklist.
Recommended fix — add a single dummy sign before the channel loop to seed the agent:
# Seed the passphrase into the agent once so rpm --addsign can use it
# without a TTY. Must run before the per-channel RPM signing loop.
echo "probe" | gpg --batch --yes \
--pinentry-mode loopback \
--passphrase "${GPG_PASSPHRASE:-}" \
-u "$GPG_KEY_ID" \
--clearsign >/dev/null
for CHANNEL in $CHANNELS; do
...
doneThis is consistent with how every other gpg operation in this script already supplies --passphrase "${GPG_PASSPHRASE:-}" — the RPM signing loop is the lone exception.
Conventions
--addsign adds a signature, it does not replace one
tools/distribution-drafts/yum-repo/scripts/regenerate-yum-metadata.sh, line 28:
# Re-runs are idempotent (rpm --addsign replaces any existing signature).rpm --addsign appends a new signature header; rpm --resign is the replacing form. In the normal workflow path this does not cause problems (the slot step overwrites the RPM with a fresh unsigned file before this script runs), but the comment is factually wrong and will mislead anyone running the script manually against an already-signed pool. Suggest changing the comment to:
# Re-runs against a freshly slotted (unsigned) RPM are safe.
# If signing a pre-signed RPM, use --resign instead of --addsign to avoid
# accumulating duplicate signature headers.RPM glob loosening is a no-op
.github/workflows/release.yml, lines 439 / 498 (original → new):
- artifacts/wheels/${{ env.WHEELS_VERSION }}/wheels-*.x86_64.rpm
+ artifacts/wheels/${{ env.WHEELS_VERSION }}/wheels*.x86_64.rpmThe old glob wheels-*.x86_64.rpm already matched both wheels-4.0.0.x86_64.rpm (stable) and wheels-be-4.0.0-snapshot.N.x86_64.rpm (bleeding-edge), because * can expand to be-4.0.0-snapshot.N. The .deb fix (wheels_* → wheels*) was necessary; the RPM companion change was not. No functional impact, but worth removing to keep the diff minimal and avoid the implication that the old RPM glob was broken.
Commits
Single commit, DCO Signed-off-by present, subject under 100 chars, conventional type fix is correct. No issues.
Docs
No CHANGELOG.md entry under [Unreleased]. Given that every other recent CI/infrastructure fix in this repo (e.g. cfheader / cfcache attributeCollection fixes) has a changelog entry, a line under ### Fixed such as:
- Linux package CI: derive CHANNEL from version string so snapshot builds produce
wheels-be_*artifacts; fix RPM signing, empty apt pool, and GPG ownertrust fingerprint (#2759)
…would be consistent and helps future backfillers understand when the apt/yum pipeline became operational.
Wheels Bot — Reviewer B (round 1)A's review is technically sound and appropriate. The passphrase-priming finding is a real correctness issue, no false positives were detected, and CHANGES_REQUESTED is the right verdict. One precision note on the passphrase concern below, but it does not change the recommendation. SycophancyNone detected. A requests changes and backs every finding with cited lines. False positivesNone detected.
Missed issuesOne precision gap worth noting: the passphrase issue strictly affects the first channel only in the loop. After the first channel's The Verdict alignmentA's CHANGES_REQUESTED is fully consistent with the findings. The passphrase issue is a genuine blocker for any passphrase-protected production key, and the supporting nits (comment text, no-op glob, CHANGELOG) are correctly ranked below it. ConvergenceAligned. A's diagnosis of the passphrase concern is correct, the two comment nits are valid, and the no-op RPM glob observation is accurate. Joint recommendation: fix the passphrase seeding (add a dummy clearsign before the channel loop as A suggests), correct the |
Wheels Bot — Address Review (round 1)Applied consensus findings from Reviewer A and Reviewer B's
No skipped findings — every item in the converged-changes summary was addressed. Reviewer B's "first channel only" precision note on the passphrase issue was incorporated into the code comment but did not change the fix shape (the script signs Test runner: the PR touches only the Linux distribution-prep shell scripts, a GitHub Actions workflow, and The new commit will trigger a fresh Reviewer A run on the updated SHA. Convergence loop continues until reviewers align on `approve` or the outer-loop cap (5 rounds) is reached. |
…ux switching semantics Round-1 reviewer findings on PR #2777: A's Nit 1 — primary install Tabs at lines 104-105 (Debian/Ubuntu BE) and 112 (Fedora/RHEL BE) of `release-channels.mdx` still resolved to 404. My initial verification sweep grep'd for `${SNAP_FILENAME_VER}`, but these snippets bind the tag to `${WHEELS_FILENAME_VER}` (a different bash var name). The fix is the same — point at the `wheels-be_` / `wheels-be-` artifacts. A's Nit 2 + B's catch — the "Switching channels" section had three related staleness bugs after #2759 renamed the BE package: 1. Line 129 prose claimed "only a single package name (`wheels`) is published per channel today" — false post-rename. 2. Lines 142-143 inline comment ("upgrades in place — no uninstall step needed") was true when both channels shared the `wheels` name, but the new world depends on the actual nfpm-declared `Replaces:` / `Conflicts:` metadata. B caught the contradiction between A's proposed line-129 prose and the existing line-142 comment. 3. Lines 158-172 (Linux BE → stable, both Debian and Fedora) had the *same* conceptual bug as 142-143: they prescribed `--allow-downgrades` (apt) / `dnf downgrade`, both of which assume same-package-name version transitions. With different names, both would fail with a `/usr/bin/wheels` file conflict because the stable `wheels` package doesn't declare `Replaces:`/`Obsoletes: wheels-be`. Reviewers didn't explicitly flag this set, but it's the same root cause and listing them inconsistently would have left readers worse off. Verified the actual nfpm metadata before rewriting (so the prose matches what the packages really declare): wheels-be deb: Replaces: wheels + Conflicts: wheels wheels-be rpm: Conflicts: wheels (no Obsoletes) wheels deb: no Replaces/Conflicts against wheels-be wheels rpm: no Conflicts/Obsoletes against wheels-be The new prose at line 129 explains the asymmetry up front; each snippet now carries a short comment naming the specific metadata that drives its action (or the lack of metadata that requires the explicit `apt remove` / `dnf remove`). Stable-channel snippets and stable install Tabs are unchanged. Signed-off-by: Peter Amiri <peter@alurium.com>
…be_* (#2777) * docs(web/guides): correct Linux bleeding-edge install URLs to wheels-be_* PR #2759 (2026-05-18) renamed the snapshot Linux artifacts from `wheels_*` to `wheels-be_*` (debs) and `wheels-be-*.x86_64.rpm` (rpms) so the package name itself differentiates the channel. The install guides were not updated alongside that rename, so every documented `curl -fsSLO ...` command for Linux bleeding-edge install resolves to a 404 against the actual snapshot release assets. Verified against v4.0.2-snapshot.1923 (published 2026-05-20): Guide says: .../wheels_4.0.2.snapshot.1923_amd64.deb → 404 Actual asset: .../wheels-be_4.0.2.snapshot.1923_amd64.deb Fix all six pages where the snippets / prose examples appear (three unique pages mirrored across v4-0-0 and v4-0-1-snapshot doc versions): start-here/installing.mdx — "Want bleeding-edge?" aside start-here/release-channels.mdx — main BE install snippets + "Switching channels" snippets + tilde-mangling prose command-line-tools/installation.mdx — bleeding-edge install snippets The substitutions are scoped to bleeding-edge contexts (snippets using `${SNAP_FILENAME_VER}` and prose `wheels_4.0.0.snapshot.*` filename examples). Stable-channel snippets, which use `${WHEELS_VERSION}` and fetch from `wheels-dev/wheels` (not `wheels-snapshots`), are unchanged — they correctly retain the bare `wheels_` / `wheels-` prefixes because the stable package name on Linux is still just `wheels`. Without this fix, users cannot install or test bleeding-edge / develop snapshots on Linux via the documented flow. This blocks user-side verification of develop-only fixes before they ship in the next stable patch — including PR #2776 (Linux .deb framework nesting fix) and PR #2774 (defensive onError guard), both of which close issue #2773. Signed-off-by: Peter Amiri <peter@alurium.com> * docs(web/guides): fix release-channels.mdx — missed BE Tab URLs + Linux switching semantics Round-1 reviewer findings on PR #2777: A's Nit 1 — primary install Tabs at lines 104-105 (Debian/Ubuntu BE) and 112 (Fedora/RHEL BE) of `release-channels.mdx` still resolved to 404. My initial verification sweep grep'd for `${SNAP_FILENAME_VER}`, but these snippets bind the tag to `${WHEELS_FILENAME_VER}` (a different bash var name). The fix is the same — point at the `wheels-be_` / `wheels-be-` artifacts. A's Nit 2 + B's catch — the "Switching channels" section had three related staleness bugs after #2759 renamed the BE package: 1. Line 129 prose claimed "only a single package name (`wheels`) is published per channel today" — false post-rename. 2. Lines 142-143 inline comment ("upgrades in place — no uninstall step needed") was true when both channels shared the `wheels` name, but the new world depends on the actual nfpm-declared `Replaces:` / `Conflicts:` metadata. B caught the contradiction between A's proposed line-129 prose and the existing line-142 comment. 3. Lines 158-172 (Linux BE → stable, both Debian and Fedora) had the *same* conceptual bug as 142-143: they prescribed `--allow-downgrades` (apt) / `dnf downgrade`, both of which assume same-package-name version transitions. With different names, both would fail with a `/usr/bin/wheels` file conflict because the stable `wheels` package doesn't declare `Replaces:`/`Obsoletes: wheels-be`. Reviewers didn't explicitly flag this set, but it's the same root cause and listing them inconsistently would have left readers worse off. Verified the actual nfpm metadata before rewriting (so the prose matches what the packages really declare): wheels-be deb: Replaces: wheels + Conflicts: wheels wheels-be rpm: Conflicts: wheels (no Obsoletes) wheels deb: no Replaces/Conflicts against wheels-be wheels rpm: no Conflicts/Obsoletes against wheels-be The new prose at line 129 explains the asymmetry up front; each snippet now carries a short comment naming the specific metadata that drives its action (or the lack of metadata that requires the explicit `apt remove` / `dnf remove`). Stable-channel snippets and stable install Tabs are unchanged. Signed-off-by: Peter Amiri <peter@alurium.com> * docs(web/guides): name the actual Conflicts declaration in BE→stable comments Reviewer A round-2 nit on PR #2777: the BE → stable (Debian) snippet's comment said apt "would fail with a /usr/bin/wheels file conflict", framing the failure mode as a dpkg-level file-ownership conflict. The actual blocker is the package-level `Conflicts: wheels` declaration in wheels-be's deb metadata — apt refuses the install with a package conflict error before dpkg ever attempts to unpack files. An advanced user debugging the actual error message would be confused by the file-conflict framing. Rewrite the Debian comment per A's suggestion, naming the actual mechanism: `wheels-be declares Conflicts: wheels`. Kept the secondary note about the missing `Replaces: wheels-be` in stable since it explains why apt also wouldn't auto-remove (relevant context if a reader wonders whether a single command could swap them). Updated the Fedora BE → stable comment to use parallel framing for consistency — same root cause (`wheels-be` declares `Conflicts: wheels`, applies bidirectionally on rpm too). Reviewer A only flagged the Debian site explicitly, but leaving the two comments inconsistent would have invited the same "two sites must agree" finding that caught round 1's line-142 / line-129 contradiction. Signed-off-by: Peter Amiri <peter@alurium.com> --------- Signed-off-by: Peter Amiri <peter@alurium.com>
Summary
End-to-end smoke test of the Phase 2 bucket-repo templates against real
v4.0.0 artifacts surfaced five blockers. Fixed together because they all
prevent issue #2605 (apt.wheels.dev / yum.wheels.dev standup) from
shipping. Single PR to keep the dependency chain reviewable in one diff.
Bugs fixed
CHANNEL: stable— every snapshot buildproduced
wheels_<v>_amd64.deb, but the Phase 2 BE receiver fetcheswheels-be_<v>_amd64.deb→ guaranteed 404 on every snapshotdispatch. Derived from
env.WHEELS_VERSIONinstead.wheels_*_amd64.debdidn't match thewheels-be_*filenames a fixed-CHANNEL build would produce.Loosened to
wheels*_amd64.deb.brand-new dist). Added
mkdir -p pool/<dist>guard soapt-ftparchive sees an empty scan dir instead of a missing one.
gpgcheck=1clients →dnf refused with
"Package is not signed: GPG check FAILED".regenerate-yum-metadata.shnow signs each RPM withrpm --addsignbefore
createrepo_c. Also pinned%__gpgsince Ubuntu runnersship
gpg(rpm's macro defaults togpg2, not present on Ubuntu).gpg --import-ownertrustgot the 16-char key_id, not the40-char fingerprint → emitted
\"gpg: error in '[stdin]': invalid fingerprint\"every run. Cosmetic but noisy in CI; fixed in bothreceiver workflows.
Docs side: README + placeholder files updated to point at
op://Wheels/wheels-linux-repo-signing/(Wheels project vault onmy.1password.com) rather than
op://Infrastructure/(PAI tenant).Test plan
Local end-to-end via Docker (no operational prereqs):
staged from the template dirs
regenerate-apt-metadata.sh→ producesRelease/Release.gpg/InRelease/Packages.gzfor bothstable + bleeding-edge distributions (empty BE pool no longer
crashes the apt regen)
regenerate-yum-metadata.sh→ signseach RPM, runs
createrepo_c, signsrepomd.xml.asc.rpm --checksigreportsdigests signatures OKapt update && apt-cache policy wheelsfinds 4.0.0;
apt-get download wheelsreturns the 56MB .deb,MD5 matches
Packages.gzdnf install -y --downloadonly wheelsreports `Complete!` with all 18 deps including the signed
wheels-4.0.0.x86_64.rpmProduction validation deferred to after operational standup (mint
signing key, create the two bucket repos, bind Cloudflare Pages,
set CI secrets — see #2605 + the updated
tools/distribution-drafts/linux-packages/README.md).Risk
start producing
wheels-be_*.deb/wheels-be-*.rpmartifacts inthe snapshot release. These are additive — no existing consumer
references the old
wheels_<...snapshot>...filename for snapshots(Phase 1 docs reference stable
wheels_4.0.0_amd64.deb, notsnapshot artifacts). Bleeding-edge users currently install via
brew install wheels-be, which uses the .tar.gz path.produces
wheels-be_*files, which only happens on develop pushes.🤖 Generated with Claude Code