Stop an AUR outage from masking release verification - #600
Conversation
The AUR publish ran as a step inside the release job, sitting after both GoReleaser and attestation. When aur.archlinux.org went down for maintenance during v0.8.0, that step failed with the release already published — turning the job red and taking macos-verify, windows-verify, nix-verify and sync-skills down with it, all four skipped because they are `needs: [release]`. So an unreachable third party could silently cost us every signing verification on a release that had already shipped. Split AUR into its own continue-on-error job. The release job now ends at attestation, so its conclusion reflects only what we control, and the verification jobs run regardless of the AUR's availability. Three attempts with backoff absorb brief blips, and a failure opens an issue rather than passing quietly. Add an aur-publish workflow_dispatch workflow for the longer outages retries cannot cover. publish-aur.sh builds the PKGBUILD from the published release assets and no-ops when the AUR copy is current, so republishing an existing version is idempotent and needs no new tag. It refuses versions that are not actually published, so a typo cannot point the PKGBUILD at assets that were never built.
Sensitive Change Detection (shadow mode)This PR modifies control-plane files:
|
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: cec366f187
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
The AUR outage also skipped sync-skills, and basecamp/skills is now stale against v0.8.0 — the skills tree changed by +375/-54 lines since v0.7.2, so this is a real gap, not a no-op. release.yml no longer fails that way, but a sync that is skipped or fails on its own still needs a way back that does not involve cutting a new tag. Checks out the tag rather than main, so the sync mirrors the skills tree as it was released even after main moves on, and resolves the tagged commit for provenance instead of trusting github.sha, which points at the dispatching ref. Carries the script's DRY_RUN=local mode through as a preview toggle.
|
Pushed a second commit — the same outage left a second gap.
So this is a real gap, not a no-op sync. Added Two details worth review:
|
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9f1eb5bd04
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.
Suppressed comments (2)
.github/workflows/release.yml:295
- Unlike the
sync-skillsjob (which sets a job-levelconcurrency: group: sync-skillsthat matches the manualsync-skills.ymlworkflow), thisaur-publishjob has no concurrency group, while the new manualaur-publish.ymlusesgroup: aur-publish. Because concurrency groups only coordinate when the group names match, the release-time AUR publish and a manually-dispatchedaur-publish.ymlrun can execute simultaneously, both cloning and pushing to the same AUR git repo.scripts/publish-aur.shhas no non-fast-forward retry (unlikesync-skills.sh), so a concurrent push would fail and open a needless failure issue. Add the matching group here to serialize AUR pushes across both workflows.
continue-on-error: true
runs-on: ubuntu-latest
.github/workflows/sync-skills.yml:46
- The manual sync tag validation accepts prerelease tags (the optional
(-[a-zA-Z0-9.]+)?group), but the automated release-time sync job never syncs prereleases — release.yml:520 gates it on!contains(github.ref_name, '-'), and the sibling manualaur-publish.yml(line 45) also rejects any-suffix. Since this workflow's stated purpose is to recover a shipped release sync that the automated flow missed, accepting a prerelease tag here has no recovery use and could push prerelease skills into thebasecamp/skillsdistribution repo. Consider tightening the regex to reject prereleases so it matches the automated policy. (Optional — confirm the intended behavior.)
if [[ ! "$TAG" =~ ^v[0-9]+\.[0-9]+\.[0-9]+(-[a-zA-Z0-9.]+)?$ ]]; then
Checking that the requested tag has a published release was not enough. publish-aur.sh rewrites pkgver unconditionally, so dispatching an older but perfectly valid release — after a newer one had already reached the AUR — would push every Arch user backwards. Compare against what is actually in the AUR via its RPC rather than against GitHub's release list, since the AUR is the thing being protected and may lag or lead for unrelated reasons. sort -V does the comparison so 0.10.0 ranks above 0.9.0 instead of below it. Fails closed when the RPC is unreachable: publishing blind risks a silent downgrade for every Arch user, while re-dispatching once the AUR responds costs nothing. Republishing the current version stays a permitted no-op, which is the ordinary recovery case.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: dbe790477f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.
Suppressed comments (1)
.github/workflows/release.yml:294
- Optional: the two skills-sync paths are serialized against each other — the release-time
sync-skillsjob (line 522) and the manualsync-skills.yml(line 30) share the concurrency groupsync-skills. The two AUR paths are not: thisaur-publishjob has noconcurrencyblock, while the manualaur-publish.ymluses groupaur-publish. Since both AUR paths push to the sameaur@aur.archlinux.org/basecamp-cli.gitrepo, a manual recovery dispatched while a release-time publish is running can race on the AUR push. The retry loop re-clones, so a transient non-fast-forward would usually self-heal, but adding a matching concurrency group here would make the two AUR paths serialize the same way the sync-skills paths already do.
aur-publish:
name: Publish to AUR
needs: [release]
if: startsWith(github.ref, 'refs/tags/v') && !contains(github.ref_name, '-')
continue-on-error: true
Four more holes in the manual recovery workflows, all reachable: Preserve newer AUR packaging revisions. Comparing only pkgver meant an AUR-side packaging fix at 0.8.0-2 counted as "same version", and publish-aur.sh, which hardcodes pkgrel=1, would then quietly replace it with 0.8.0-1. Refuse when the published revision is above -1; republishing over -1 stays the permitted no-op. Serialize the two publishers. The recovery workflow's concurrency group covered only itself, so a recovery that had already passed its version check could clone after a newer automatic publish landed and push the older PKGBUILD as a fast-forward. Both now share the aur-publish group. Require the latest stable tag when syncing skills. The tag pattern accepted prereleases, which release.yml deliberately excludes, and older stable tags, which would roll the distribution repo back — sync-skills.sh mirrors the tree wholesale. Unlike the AUR there is no independent record of what basecamp/skills holds, and the only reason to run this by hand is that the newest release failed to sync, so require exactly that release. Make the dry run honest. DRY_RUN=local never clones the target and diffs against an empty repo, so every skill reads as newly added and the deletions a real sync would make never appear — the opposite of what a preview is for. Use the script's remote mode, which clones the real target and stops before pushing, and scope the token to read for dry runs so the preview cannot write regardless.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 139b1fcb0f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.
Suppressed comments (1)
.github/workflows/release.yml:347
- Optional (maintainability): the AUR SSH/
git configsetup and the identical 3-attempt retry loop are now duplicated verbatim between this job and.github/workflows/aur-publish.yml(lines 118-135). Any future change to the backoff, attempt count, or SSH host config has to be made in both places or they will drift. Sincepublish-aur.shalready no-ops when the PKGBUILD is current, consider moving the retry/backoff intoscripts/publish-aur.sh(and optionally a small shared setup snippet) so both callers just invoke the script once.
for attempt in 1 2 3; do
if scripts/publish-aur.sh "$VERSION"; then
exit 0
fi
if [ "$attempt" -lt 3 ]; then
echo "AUR publish attempt ${attempt} failed; retrying in $((attempt * 60))s"
sleep $((attempt * 60))
fi
done
echo "AUR publish failed after 3 attempts"
exit 1
PKGBUILD(5) permits a dotted subrelease, so pkgrel can be 1.1. `[ -gt ]` rejects that as a non-integer and exits 2 — and the 2>/dev/null guarding the test turned that error into a silent "not greater", so the clobber the previous commit set out to prevent went through anyway for exactly the revisions most likely to carry a hand-made packaging fix. Compare with sort -V instead, which orders dotted revisions correctly and cannot fail open: 1.10 ranks above 1.1, and anything other than a bare 1 is refused.
What happened
During the v0.8.0 release,
aur.archlinux.orgwas down for maintenance:AUR publishing was a step inside the
releasejob, positioned after bothGoReleaser (
:241) and attestation (:264). So the step failed with the releasealready published, which turned the job red — and took all four verification
jobs with it, skipped because they are
needs: [release]:Verify Windows signingVerify macOS signingVerify Nix flakeSync skillsAn unreachable third party silently cost us every signing verification on a
release that had already shipped.
nix-verifyandsync-skillswere alreadycontinue-on-error, but that does nothing when the job they depend on fails.The fix
Split AUR into its own
continue-on-errorjob. Thereleasejob now ends atattestation, so its conclusion reflects only what we control, and the four
verification jobs run regardless of whether the AUR is reachable.
sync-skillsidiom) instead ofpassing quietly, and always emits a
::error::annotation.Add
aur-publish.yml, aworkflow_dispatchrecovery path for outages longerthan retries can cover.
scripts/publish-aur.shbuilds the PKGBUILD entirely fromthe published GitHub release assets and no-ops when the AUR copy is already
current, so republishing an existing version is idempotent and needs no new tag.
It refuses versions that are not actually published, so a typo cannot push a
PKGBUILD pointing at assets that were never built.
Note on the alternative
Step-level
continue-on-errorwould have been a smaller diff, but it leaves afailed AUR publish sitting inside an otherwise-green job — it reads as a clean
release unless someone opens the job. A separate job keeps the failure visible on
its own.
Verification
bin/ciexit 0,actionlintclean,zizmorreports no findings.preventing recurrence, not recovering that release.
Summary by cubic
Isolates AUR publishing into a continue-on-error job so AUR outages can’t block or hide release verification, and adds manual recovery workflows for AUR and skills sync.
aur-publishjob; thereleasejob now ends at attestation so macOS/Windows signing, Nix flake, and skills verification always run.aur-publish.yml(workflow_dispatch) to republish a released version; verifies the version exists, refuses downgrades by checking the current AUR version via RPC (fails closed), preserves newer AURpkgrel(handles dotted revisions viasort -V), and shares aconcurrencygroup with the automatic publisher to prevent races; usesscripts/publish-aur.sh(idempotent, no new tag).sync-skills.yml(workflow_dispatch) to mirror theskills/tree from a release tag intobasecamp/skills; requires the latest stable tag, checks out the tag and records the tagged SHA, and supports an honestdry_run(remote clone with read-only token).Written for commit 1d59858. Summary will update on new commits.