ci: detect unreleased work and silently skipped publishes

[abrichr]

Why

PR #51 removed a hosted-recognizer path that uploaded raw audio waveforms to a third party. It merged to main and stayed unreleased while PyPI's only installable version, 1.2.0, still contained the upload path — and two downstream packages resolve openadapt-capture>=1.1.0 unpinned. Nothing alerted. It shipped as 1.2.1 only after a human noticed and dispatched it.

The sibling failure is the same shape from the other end: openadapt-desktop tags desktop-v0.12.0, 0.8.0, 0.7.0, 0.1.0 exist with no release object (a failed macOS x86_64 build, an ungranted native-release environment approval, a wholly cancelled run, and a tomllib import error on a pre-3.11 runner), and the launcher's Release and PyPI Publish runs 30275153205 / 30226357239 were cancelled seconds in. if: failure() sees none of that: cancelled and skipped are not failure.

What this is not

Nothing here publishes anything. Releases stay workflow_dispatch-only by deliberate design — the header of release.yml reads "Publication is a deliberate operation from protected main", and that gate exists because Flow 1.13.0/1.14.0 were yanked for shipping AGPL benchmark files. This is visibility only.

What it is

scripts/check_release_health.py (stdlib only, no dependency install, no lockfile, no cache) evaluates three state-based detectors per release lane declared in .github/release-health.json:

detector	fires when
unreleased-work	main carries commits past the newest release tag that would bump the version under this repo's own [tool.semantic_release.commit_parser_options], main is green, no release run is in flight, and the oldest such commit is past the grace window
tag-without-release	a release-shaped tag has no release object, no run for it is queued/waiting/running, and it is past the grace window — reporting failure/cancelled/skipped, or that the workflow never started at all
unpublished-release	the newest release tag's version is absent from PyPI past the CDN grace window

State-based rather than event-based on purpose. "A release run was cancelled" is not the alarm; "something is still unpublished" is. The launcher's release workflow cancels superseded runs constantly under its concurrency: release group, and a detector that fired on those would go the way of the platform-manifest drift check — correct, permanently red after every release, and therefore indistinguishable from noise.

False positives are the failure mode

--self-test enumerates every transient this repository actually produces and proves the detector stays quiet on each, alongside the cases it must catch. It runs offline in CI whenever the detector itself changes.

A. It FIRES on the states it exists to catch
  [PASS] unreleased fix: commit, green main, past grace -> alert
  [PASS] `type!:` breaking change -> alert
  [PASS] `BREAKING CHANGE:` footer -> alert
  [PASS] tag with a cancelled/failed/never-started publish -> alert
  [PASS] conclusion `skipped` (invisible to `if: failure()`) -> alert
  [PASS] tag exists but PyPI never got the version -> alert
B. It stays QUIET on every transient this repository actually produces
  [PASS] semantic-release + reconcile commits are `chore` -> quiet
  [PASS] launcher's bare `1.10.0` release commit is not a bump -> quiet
  [PASS] docs/ci/chore(deps) merges -> quiet
  [PASS] a fix merged 1h ago is inside the 4h grace -> quiet
  [PASS] a release run is in flight -> quiet
  [PASS] main is red -> quiet (a red main is its own signal)
  [PASS] main CI still running -> quiet
  [PASS] PyPI CDN has not propagated yet (10m) -> quiet
  [PASS] PyPI unreachable -> warning, never an alert
  [PASS] tag awaiting a human `native-release` approval -> quiet
  [PASS] tag pushed 90m ago, installers still building -> quiet
  [PASS] tag whose artifact IS on PyPI -> warning, never an alert
  [PASS] explicitly acknowledged historical tag -> quiet
  [PASS] launcher concurrency cancellations that left no gap -> quiet (the exact runs 30275153205/30276970325)
C. Commit parsing honours THIS repository's configured tags
  [PASS] `perf:` bumps only when patch_tags says so
  [PASS] `feat(scope): ...` -> minor
  [PASS] Merge commits and free text -> no bump

The ~90s window matters most: chore: release 1.2.1 and the launcher's chore(release): reconcile platform manifest are in allowed_tags but in neither minor_tags nor patch_tags, so they produce no bump and no alert; the launcher's bare 1.10.0 subject is not a Conventional Commit at all.

Proof it fires on the real miss

Replaying the exact commits that sat on main between v1.2.0 and the human's dispatch, in the counterfactual where nobody looked:

ALERT [unreleased-work/python] 1 releasable commit(s) unpublished for 9h 47m

main has been carrying 1 releasable commit(s) for 9h 47m past v1.2.0. main CI is green and no release run is in flight, so nothing is going to publish these on its own.

commit	bump	age	subject
27ccd262	patch	9h 47m	fix(audio): make narration capture on-device only and fail closed (#51)

No release workflow run has been recorded for this lane at all. The publish did not fail; it never started.

Publish with: gh workflow run release.yml --repo OpenAdaptAI/openadapt-capture --ref main -f operation=semantic-release

Proof it is quiet today

Run against live origin/main, PyPI and the GitHub API right now:

WARNING: [python] tag v0.1.0 has no GitHub release object, but openadapt-capture 0.1.0
         is installable from PyPI; the artifact shipped and only the release page is absent.
ok: [python] 0 commit(s) since v1.2.1, none of which trigger a version bump.
ok: [python] openadapt-capture 1.2.1 is on PyPI.

No release-health alerts: every release lane is published and current.

Triggers and cost

• schedule every 3h — the actual detector; bounds "how long can a releasable fix sit unnoticed" to ~7h with the 4h grace.
• workflow_run on Release and PyPI Publish completed with a non-success conclusion — catches cancelled and skipped, including a run cancelled at a human approval gate.
• pull_request on the detector's own files — runs --self-test.

One stdlib step, no dependency install, no cache. Python pinned to 3.12, because openadapt-desktop run 29514474536 died on ModuleNotFoundError: No module named 'tomllib'; the script additionally degrades to documented defaults with a warning rather than crashing on an old interpreter.

Issue behaviour

Exactly one open issue per repository, rewritten in place on each run (editing a body does not notify, so a persistent gap is not a daily ping) and closed automatically with a comment once every gap is closed. Never duplicated.

🤖 Generated with Claude Code

https://claude.ai/code/session_01NyCHrzA1psrKMFfroYbzaM

---

Rollout

The detector script is byte-identical across repositories (sha256:97ecff275830e3be626e4235a74da260ed79085735d9cf8fe2a1917662d77d27); only .github/release-health.json differs. Companion PRs: OpenAdaptAI/OpenAdapt#1063 and OpenAdaptAI/openadapt-desktop#70. A ready-to-apply patch for openadapt-flow exists but is not opened here, because that repository is owned by another stream this session.

[abrichr]

Holding this for an 80/20 revision before merge.

The signal is worth adding, but 1,399 new lines plus eight scheduled runs per day is disproportionate for release visibility and creates a new subsystem to maintain. Please replace it with one small release-state check attached to the cheapest justified existing event or cadence, keeping only the actionable gap contract.

Two correctness details must also change:

1. Use the REST issues API for create, update, close, and comment. The current gh issue edit path hits the Projects Classic GraphQL deprecation with the current CLI, and the offline self-test does not exercise this core mutation path.
2. Fix tag-to-run association for workflow_dispatch semantics. The release workflow is dispatched from main, so matching head_branch to the eventual tag can report workflow never started even when the main-dispatched publish failed.

Please exercise the simplified path end to end against a temporary or recorded issue state, then re-run exact-head CI. Nothing here needs to publish automatically.