ci(llm-gate): mirror conclusion to PR head SHA on workflow_dispatch#288
Conversation
When the LLM-Based Quality Gate is re-fired via `workflow_dispatch` (e.g. after pushing fixes for a WARN, since the gate intentionally does not re-run on `synchronize` for cost reasons), the auto-generated check_run for the "Aggregate and post review" job is attached to the PR's head SHA and appears green in the Checks API — but GitHub's PR `statusCheckRollup` (which branch protection consults) does NOT include workflow_dispatch-triggered check_runs. Result: the required check appears green but the PR stays `mergeStateStatus: BLOCKED`. The only workaround today is toggling PR draft status (or the `llm-gate/override` label) to force a `pull_request` event. This ports the fix already shipped to the open-agreements LLM gate (open-agreements/open-agreements#393, where open-agreements PR 360 has the in-the-wild "2× Aggregate and post review" entries as living proof). Three surgical edits to the aggregate job: 1. Add `checks: write` permission so the mirror step can POST to /repos/.../check-runs without 403. 2. Add `id: compose` to the "Compose and post checklist comment" step so its outcome is addressable from the mirror step. 3. Append a "Mirror conclusion to PR head SHA (workflow_dispatch)" step that POSTs an explicit check_run with the resolved head SHA. This explicit POST DOES appear in `statusCheckRollup`, so branch protection sees the check. The mirror step is guarded by `if: always() && github.event_name == 'workflow_dispatch' && needs.parse-checklist.outputs.head_sha != ''` so it has no effect on normal `pull_request` triggers. `always()` is critical: when the gate finds WARN rows and the github-script step calls core.setFailed(), the default `if: success()` would skip the mirror — the conclusion is derived from `steps.compose.outcome`, so a failed gate still produces a failed mirror (fails closed on WARN). The mirror's behavior under safe-docx's `skip_gate` (package-lock-only PRs) is correct as-is: when skip_gate is true the compose step still succeeds with a SKIPPED comment, so the mirror posts a passing check_run — same semantics branch protection would see from a normal skip_gate PR-triggered run.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
LLM-Based Quality GateOverall: ✅ PASS (14 pass · 0 warn · 14 total)
Full checklist questions
Estimated cost (this run): $0.0135 — 39,565 input + 595 output tokens (≈4 chars/token) on |
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
stevenobiajulu
deleted the
tweak-ci-llm-gate-mirror-check-run-on-workflow-dispatch
branch
Post-merge smoke PASSEDMerged: What was testedThe whole point of this PR is the Target: PR #277 ( Run: ResultsPR #277 head SHA
[
{"completedAt":"2026-05-27T17:24:25Z","conclusion":"SUCCESS","detailsUrl":".../actions/runs/26527086610/job/78134314845"},
{"completedAt":"2026-05-31T00:39:07Z","conclusion":"SUCCESS","detailsUrl":".../runs/78688577890"}
]
This matches the open-agreements PR #360 evidence pattern (the "2× Aggregate and post review" entries the PR description called out). Fix is working in the wild on safe-docx. What was NOT tested
Cleanup
Log: |
1 participant
Summary
Ports the LLM-gate check-mirror fix from open-agreements (open-agreements/open-agreements#393, commit
60582ad) to safe-docx. Three surgical edits to.github/workflows/llm-based-quality-gate.yml:checks: writeto the aggregate job'spermissions:block.id: composeto the "Compose and post checklist comment" step.check_runto/repos/.../check-runs.Why
When the LLM gate is re-fired via
workflow_dispatch(the documented manual re-trigger after pushing fixes for a WARN), GitHub's PRstatusCheckRollup— which branch protection consults — does not includeworkflow_dispatch-triggered check_runs. The check appears green in the Checks API but the PR staysmergeStateStatus: BLOCKED. The only workaround today is toggling PR draft status (or thellm-gate/overridelabel) to force apull_requestevent.The explicit check_run POST DOES appear in
statusCheckRollup, so branch protection sees it.Living evidence in open-agreements
PR open-agreements/open-agreements#360 has two "Aggregate and post review" entries on its head SHA — one auto-generated, one mirrored. That's the in-the-wild proof the same fix works.
Behavior under existing knobs
opened,ready_for_review, label toggle): mirror step is skipped by itsgithub.event_name == 'workflow_dispatch'guard — single check entry, unchanged behavior.skip_gate(package-lock-only PRs): the compose step still succeeds with a SKIPPED comment, so the mirror posts a passing check_run — same semantics branch protection would see from a normal skip_gate PR-triggered run.core.setFailed(), mirror'salways()guard runs it anyway, andCOMPOSE_OUTCOME != "success"produces a failing check_run — fails closed on WARN.Test plan
pull_requestpath (single "Aggregate and post review" check as today).gh workflow run llm-based-quality-gate.yml -R UseJunior/safe-docx -f pr_number=<N>and verify:gh pr view <N> --json statusCheckRollupshows the mirrored check (the auto-generated one will not appear in the rollup — that's the bug being worked around).mergeStateStatusisCLEAN(assuming other required checks pass), notBLOCKED.failure(notsuccess) when the gate finds WARN rows in blocking mode.