ci: gate ray-update PRs on green buildkite/template-test#642
Open
ci: gate ray-update PRs on green buildkite/template-test#642
Conversation
New workflow that fails when a PR carries the ray-update label without a passing buildkite/template-test commit-status on its head SHA. PRs without the label always pass. Triggers on pull_request changes and on status events so it re-evaluates automatically when the Buildkite build result lands — no manual rerun needed. To activate enforcement, add this workflow's check (ray-update-requires-template-test / check) to the main branch protection's required status checks. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replace the single 'needs a green build' error with a case that names each failure mode distinctly: 'missing' (never ran) vs 'pending' vs 'failure'/'error' vs unexpected. Each suggests the right next action. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a new GitHub Actions workflow to enforce the repo’s existing template-validation process for ray-update PRs by requiring a green buildkite/template-test status before merge. This fits the codebase’s template automation flow, where /test-template is the mechanism that triggers template CI.
Changes:
- Adds
ray-update-requires-template-testworkflow triggered onpull_requestandstatus. - Resolves the PR/head SHA, checks for the
ray-updatelabel, and inspects the latestbuildkite/template-testcommit status. - Emits actionable failure messages for missing, pending, failed, or unexpected statuses.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Two edge cases the previous logic mishandled: 1. status events can return multiple PRs sharing a commit (stacked PRs, backports). Picking [0] silently missed the ray-update PR if a non-ray-update PR happened to come first. Now iterate all PRs that contain the commit. 2. Stale SHA: status events arrive for a specific commit; if a newer commit was pushed in the meantime, gating on the event's SHA could report 'green' based on an outdated commit's status. Always re-fetch the PR's current head (headRefOid) via the API and gate on that. Also: on status events, only gate PRs whose current head matches the status's SHA — the check-run we produce is attached to that commit, so only those PRs see it via branch protection. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 1 out of 1 changed files in this pull request and generated 1 comment.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
…pdate Per the review: if a fork PR is labeled ray-update, this gate would block forever — /test-template (in .github/workflows/test-template.yaml) explicitly refuses cross-repo PRs for security, so buildkite/template-test can never appear, so the gate stays red with no useful explanation. Add isCrossRepository to the PR query and emit a fork-specific error that names the actual constraint and points at the workaround (mirror to an internal branch). Same outcome (the PR can't merge), but the failure mode is now discoverable instead of silently confusing. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
No semantic change. Cuts the verbose multi-line comment blocks down to single-line whys, collapses the per-PR HEAD/labels/fork echos onto one line, and shortens the case-arm error messages while keeping each distinguishable. 125 → 88 lines. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…-test Quick context for anyone applying the label — the new gate workflow added in this PR enforces it. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
New workflow `ray-update-requires-template-test`: fails when a PR carries the `ray-update` label but its head commit lacks a green `buildkite/template-test` status. PRs without the label always pass.
Why
Today, a `ray-update` PR can technically merge without ever running `/test-template` — the `/template` skill mandates it but only the agent flow enforces it. This workflow adds a hard gate so a green Buildkite build is required for any PR labeled `ray-update` (human-opened or otherwise).
How
Required follow-up to actually enforce
This PR adds the workflow but doesn't enforce it. Once merged, an admin needs to add the workflow's check (`ray-update-requires-template-test / check`) to `main` branch protection → Required status checks. Without that step, the workflow runs but doesn't block merges.
Test plan