Skip to content

ci(merge-queue): event-driven reconciles for checks, priority labels, and reviews - #10571

Merged
davidfirst merged 9 commits into
masterfrom
merge-queue-check-suite-trigger
Aug 5, 2026
Merged

ci(merge-queue): event-driven reconciles for checks, priority labels, and reviews#10571
davidfirst merged 9 commits into
masterfrom
merge-queue-check-suite-trigger

Conversation

@davidfirst

@davidfirst davidfirst commented Aug 5, 2026

Copy link
Copy Markdown
Member

GitHub throttles scheduled workflows far beyond the */5 spec (20m+ gaps observed today), so any queue transition that relied on the cron left green PRs sitting idle. This makes the remaining cron-only transitions event-driven; the cron stays as a last-resort safety net.

  • check_suite: completed — a queued PR's CI finishing fired no event before; now the green→turn handoff happens in seconds. Master suites are filtered out (push/repository_dispatch already cover master's transitions); fork-PR suites still pass. No self-recursion: the workflow's own runs complete their suites via GITHUB_TOKEN, whose events never trigger workflows.
  • pull_request_target: labeled/unlabeled — applying merge-queue:priority now reorders the queue immediately (filtered in the job condition to exactly that label on queued PRs).
  • workflow_run on a new secret-less merge-queue-review-ping workflow — an approval landing on a queued PR un-sticks a demoted winner right away. Reviews can't trigger the reconcile directly: pull_request_review executes the PR merge commit's workflow copy, which must never see CIRCLE_TOKEN; the no-op bridge absorbs the untrusted context and its completion lands on master's trusted copy.

@qodo-free-for-open-source-projects

Copy link
Copy Markdown

PR Summary by Qodo

CI: trigger merge-queue reconcile on check_suite completion

✨ Enhancement ⚙️ Configuration changes 🕐 Less than 10 minutes

Grey Divider

AI Description

• Trigger merge-queue reconcile on CI completion to reduce green-to-turn latency.
• Keep cron as a safety net while relying on event-driven reconciliation.
• Document why check_suite is safe (idempotent, burst-collapsed, no recursion).
Diagram

graph TD
  A["CI check_suite completed"] --> B["merge-queue.yml"] --> C["reconcile job"] --> D["GitHub API"] --> E["merge queue state"]
  F["schedule (cron)"] --> B
  G["PR/push/dispatch events"] --> B
Loading
High-Level Assessment

The following are alternative approaches to this PR:

1. Use check_run.completed instead of check_suite.completed
  • ➕ More granular (fires per check run), potentially earlier signal for partial completion.
  • ➖ Higher event volume and more burstiness; more filtering needed to detect suite-level completion.
  • ➖ More chances to run reconcile prematurely and rely on additional gating logic.
2. Use workflow_run.completed for the CI workflow(s)
  • ➕ Tied directly to specific workflows that matter for the queue; easier targeting.
  • ➕ Can reduce noise compared to all check suites in the repo.
  • ➖ Requires enumerating/maintaining workflow names and handling multiple CI workflows.
  • ➖ May miss non-Actions checks reported via checks API that still contribute to suite completion.

Recommendation: The chosen check_suite.completed trigger is the best fit for reducing green-to-turn latency with minimal maintenance. It reacts to the repository’s canonical “suite finished” signal (including non-Actions checks), keeps reconcile idempotent/cheap, and relies on existing concurrency to collapse bursts; keeping cron as a safety net is appropriate.

Files changed (1) +8 / -0

Other (1) +8 / -0
merge-queue.ymlTrigger reconcile on check_suite completion +8/-0

Trigger reconcile on check_suite completion

• Adds a check_suite: [completed] workflow trigger so merge-queue reconciliation runs immediately after CI finishes, instead of waiting on throttled cron. Extends comments to document event rationale and why it won’t self-recurse (GITHUB_TOKEN-generated events don’t trigger workflows and default-branch workflow copy runs).

.github/workflows/merge-queue.yml

@qodo-free-for-open-source-projects

qodo-free-for-open-source-projects Bot commented Aug 5, 2026

Copy link
Copy Markdown

Code Review by Qodo

🐞 Bugs (4) 📘 Rule violations (0) 📎 Requirement gaps (0) 🎨 UX issues (0) 🔗 Cross-repo conflicts (0) 📜 Skill insights (0)

Grey Divider


Action required

1. PR-controlled review workflow 🐞 Bug ⛨ Security
Description
The new merge-queue-review-ping workflow runs on pull_request_review using the PR’s workflow
definition, so a same-repo PR can modify this workflow to read and exfiltrate repository secrets
(permissions: {} limits GITHUB_TOKEN scopes but does not prevent secrets.* access). This undermines
the intended secret-isolation model described in merge-queue.yml and can leak CIRCLE_TOKEN or other
repo secrets.
Code

.github/workflows/merge-queue-review-ping.yml[R15-19]

+on:
+  pull_request_review:
+    types: [submitted, dismissed]
+
+permissions: {}
Evidence
The bridge workflow is triggered by pull_request_review, which (per the repo’s own comments)
executes the PR merge commit’s workflow copy; that makes the new workflow definition PR-modifiable.
Separately, the merge-queue workflow uses secrets.CIRCLE_TOKEN, establishing that a high-value
secret exists in this repo and is relevant to the threat model; permissions: {} on the ping
workflow does not prevent adding secrets.* references in a PR-modified copy.

.github/workflows/merge-queue-review-ping.yml[15-29]
.github/workflows/merge-queue.yml[23-27]
.github/workflows/merge-queue.yml[96-101]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
A workflow triggered by `pull_request_review` executes the PR’s version of the workflow file. Even though the new `merge-queue-review-ping` workflow currently does a no-op and sets `permissions: {}`, a same-repo PR can modify this workflow to reference `secrets.*` and exfiltrate them during a review event.
## Issue Context
`merge-queue.yml` explicitly treats PR-controlled workflow execution as unsafe around `CIRCLE_TOKEN`, but the new bridge workflow is itself PR-controlled under `pull_request_review`.
## Fix Focus Areas
- .github/workflows/merge-queue.yml[61-101]
- .github/workflows/merge-queue-review-ping.yml[15-29]
## Suggested remediation
1. Move `CIRCLE_TOKEN` (and any other high-value secrets used by merge-queue) from repository-level secrets into a GitHub **Environment** (e.g., `merge-queue`).
2. Add `environment: merge-queue` to the `reconcile` job in `merge-queue.yml`.
3. Configure the environment in GitHub settings to **restrict deployment branches to `master` only** (or require approvals), so PR refs (including `pull_request_review` runs) cannot access those environment secrets even if a PR edits the ping workflow.
This keeps the bridge workflow safe even if a PR attempts to modify it to access secrets.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools



Remediation recommended

2. Refresh aborts whole reconcile 🐞 Bug ☼ Reliability
Description
refreshUnknownMergeability() awaits githubRequest() without error isolation, so any transient REST
failure will throw and abort the entire reconcile run before winner selection/status updates. This
introduces a new single-point failure on the settled-master path, leaving gate statuses and queue
handoff stale until the next event/cron run.
Code

.github/scripts/merge-queue.js[R265-268]

+      await sleep(delayMs);
+      const fresh = await githubRequest('GET', `/repos/${OWNER}/${REPO}/pulls/${pullRequest.number}`);
+      if (fresh.mergeable === null) continue;
+      pullRequest.mergeable = fresh.mergeable ? 'MERGEABLE' : 'CONFLICTING';
Evidence
The new function calls githubRequest() inside its retry loop with no try/catch, and
githubRequest() throws on any non-OK response. Elsewhere in the same script, API failures are
intentionally caught and logged so one PR doesn’t fail the whole reconcile, showing this new
behavior is an inconsistency introduced by this PR.

.github/scripts/merge-queue.js[87-108]
.github/scripts/merge-queue.js[258-275]
.github/scripts/merge-queue.js[410-423]
.github/scripts/merge-queue.js[622-629]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
`refreshUnknownMergeability()` makes per-PR REST calls via `githubRequest()` (which throws on non-2xx) but does not catch errors. A single API error can abort the entire reconcile run, contrary to the script’s existing pattern of isolating per-PR failures.
### Issue Context
This polling runs when `masterState.settled` and happens before winner selection and any status posting, so a thrown error prevents the reconcile from making progress.
### Fix Focus Areas
- .github/scripts/merge-queue.js[258-275]
- .github/scripts/merge-queue.js[558-561]
### Suggested fix
- Wrap the REST refresh logic in a `try/catch` per PR (or even per attempt).
- On error: log once with PR number + attempt, leave `mergeable` as `UNKNOWN`, and continue to the next entry so the reconcile can still post statuses/update dashboard.
- (Optional) Track refresh failures and set `process.exitCode = 1` at end if you want visibility without breaking the run mid-way.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


3. Sequential polling slows reconcile 🐞 Bug ➹ Performance
Description
refreshUnknownMergeability() adds fixed 5s sleeps in a sequential retry loop for each queued PR with
mergeable=UNKNOWN, delaying winner selection and status updates and consuming a non-trivial portion
of the job’s fixed 10-minute runtime budget. Under larger queues or prolonged UNKNOWN windows, this
increases the risk of delayed handoffs or hitting the workflow timeout before completing the
reconcile loop.
Code

.github/scripts/merge-queue.js[R264-266]

+    for (let attempt = 0; attempt < maxAttempts && pullRequest.mergeable === 'UNKNOWN'; attempt += 1) {
+      await sleep(delayMs);
+      const fresh = await githubRequest('GET', `/repos/${OWNER}/${REPO}/pulls/${pullRequest.number}`);
Evidence
The new refresh loop awaits sleep(delayMs) inside a retry loop and is run before winner selection
when masterState.settled. The workflow caps the reconcile job at 10 minutes, so this added waiting
time directly eats into the available time for the rest of the run.

.github/scripts/merge-queue.js[258-270]
.github/scripts/merge-queue.js[558-595]
.github/workflows/merge-queue.yml[75-96]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
`refreshUnknownMergeability()` uses sequential polling with a fixed 5s sleep per attempt. This can add substantial wall-clock delay before any reconcile actions occur.
### Issue Context
The reconcile job is configured with `timeout-minutes: 10`, so added waiting time directly reduces headroom for the rest of the reconcile (posting statuses for all PRs, dashboard updates, etc.).
### Fix Focus Areas
- .github/scripts/merge-queue.js[258-275]
- .github/workflows/merge-queue.yml[75-96]
### Suggested fix
- Remove the unconditional initial sleep: try an immediate REST read first, then sleep only if still `null`/UNKNOWN.
- Add a global deadline/budget for refresh (e.g., stop refreshing after X seconds total).
- Consider bounded concurrency (e.g., refresh up to N PRs at a time) to prevent worst-case `O(queue_length)` wall time.
- Keep logging when UNKNOWN resolves, but also log when it remains UNKNOWN after the budget so it’s diagnosable.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


4. Review ping too broad 🐞 Bug ➹ Performance
Description
merge-queue-review-ping triggers on every pull_request_review submission (including comment-only
reviews) because it doesn’t filter on github.event.review.state, so it can cause unnecessary
merge-queue reconciles unrelated to approval/dismissal changes. This increases GitHub/CircleCI API
traffic because each reconcile run re-derives master state and scans open PRs.
Code

.github/workflows/merge-queue-review-ping.yml[R16-17]

+  pull_request_review:
+    types: [submitted, dismissed]
Evidence
The ping workflow is configured for review submitted events without any check of the review’s
state, and merge-queue.yml is set up to reconcile on completion of this ping workflow; the reconcile
script performs GitHub and CircleCI API calls per run, so extra triggers translate directly into
avoidable load.

.github/workflows/merge-queue-review-ping.yml[15-26]
.github/workflows/merge-queue.yml[54-58]
.github/workflows/merge-queue.yml[80-86]
.github/scripts/merge-queue.js[149-180]
.github/scripts/merge-queue.js[504-526]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
`merge-queue-review-ping` runs for `pull_request_review: submitted` regardless of the review’s state. That means comment-only reviews (and other non-approval submissions) can trigger `workflow_run` → `merge-queue` reconciles even though the stated intent is to react to approvals/dismissals.
## Issue Context
The downstream `merge-queue` workflow listens for successful completions of `merge-queue-review-ping` and then runs `.github/scripts/merge-queue.js`, which calls GitHub + CircleCI APIs each time.
## Fix Focus Areas
- .github/workflows/merge-queue-review-ping.yml[15-29]
### Suggested change
Tighten the `ping` job condition to only succeed when the review event can actually affect merge eligibility, e.g.:
- keep `dismissed` (review dismissal can change approval requirements)
- for `submitted`, require `github.event.review.state == 'approved'`
Example:

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


View more (1)
5. Broad check_suite trigger ✓ Resolved 🐞 Bug ➹ Performance
Description
check_suite: [completed] runs the reconcile job for every completed check suite in the repo
(including non-PR branches and PRs not in the merge queue), which bypasses the workflow’s existing
effort to avoid unnecessary reconciles. This can increase GitHub/CircleCI API traffic and create
sustained reconcile churn under high CI activity, potentially delaying queue handoffs if runs back
up.
Code

.github/workflows/merge-queue.yml[R42-43]

+  check_suite:
+    types: [completed]
Evidence
The workflow adds a repo-wide check_suite.completed trigger, and the job-level if condition only
special-cases pull_request_target synchronize events—so check_suite events always run the
reconcile job. The reconcile script does non-trivial work each run (CircleCI scans, GitHub GraphQL
pagination, status posting, dashboard issue updates), so triggering it for unrelated suite
completions can create unnecessary load/churn.

.github/workflows/merge-queue.yml[24-44]
.github/workflows/merge-queue.yml[60-69]
.github/scripts/merge-queue.js[149-181]
.github/scripts/merge-queue.js[183-197]
.github/scripts/merge-queue.js[495-526]
.github/scripts/merge-queue.js[579-614]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The workflow now triggers on **all** `check_suite.completed` events. Since the reconcile job always runs for `check_suite` events, unrelated suite completions (e.g., non-PR branches, non-queued PRs) can repeatedly invoke the full reconcile loop.
### Issue Context
The reconcile step runs `.github/scripts/merge-queue.js`, which performs multiple GitHub and CircleCI API calls and posts statuses/updates an issue each run. This is appropriate for queue changes, but can be unnecessary churn when the completed suite is unrelated to the queue.
### Fix Focus Areas
- .github/workflows/merge-queue.yml[60-69]
### Suggested change
Add an event-specific guard to the job `if:` so `check_suite` only triggers reconciles when it’s relevant (e.g., associated with at least one PR, and optionally not `master` since `push: master` already covers it). For example:

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


To customize comments, go to the Qodo configuration screen, or learn more in the docs.

Qodo Logo

Comment thread .github/workflows/merge-queue.yml
@qodo-free-for-open-source-projects

Copy link
Copy Markdown

Code review by qodo was updated up to the latest commit 182cc39

@qodo-free-for-open-source-projects

qodo-free-for-open-source-projects Bot commented Aug 5, 2026

Copy link
Copy Markdown

Code Review by Qodo

Grey Divider

New Review Started

This review has been superseded by a new analysis

Grey Divider

Qodo Logo

@davidfirst davidfirst changed the title ci(merge-queue): reconcile on check_suite completion to cut green-to-turn latency ci(merge-queue): event-driven reconciles for checks, priority labels, and reviews Aug 5, 2026
Comment thread .github/workflows/merge-queue-review-ping.yml
@qodo-free-for-open-source-projects

Copy link
Copy Markdown

Code review by qodo was updated up to the latest commit 4284257

Comment on lines +16 to +17
pull_request_review:
types: [submitted, dismissed]

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remediation recommended

1. Review ping too broad 🐞 Bug ➹ Performance

merge-queue-review-ping triggers on every pull_request_review submission (including comment-only
reviews) because it doesn’t filter on github.event.review.state, so it can cause unnecessary
merge-queue reconciles unrelated to approval/dismissal changes. This increases GitHub/CircleCI API
traffic because each reconcile run re-derives master state and scans open PRs.
Agent Prompt
## Issue description
`merge-queue-review-ping` runs for `pull_request_review: submitted` regardless of the review’s state. That means comment-only reviews (and other non-approval submissions) can trigger `workflow_run` → `merge-queue` reconciles even though the stated intent is to react to approvals/dismissals.

## Issue Context
The downstream `merge-queue` workflow listens for successful completions of `merge-queue-review-ping` and then runs `.github/scripts/merge-queue.js`, which calls GitHub + CircleCI APIs each time.

## Fix Focus Areas
- .github/workflows/merge-queue-review-ping.yml[15-29]

### Suggested change
Tighten the `ping` job condition to only succeed when the review event can actually affect merge eligibility, e.g.:
- keep `dismissed` (review dismissal can change approval requirements)
- for `submitted`, require `github.event.review.state == 'approved'`

Example:
```yaml
if: >-
  github.repository == 'teambit/bit' &&
  github.event.pull_request.auto_merge != null &&
  (
    github.event.action == 'dismissed' ||
    (github.event.action == 'submitted' && github.event.review.state == 'approved')
  )
```

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

@qodo-free-for-open-source-projects

Copy link
Copy Markdown

Code review by qodo was updated up to the latest commit f74619e

@davidfirst
davidfirst enabled auto-merge (squash) August 5, 2026 18:24
Comment on lines +265 to +268
await sleep(delayMs);
const fresh = await githubRequest('GET', `/repos/${OWNER}/${REPO}/pulls/${pullRequest.number}`);
if (fresh.mergeable === null) continue;
pullRequest.mergeable = fresh.mergeable ? 'MERGEABLE' : 'CONFLICTING';

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remediation recommended

1. Refresh aborts whole reconcile 🐞 Bug ☼ Reliability

refreshUnknownMergeability() awaits githubRequest() without error isolation, so any transient REST
failure will throw and abort the entire reconcile run before winner selection/status updates. This
introduces a new single-point failure on the settled-master path, leaving gate statuses and queue
handoff stale until the next event/cron run.
Agent Prompt
### Issue description
`refreshUnknownMergeability()` makes per-PR REST calls via `githubRequest()` (which throws on non-2xx) but does not catch errors. A single API error can abort the entire reconcile run, contrary to the script’s existing pattern of isolating per-PR failures.

### Issue Context
This polling runs when `masterState.settled` and happens before winner selection and any status posting, so a thrown error prevents the reconcile from making progress.

### Fix Focus Areas
- .github/scripts/merge-queue.js[258-275]
- .github/scripts/merge-queue.js[558-561]

### Suggested fix
- Wrap the REST refresh logic in a `try/catch` per PR (or even per attempt).
- On error: log once with PR number + attempt, leave `mergeable` as `UNKNOWN`, and continue to the next entry so the reconcile can still post statuses/update dashboard.
- (Optional) Track refresh failures and set `process.exitCode = 1` at end if you want visibility without breaking the run mid-way.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

Comment on lines +264 to +266
for (let attempt = 0; attempt < maxAttempts && pullRequest.mergeable === 'UNKNOWN'; attempt += 1) {
await sleep(delayMs);
const fresh = await githubRequest('GET', `/repos/${OWNER}/${REPO}/pulls/${pullRequest.number}`);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remediation recommended

2. Sequential polling slows reconcile 🐞 Bug ➹ Performance

refreshUnknownMergeability() adds fixed 5s sleeps in a sequential retry loop for each queued PR with
mergeable=UNKNOWN, delaying winner selection and status updates and consuming a non-trivial portion
of the job’s fixed 10-minute runtime budget. Under larger queues or prolonged UNKNOWN windows, this
increases the risk of delayed handoffs or hitting the workflow timeout before completing the
reconcile loop.
Agent Prompt
### Issue description
`refreshUnknownMergeability()` uses sequential polling with a fixed 5s sleep per attempt. This can add substantial wall-clock delay before any reconcile actions occur.

### Issue Context
The reconcile job is configured with `timeout-minutes: 10`, so added waiting time directly reduces headroom for the rest of the reconcile (posting statuses for all PRs, dashboard updates, etc.).

### Fix Focus Areas
- .github/scripts/merge-queue.js[258-275]
- .github/workflows/merge-queue.yml[75-96]

### Suggested fix
- Remove the unconditional initial sleep: try an immediate REST read first, then sleep only if still `null`/UNKNOWN.
- Add a global deadline/budget for refresh (e.g., stop refreshing after X seconds total).
- Consider bounded concurrency (e.g., refresh up to N PRs at a time) to prevent worst-case `O(queue_length)` wall time.
- Keep logging when UNKNOWN resolves, but also log when it remains UNKNOWN after the budget so it’s diagnosable.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

@qodo-free-for-open-source-projects

Copy link
Copy Markdown

Code review by qodo was updated up to the latest commit fc54f9c

@davidfirst
davidfirst merged commit ed176a2 into master Aug 5, 2026
16 checks passed
@davidfirst
davidfirst deleted the merge-queue-check-suite-trigger branch August 5, 2026 21:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants