Investigate organization-wide GitHub Actions queue saturation blocking current-head PR merges

[seonghobae]

Incident

Multiple current-head PRs in ContextualWisdomLab/naruon are mergeable and have clean/resolved review evidence, but every repository-local required workflow remains in queued state instead of starting. This prevents the central OpenCode/Noema/Strix review and guarded merge scheduler from publishing current-head approvals.

Observed exact-head examples on 2026-08-03 UTC:

• naruon#1192 head 8792334e9b85e901d72f7115bf2a439db61efbbb
  • Application CI 30786852446
  • Bandit 30786852465
  • Security Scan 30786852447
  • Semgrep 30786852459
  • Dependency Review 30786852443
  • Docker validation 30786852577
• naruon#1219 head 807967e61fc6d0c263eb5693e1a6fd64dce465b1
  • all six corresponding repository-local workflows remain queued
• naruon#1215 head 712a76feb477fbda8cd9b20de8ec6078450766bb
  • Application CI 30789099991, with both backend and frontend jobs queued

GitHub's public status reports Actions operational, so the exact cause is not established. Plausible administrative causes include organization/repository concurrency exhaustion, spending or minutes limits, hosted-runner policy, or a runner-group capacity restriction. Do not treat any one cause as confirmed until the organization Actions settings and billing/audit logs are inspected.

Required investigation

1. Inspect organization and repository Actions usage, concurrent-job limits, spending/minutes limits, runner-group access, and policy denials.
2. Confirm whether public-repository hosted runners are eligible for these workflows and whether any enterprise policy is holding jobs before assignment.
3. Check whether the large PR fan-out (six repository-local workflows plus central required workflows per head) should be serialized or consolidated without weakening required security/review contexts.
4. Restore execution for the listed immutable heads, then verify the central 15-minute scheduler can publish independent reviews and merge them without manual bypass.
5. Record the root cause and an operational guardrail/alert so a release-ready PR cannot remain queued without a concrete reason.

Safety boundary

Do not remove required checks, bypass repository rulesets, synthesize successful statuses, or weaken OpenCode/Noema/Strix/security evidence. This issue concerns runner assignment and queue capacity only.

[seonghobae]

Concrete evidence from driving semantic-data-portal's PR queue on 2026-08-03/04 (UTC), adding a second failure dimension beyond queue saturation — the OpenCode review model pool itself is exhausting, so even dispatches that do run fail closed:

Timeline observed

• ~12:49–15:50 and ~15:52–20:22: org Actions event processing fully stalled (no cron sweeps, no pull_request runs from pushes; brief processing bursts at ~15:50 and ~20:22).
• Review dispatches that did execute failed with model-pool exhaustion, not queue issues:
  • Run 30798884116 (sdp#34): failed at "Publish OpenCode review outcome" after ~2h.
  • Run 30855431448 (sdp#348130be13, ran 21:37–22:35): OpenCode model-pool outcome=exhausted model=none → MODEL_OUTPUT_UNAVAILABLE fail-closed. Contributing causes seen in sibling runs: nvidia-nim 429 Too Many Requests and github-models 410 github_models_retirement_brownout.
• Secondary bug (matches OpenCode: expose cross-repository review failure when App status publication is forbidden #691): the failure-status publisher then gets gh: Resource not accessible by integration (HTTP 403) on POST /repos/{target}/statuses/{sha} and on code-scanning alert reads, so the review gap is not surfaced as a commit status either.
• One review did land in a capacity window (sdp#34 review 4847823477, APPROVE at 19:25) but was dismissed after the head/evidence rotation, so intermittent capacity is not converging to merges.

Impact: semantic-data-portal has its whole queue (16 open PRs) green-checked with auto-merge armed; nothing can merge because current-head OpenCode approvals cannot be produced reliably. Relates to #707 (model pool needs an operational fallback with real capacity) and #691 (403 on cross-repo status publish).

---

Generated by Claude Code

[seonghobae]

Additional exact-head evidence on 2026-08-04 UTC:

• .github#728 head 8d698cf5bbf68e31ecaeb76f1b9044f7361ba5dd: OSV, Semgrep, CodeQL, Security Scan, Scorecard, Secret Scan, Python Security, and SBOM runs are all queued. Direct merge reports 15 required checks not succeeded plus independent approval required.
• .github#727 head 2133b5e1725124e3f9b233b59559f5a3be5d5564: the same eight hosted workflows are queued.
• newsdom-api#467 head 4ba5e81d8dba98fc97ba2f6964751cc958b58c9e: all nine repository-owned checks succeeded, but the central OpenCode coverage evidence failed while provisioning the protected base lock (pypdf==6.13.3 unavailable on a reachable index), creating a self-healing deadlock. Focused repairs are fix(ci): keep CodeQL Action versions consistent #728 (existing CodeQL contract failure on central main) followed by fix(ci): defer stale base-lock pins when the index is reachable #727 (defer only reachable-index stale pins).

This reinforces that the queue incident affects new central control-plane repairs themselves, so runner/billing/policy inspection remains an external administrative blocker. Required checks and approval rules have not been bypassed.

[seonghobae]

Additional exact-head evidence from ContextualWisdomLab/scopeweave on 2026-08-04 UTC:

• scopeweave#414 head 7cf119fc4a4aef7d488e92cd51d7298a9caf0b25
  • SAST Semgrep 30883090160
  • Fuzz 30883090126
  • Security Scan 30883090138
  • Server Tests 30883090194
  • Dependency Review 30883090206
  • OSV Scanner 30883090560
  • all remained queued after a focused JSDoc-only current-head update
• scopeweave#420 head 63a8f18b305b33b7a8c1b12584aa28aff448990b
  • one-shot focused repair 30883729755 / job 91910311152
  • SAST Semgrep 30883719952
  • OSV 30883720266
  • Server Tests 30883720021
  • Dependency Review 30883719935
  • Security Scan 30883719954
  • Fuzz 30883719996
  • every job remained queued; PR was returned to Draft so partial code cannot merge
• scopeweave#419 head 99dc57b383b0ed599230c474c6fd7d8d495de677 has all six repository-local workflows and CodeRabbit green, a valid non-author approval, and auto-merge enabled, but direct merge reports 2 of 5 required status checks are expected; ready-for-review/auto-merge events and a Dependabot recreate request have not yet produced the missing central review contexts.

GitHub's public status page currently reports all systems, including Actions, operational and no incident on August 4. This strengthens the hypothesis that the blocker is organization/repository capacity, spending, runner policy, or queue fan-out rather than a declared GitHub-wide incident, but it does not establish which cause. No required check or protection has been bypassed.