You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The Dependabot AI risk gate (RealRate-Private#2229, implemented here in #9) is not working as expected: on every major bump its AI verdict step fails and it falls back to verdict=human. Root cause is that the GitHub Models inference call returns HTTP 403 when made with the org's Actions GITHUB_TOKEN. The gate fails safe (routes to manual review, never mis-merges), but the "auto-approve low-risk majors" capability is effectively dormant. First observed on the first real major bump: realrate/EBA#101 (numpy 1.26.4 → 2.5.0).
How we got here
2026-07-13 — Dependabot auto-merge: AI risk gate for every major bump + CI gate on every merge #9 merged (44747d8): AI risk gate for every major bump + CI gate on every merge. Inference uses GitHub Models (openai/gpt-4.1) via the built-in GITHUB_TOKEN + models: read, no external key. Rollout validated patch/minor auto-merge, but the AI (major) path was never exercised on a live runner — noted at the time as the one residual risk.
2026-07-15 — realrate/EBA#101 (numpy, a semver-major bump) became the first PR to actually invoke the AI path. The gate posted:
🤖 AI risk gate: human. AI verdict unavailable (PermissionDeniedError); manual review. Left for manual review.
Causal chain
GitHub Models NOT enabled for the org's Actions token (policy)
│
▼
inference POST https://models.github.ai/inference → HTTP 403 (openai.PermissionDeniedError)
│
▼
review_dependabot_major.py catches it → _fallback(verdict="human", reason="AI verdict unavailable (PermissionDeniedError)")
│
▼
workflow "Route major bump to human" step → needs-manual-review label + the comment above
│
▼
every major bump silently bypasses the AI auto-approve path (safe, but the feature does nothing)
Why the diagnosis holds
The workflow is correct: it triggers on pull_request_target (full token, NOT Dependabot's read-only token — the same token already approves + merges patch/minor bumps org-wide), and it declares models: read.
Calling the identical endpoint + model + payload with a personal token returns HTTP 200 (verified 2026-07-15). So the endpoint, model id (openai/gpt-4.1), and script are all fine.
The org's Actions GITHUB_TOKEN gets 403 for the same call. A 403 (not 401, not a 404 unavailable_model) on both the structured and plain attempts ⇒ the identity is not permitted to use GitHub Models ⇒ org-level Models policy is off for Actions, not a code bug.
RealRate-Private#2229 — AI gate delivery/design; RealRate-Private#2244 — CI test gate.
realrate/EBA#101 — evidence (first major bump; the failing bot comment).
Proposed fix
Part A — unblock (org owner, external to this repo; NOT a code change).
Enable GitHub Models for the realrate org and allow GitHub Actions / GITHUB_TOKEN to use it (Org → Settings → Models; accept terms if prompted). Requested from Holger (org owner) via Slack. This is the actual unblock — nothing here merges the feature into a working state until it's on.
Part B — implementation hardening (this repo).
Actionable failure message: in review_dependabot_major.py, distinguish a 403 from a generic error so the routed-to-human comment says something like "GitHub Models is not enabled for this org — an owner must enable it" instead of the cryptic (PermissionDeniedError). Saves the next person the whole diagnosis.
End-to-end verification once Part A lands: re-trigger the gate on a real major bump (close+reopen, or the next one) and confirm the comment becomes a genuine auto/human verdict with a risk reason.
Optional self-check: a lightweight one-off/manual workflow that pings Models with the Actions token and reports status + body, so a future config regression (Models turned off, model retired) is caught proactively rather than on the next major bump.
Acceptance criteria
Part A confirmed: GitHub Models enabled for the org's Actions token (owner action).
AI verdict succeeds end-to-end on a live major bump (real auto/human verdict, no 403).
Fallback comment names the cause on a 403 (no bare PermissionDeniedError).
(Optional) self-check/diagnostic in place to catch Models-availability regressions.
Key references for the implementer
.github/workflows/dependabot-automerge.yaml — trigger pull_request_target (~L26), models: read (~L41), "AI risk verdict for major bump" step (~L115–142), "Route major bump to human" step (~L160–175).
Summary
The Dependabot AI risk gate (RealRate-Private#2229, implemented here in #9) is not working as expected: on every major bump its AI verdict step fails and it falls back to
verdict=human. Root cause is that the GitHub Models inference call returns HTTP 403 when made with the org's ActionsGITHUB_TOKEN. The gate fails safe (routes to manual review, never mis-merges), but the "auto-approve low-risk majors" capability is effectively dormant. First observed on the first real major bump: realrate/EBA#101 (numpy 1.26.4 → 2.5.0).How we got here
44747d8): AI risk gate for every major bump + CI gate on every merge. Inference uses GitHub Models (openai/gpt-4.1) via the built-inGITHUB_TOKEN+models: read, no external key. Rollout validated patch/minor auto-merge, but the AI (major) path was never exercised on a live runner — noted at the time as the one residual risk.semver-majorbump) became the first PR to actually invoke the AI path. The gate posted:Causal chain
Why the diagnosis holds
pull_request_target(full token, NOT Dependabot's read-only token — the same token already approves + merges patch/minor bumps org-wide), and it declaresmodels: read.openai/gpt-4.1), and script are all fine.GITHUB_TOKENgets 403 for the same call. A 403 (not 401, not a 404unavailable_model) on both the structured and plain attempts ⇒ the identity is not permitted to use GitHub Models ⇒ org-level Models policy is off for Actions, not a code bug.Related
44747d8) — the implementation being fixed.Proposed fix
Part A — unblock (org owner, external to this repo; NOT a code change).
Enable GitHub Models for the
realrateorg and allow GitHub Actions /GITHUB_TOKENto use it (Org → Settings → Models; accept terms if prompted). Requested from Holger (org owner) via Slack. This is the actual unblock — nothing here merges the feature into a working state until it's on.Part B — implementation hardening (this repo).
review_dependabot_major.py, distinguish a 403 from a generic error so the routed-to-human comment says something like "GitHub Models is not enabled for this org — an owner must enable it" instead of the cryptic(PermissionDeniedError). Saves the next person the whole diagnosis.auto/humanverdict with a risk reason.Acceptance criteria
auto/humanverdict, no 403).PermissionDeniedError).Key references for the implementer
.github/workflows/dependabot-automerge.yaml— triggerpull_request_target(~L26),models: read(~L41), "AI risk verdict for major bump" step (~L115–142), "Route major bump to human" step (~L160–175)..github/review-tooling/review_dependabot_major.py—BASE_URL(~L39),OpenAI(base_url, api_key=TOKEN)(~L135), retry +_fallback("AI verdict unavailable (...)")(~L139–147),get_verdict()(~L116).4976796622).https://models.github.ai/inference/chat/completionswithopenai/gpt-4.1→ 200.