Skip to content

AI risk gate: GitHub Models call returns 403 — major-bump AI verdict never runs #14

Description

@bbkrr

Summary

The Dependabot AI risk gate (RealRate-Private#2229, implemented here in #9) is not working as expected: on every major bump its AI verdict step fails and it falls back to verdict=human. Root cause is that the GitHub Models inference call returns HTTP 403 when made with the org's Actions GITHUB_TOKEN. The gate fails safe (routes to manual review, never mis-merges), but the "auto-approve low-risk majors" capability is effectively dormant. First observed on the first real major bump: realrate/EBA#101 (numpy 1.26.4 → 2.5.0).

How we got here

  • 2026-07-13Dependabot auto-merge: AI risk gate for every major bump + CI gate on every merge #9 merged (44747d8): AI risk gate for every major bump + CI gate on every merge. Inference uses GitHub Models (openai/gpt-4.1) via the built-in GITHUB_TOKEN + models: read, no external key. Rollout validated patch/minor auto-merge, but the AI (major) path was never exercised on a live runner — noted at the time as the one residual risk.
  • 2026-07-15 — realrate/EBA#101 (numpy, a semver-major bump) became the first PR to actually invoke the AI path. The gate posted:

    🤖 AI risk gate: human. AI verdict unavailable (PermissionDeniedError); manual review. Left for manual review.

Causal chain

GitHub Models NOT enabled for the org's Actions token (policy)
        │
        ▼
inference POST https://models.github.ai/inference  →  HTTP 403 (openai.PermissionDeniedError)
        │
        ▼
review_dependabot_major.py catches it → _fallback(verdict="human", reason="AI verdict unavailable (PermissionDeniedError)")
        │
        ▼
workflow "Route major bump to human" step → needs-manual-review label + the comment above
        │
        ▼
every major bump silently bypasses the AI auto-approve path (safe, but the feature does nothing)

Why the diagnosis holds

  • The workflow is correct: it triggers on pull_request_target (full token, NOT Dependabot's read-only token — the same token already approves + merges patch/minor bumps org-wide), and it declares models: read.
  • Calling the identical endpoint + model + payload with a personal token returns HTTP 200 (verified 2026-07-15). So the endpoint, model id (openai/gpt-4.1), and script are all fine.
  • The org's Actions GITHUB_TOKEN gets 403 for the same call. A 403 (not 401, not a 404 unavailable_model) on both the structured and plain attempts ⇒ the identity is not permitted to use GitHub Models ⇒ org-level Models policy is off for Actions, not a code bug.

Related

Proposed fix

Part A — unblock (org owner, external to this repo; NOT a code change).
Enable GitHub Models for the realrate org and allow GitHub Actions / GITHUB_TOKEN to use it (Org → Settings → Models; accept terms if prompted). Requested from Holger (org owner) via Slack. This is the actual unblock — nothing here merges the feature into a working state until it's on.

Part B — implementation hardening (this repo).

  1. Actionable failure message: in review_dependabot_major.py, distinguish a 403 from a generic error so the routed-to-human comment says something like "GitHub Models is not enabled for this org — an owner must enable it" instead of the cryptic (PermissionDeniedError). Saves the next person the whole diagnosis.
  2. End-to-end verification once Part A lands: re-trigger the gate on a real major bump (close+reopen, or the next one) and confirm the comment becomes a genuine auto/human verdict with a risk reason.
  3. Optional self-check: a lightweight one-off/manual workflow that pings Models with the Actions token and reports status + body, so a future config regression (Models turned off, model retired) is caught proactively rather than on the next major bump.

Acceptance criteria

  • Part A confirmed: GitHub Models enabled for the org's Actions token (owner action).
  • AI verdict succeeds end-to-end on a live major bump (real auto/human verdict, no 403).
  • Fallback comment names the cause on a 403 (no bare PermissionDeniedError).
  • (Optional) self-check/diagnostic in place to catch Models-availability regressions.

Key references for the implementer

  • .github/workflows/dependabot-automerge.yaml — trigger pull_request_target (~L26), models: read (~L41), "AI risk verdict for major bump" step (~L115–142), "Route major bump to human" step (~L160–175).
  • .github/review-tooling/review_dependabot_major.pyBASE_URL (~L39), OpenAI(base_url, api_key=TOKEN) (~L135), retry + _fallback("AI verdict unavailable (...)") (~L139–147), get_verdict() (~L116).
  • Failing run/comment: realrate/EBA#101 (bot comment id 4976796622).
  • Verification: personal-token probe of https://models.github.ai/inference/chat/completions with openai/gpt-4.1 → 200.

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions