Fast-fail Copilot "No model available" policy error instead of exhausting retries - #52953
Conversation
…trying Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
Adds fast-fail handling for Copilot model-policy errors to avoid exhausting retries.
Changes:
- Classifies policy-enablement failures as
model_not_supported. - Adds positive and negative detection tests.
- Documents policy and subagent remediation.
Show a summary per file
| File | Description |
|---|---|
actions/setup/js/copilot_harness.cjs |
Detects policy-related model failures. |
actions/setup/js/copilot_harness.test.cjs |
Tests harness classification. |
actions/setup/js/detect_agent_errors.cjs |
Extends shared error detection. |
actions/setup/js/detect_agent_errors.test.cjs |
Tests matching boundaries. |
actions/setup/md/model_not_supported_error.md |
Adds policy remediation guidance. |
Review details
💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.
- Files reviewed: 5/5 changed files
- Comments generated: 1
- Review effort level: Balanced
| // "No model available" wording alone does not produce false positives. | ||
| // This is a persistent configuration error — retrying with --continue will not help. | ||
| const MODEL_NOT_SUPPORTED_PATTERN = /The requested model is not supported/; | ||
| const MODEL_NOT_SUPPORTED_PATTERN = /The requested model is not supported|No model available\b[^\n]*policy enablement/i; |
|
@copilot This PR can move forward before maintainers investigate. Please refresh the branch and run the
|
|
/matt |
|
🧠 Matt Pocock Skills Reviewer has completed the skills-based review. ✅
|
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
There was a problem hiding this comment.
Skills-Based Review 🧠
Applied /diagnosing-bugs and /tdd — two minor issues found; COMMENT (no blocking changes).
📋 Key Themes & Highlights
Key Themes
- Unintentional
/iflag broadening (copilot_harness.cjs): adding/ito the previously case-sensitive pattern is likely harmless in practice but should be documented as deliberate to prevent future accidental removal. - Bundled test assertions (
copilot_harness.test.cjs): oneitblock tests detection and classification together; splitting improves failure diagnostics.
Positive Highlights
- ✅ Pattern anchored to
policy enablementphrase — avoids false positives on generic "No model available" log noise - ✅ Both files that share the pattern updated together with explicit sync comments
- ✅ Negative tests verify the newline-split variant (
\nbetween sentences) does not match — good edge-case coverage - ✅ Remediation doc updated with subagent-trigger nuance
🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer · sonnet46 · 41.1 AIC · ⌖ 9.11 AIC · ⊞ 7.7K
Comment /matt to run again
|
@copilot Quick triage for maintainer-ready follow-up: Please refresh the branch if GitHub allows it and run the Outstanding review items (newest first):
Failed checks from the compact candidate set:
Branch update was requested automatically for this run when GitHub allows it.
|
|
🎉 This pull request is included in a new release. Release: |
When a Copilot subagent (
tasktool) dispatch requests a model disabled by org/repo Copilot policy, the SDK driver emits a deterministic policy error that the harness misclassified as a generic partial-execution failure — burning all 3 retries (~15 min) per run before failing red.Changes
actions/setup/js/copilot_harness.cjs—MODEL_NOT_SUPPORTED_PATTERNnow matches the SDK policy-enablement error, so the attempt is classifiedmodel_not_supportedand the retry loop stops on the first attempt.actions/setup/js/detect_agent_errors.cjs— same alternative added to its copy of the pattern (the two are documented as needing to stay in sync), so themodel_not_supported_errorjob output is set and the existing conclusion/remediation path fires.actions/setup/md/model_not_supported_error.md— remediation guidance for this variant: the model isn't disabled in frontmatter but by policy, and it can be triggered by a subagent even when the main agent's model is enabled.detectCopilotErrors→classifyCopilotFailure=model_not_supported), plus negatives asserting that bareNo model availablewithout thepolicy enablementhint does not match.Both patterns anchor on the
policy enablementphrase with[^\n]*rather than matchingNo model availablealone, to avoid false positives on transient/informational logging.No
.lock.ymlchanges: workflows reference./actions/setupby path rather than inlining the harness.Scope
This covers item 4 of the issue's remediation list. Items 1–3 require GitHub Settings > Copilot > Policies access to enable the model the subagent requests, so the "3 consecutive clean Linter Miner runs" criterion still depends on that change. The item-3 stopgap (removing subagent usage from Linter Miner) was not applied —
taskis a built-in Copilot capability, not a frontmatter-declared tool inlinter-miner.md, so disabling it would be a behavioural rewrite rather than a surgical fix.Unrelated pre-existing failure
make agent-report-progressfails at the JavaScript lint step withTS2688: Cannot find type definition file for 'node'andTS5108: Option 'moduleResolution=node10' has been removed. This reproduces on a clean tree and is untouched here.