Detect and surface excessive Copilot tool-denial guardrail failures#37363
Merged
Conversation
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Copilot
AI
changed the title
[WIP] Fix daily SPDD Spec Planner workflow failure
Detect and surface excessive Copilot tool-denial guardrail failures
Jun 6, 2026
Contributor
There was a problem hiding this comment.
Pull request overview
This PR adds an explicit, first-class signal for Copilot SDK “max tool denial” guardrail stops and surfaces that condition clearly in generated failure issues/comments, so repeated tool permission denials aren’t misclassified as generic engine failures.
Changes:
- Emit a dedicated
guard.tool_denials_exceededJSONL event from the Copilot SDK driver when the denial threshold is reached. - In the conclusion handling, detect that event from Copilot session
events.jsonl, classify it astool_denials_exceeded, and render a dedicated failure context section. - Add tests covering event emission, loading, and rendering of the new context.
Show a summary per file
| File | Description |
|---|---|
| actions/setup/md/tool_denials_exceeded_context.md | New user-facing markdown fragment explaining the guardrail and remediation. |
| actions/setup/md/agent_failure_issue.md | Adds {tool_denials_exceeded_context} into the failure issue template. |
| actions/setup/md/agent_failure_comment.md | Adds {tool_denials_exceeded_context} into the failure comment template. |
| actions/setup/js/handle_agent_failure.test.cjs | Adds tests for loading the guardrail event and rendering its context. |
| actions/setup/js/handle_agent_failure.cjs | Implements event loading, failure categorization, and context rendering for tool-denials-exceeded. |
| actions/setup/js/copilot_sdk_driver.test.cjs | Extends test to assert the driver emits the guardrail JSONL event. |
| actions/setup/js/copilot_sdk_driver.cjs | Emits guard.tool_denials_exceeded when the SDK denial threshold is hit. |
Copilot's findings
Tip
Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- Files reviewed: 7/7 changed files
- Comments generated: 2
Comment on lines
+1128
to
+1143
| const latestEvent = events[events.length - 1]; | ||
| const denialCount = String(latestEvent.denialCount); | ||
| const threshold = String(latestEvent.threshold); | ||
| const reason = latestEvent.reason || "permission denied by workflow tool permissions"; | ||
|
|
||
| try { | ||
| const templatePath = getPromptPath("tool_denials_exceeded_context.md"); | ||
| const template = fs.readFileSync(templatePath, "utf8"); | ||
| return ( | ||
| "\n" + | ||
| renderTemplate(template, { | ||
| denial_count: denialCount, | ||
| threshold, | ||
| reason: `\`${reason}\``, | ||
| workflow_id: workflowId || "the workflow", | ||
| }) |
Comment on lines
+1147
to
+1149
| `\n**⚠️ Excessive Tool Denials**: The Copilot SDK stopped the session after ${denialCount}/${threshold} permission denials.\n\n` + | ||
| `**Last denied request:** \`${reason}\`\n\n` + | ||
| "This is a guardrail stop (`guard.tool_denials_exceeded`) and indicates the workflow's allowed tool set does not match the prompt's requested actions.\n" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The Daily SPDD Spec Planner failure was caused by repeated workflow tool permission denials that tripped the Copilot SDK denial threshold, but this condition was only surfaced as a generic engine failure. This change adds a first-class signal for that guardrail stop and renders it explicitly in failure issues/comments.
events.jsonlwhen max denials are reached:type: "guard.tool_denials_exceeded"denialCount,threshold, andreason.handle_agent_failure.cjsnow reads Copilot sessionevents.jsonland extractsguard.tool_denials_exceeded.tool_denials_exceededto failure-category matching for precise issue reuse/dedup behavior.agent_failure_issue.mdagent_failure_comment.mdtool_denials_exceeded_context.mdwith clear guardrail semantics and actionable remediation.{"type":"guard.tool_denials_exceeded","timestamp":"...","data":{"denialCount":5,"threshold":5,"reason":"permission denied: read"}}