Require noop call in Multi-Device Docs Tester when no issues found#14553
Require noop call in Multi-Device Docs Tester when no issues found#14553
Conversation
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
Updates the Multi-Device Docs Tester agentic workflow prompt to ensure the run always produces a safe output (avoiding “no safe outputs” failures), especially when no issues are found.
Changes:
- Adds an explicit requirement to call the
nooptool when no issues are detected. - Splits reporting guidance into “no issues” vs “issues found” paths and updates the summary to emphasize safe outputs.
Comments suppressed due to low confidence (1)
.github/workflows/daily-multi-device-docs-tester.md:179
- This instruction is now unconditional (“Create a GitHub issue…”). Given the new “If NO Issues Found” branch, this should remain explicitly conditional (e.g., “If issues are found, create…”) and ideally also state not to create an issue when clean. Otherwise the agent may create a report issue even on successful/no-issue runs, defeating the purpose of the noop path.
Create a GitHub issue titled "🔍 Multi-Device Docs Testing Report - [Date]" with:
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| ```json | ||
| { | ||
| "noop": { | ||
| "message": "Multi-device documentation testing complete. All {device_count} devices tested successfully with no issues found." |
There was a problem hiding this comment.
The noop message uses a {device_count} placeholder that isn’t defined anywhere else in this workflow (the issue template uses {count}) and may end up being logged literally. Prefer using a single placeholder consistently (or instruct the agent to substitute the actual numeric count) so the noop output is always clear and accurate.
| "message": "Multi-device documentation testing complete. All {device_count} devices tested successfully with no issues found." | |
| "message": "Multi-device documentation testing complete. All {count} devices tested successfully with no issues found." |
|
|
||
| ### If Issues ARE Found | ||
|
|
||
| ## 📝 Report Formatting Guidelines |
There was a problem hiding this comment.
The heading structure is now inconsistent: ### If Issues ARE Found is immediately followed by a higher-level ## 📝 Report Formatting Guidelines. This makes the “issues found” branch look like it ends early and can confuse the agent about which instructions are conditional. Consider demoting the guidelines heading (e.g., to ####) or moving it so it’s clearly nested under the “issues found” path.
This issue also appears on line 178 of the same file.
| ## 📝 Report Formatting Guidelines | |
| #### 📝 Report Formatting Guidelines |
Workflow run #21801448336 succeeded but triggered a failure issue because the agent produced no safe outputs. The prompt instructed to "provide summary" and create an issue "if issues are detected" but didn't specify what to do when no issues exist.
Changes
Pattern Alignment
Matches existing workflows that explicitly require noop:
breaking-change-checker.md- "YOU MUST CALL the noop tool"daily-malicious-code-scan.md- noop or alert requireddaily-cli-tools-tester.md- "use noop if all tests pass"The workflow now guarantees
outputs.jsonlis written, preventing "agent succeeded but produced no safe outputs" failures.Original prompt
✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.