Add azd-preflight Copilot skill to auto-run and fix preflight checks#7566
Merged
jongio merged 4 commits intoAzure:mainfrom Apr 8, 2026
Merged
Add azd-preflight Copilot skill to auto-run and fix preflight checks#7566jongio merged 4 commits intoAzure:mainfrom
jongio merged 4 commits intoAzure:mainfrom
Conversation
jongio
requested review from
JeffreyCA,
danieljurek,
hemarina,
rajeshkamal5050,
tg-msft,
vhvb1989,
wbreza and
weikanglim
as code owners
Contributor
There was a problem hiding this comment.
Pull request overview
Adds GitHub Copilot workflow “skills” intended to automate cli/azd preflight checks (/azd-preflight) and to evaluate/improve skills via waza (/sensei), plus a CI workflow to validate and run skill eval suites on PRs touching .github/skills/.
Changes:
- Introduces two new skills:
azd-preflight(run/fix preflight) andsensei(waza-based skill quality evaluator). - Adds a waza eval suite for
azd-preflightand a new CI workflow to runwaza check/waza run. - Updates contributor docs to mention the new
/azd-preflightskill.
Reviewed changes
Copilot reviewed 14 out of 14 changed files in this pull request and generated 11 comments.
Show a summary per file
| File | Description |
|---|---|
cli/azd/CONTRIBUTING.md |
Adds a tip pointing contributors to /azd-preflight. |
cli/azd/AGENTS.md |
Adds a tip pointing agents/contributors to the new skill. |
.github/workflows/skill-eval.yml |
Adds CI to validate skills and run their eval suites. |
.github/skills/sensei/SKILL.md |
Defines the new sensei skill. |
.github/skills/sensei/references/workflow.md |
Documents sensei workflow steps for waza-based checks/evals. |
.github/skills/azd-preflight/SKILL.md |
Defines the new azd-preflight skill and references its workflow/docs. |
.github/skills/azd-preflight/references/workflow.md |
Documents how the skill should run and iterate preflight. |
.github/skills/azd-preflight/references/preflight-checks.md |
Documents the checks and prerequisites the skill should handle. |
.github/skills/azd-preflight/references/output-and-errors.md |
Provides output examples and error handling guidance. |
.github/skills/azd-preflight/references/fix-strategies.md |
Provides fix strategies per check. |
.github/skills/azd-preflight/evals/eval.yaml |
Adds an eval suite configuration for waza. |
.github/skills/azd-preflight/evals/tasks/*.yaml |
Adds happy-path / edge-case / anti-trigger eval tasks. |
Member
Author
|
@copilot please re-review the latest changes. |
jongio
force-pushed
the
feature/azd-preflight-skill
branch
from
97617e2 to
6167874
Compare
wbreza
requested changes
Apr 8, 2026
Contributor
wbreza
left a comment
wbreza
left a comment
There was a problem hiding this comment.
Code Review Summary
| Priority | Count |
|---|---|
| 🔴 Critical | 1 |
| 🟠 High | 3 |
| 🟡 Medium | 4 |
| 🟢 Low | 8 |
| Total | 16 |
Overall Assessment: Request Changes
The skill definitions (azd-preflight and sensei) are well-structured and solid — good frontmatter, clean reference doc organization, thorough fix strategies with proper ask_user escalation, and smart token management via {{ references/ }} includes.
The eval suite and CI workflow need more iteration before they're ready to merge alongside the skills. The eval runner step is silently bypassed in CI, grader thresholds are permissive, key scenarios (clean runs, fix-then-rerun loops) lack test coverage, and the prompt grader may reference incorrect waza API functions. Consider splitting the eval suite and CI workflow into a follow-up PR so the skill itself can land first, with the evaluation infrastructure refined separately.
✅ What Looks Good
- Well-structured skill spec with proper frontmatter following established repo patterns
- Smart token management using
{{ references/file.md }}includes - Comprehensive reference docs (fix strategies, output examples, preflight checks, workflow)
- Thorough fix-strategies with proper
ask_userescalation for ambiguous fixes - CI permissions correctly declared as
permissions: contents: read(least-privilege) - Good eval task variety: happy-path, edge-case, and anti-trigger categories
🟢 Low-Priority Items (not posted inline)
- Weak edge-case assertions: Single-keyword assertions don't validate actual error handling behavior
- Ambiguous sensei scope: "agent skill quality" should specify repo boundary (e.g.,
.github/skills/) - Anti-trigger tests only check exclusions: No
output_containsfor what responses should include - Over-permissive skill invocation grader:
allow_extra: truereasonable for v1 - Missing dependency caching: Add
cache: truetosetup-go@v5 - Version strategy not documented: No guidance on when to increment
version: "1.0" - Regex graders too broad: Patterns like
(?i)(fix|resolve|...)can match incidentally - Unversioned waza:
@latestis acceptable per team preference; install validation would be a nice-to-have
…ture - Add azd-preflight skill: runs mage preflight and auto-fixes failures across all 8 checks (gofmt, go fix, copyright, lint, cspell, build, unit tests, playback tests) with iterative fix-then-rerun cycles - Add sensei skill: evaluates and improves skill quality using waza - Add waza eval suite for azd-preflight (8 test tasks: 3 happy-path, 2 edge-case, 3 anti-trigger) - Add skill-eval CI workflow for PR validation - Update CONTRIBUTING.md and AGENTS.md with preflight skill references Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
waza run requires the GitHub Copilot CLI to execute eval suites. In CI environments where copilot is not installed, the step now detects this and exits gracefully with a warning instead of failing. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…criptions - Remove eval suite and CI workflow per reviewer recommendation to split evaluation infrastructure into a follow-up PR - Condense SKILL.md descriptions to routing signals (not impl details) - Clarify sensei scope to .github/skills/ boundary - Add version strategy comments to both skills Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
jongio
force-pushed
the
feature/azd-preflight-skill
branch
from
23aa1bb to
159f9b8
Compare
Member
Author
|
Thanks for the thorough review @wbreza. Addressed all feedback:
All 8 inline threads resolved. |
Collaborator
Azure Dev CLI Install InstructionsInstall scriptsMacOS/Linux
bash: pwsh: WindowsPowerShell install MSI install Standalone Binary
MSI
Documentationlearn.microsoft.com documentationtitle: Azure Developer CLI reference
|
wbreza
approved these changes
Apr 8, 2026
Contributor
wbreza
left a comment
wbreza
left a comment
There was a problem hiding this comment.
Code Re-Review — PR #7566 (Round 2)
Prior Review Regression Check — All 16 Findings Resolved
| Resolution | Count | Details |
|---|---|---|
| Removed (deferred to follow-up PR) | 10 | Eval suite, CI workflow, and all related findings |
| Fixed in updated code | 3 | SKILL.md description condensed, sensei scope clarified, version strategy added |
| Already resolved (Round 1) | 3 | 8-check count, repo-root detection, copyright paths |
The eval suite and CI workflow were cleanly split into a follow-up PR per review feedback. The remaining skill definitions address the 3 fixable findings: description condensed to routing signals, sensei scope explicitly bounded to .github/skills/, and version bump guidance added.
New Findings Summary
| Priority | Count |
|---|---|
| Low | 1 |
| Total | 1 |
See inline comment for details.
✅ What Looks Good
- Clean scope reduction: eval suite and CI deferred to follow-up PR — exactly as suggested
- SKILL.md description condensed to routing signals, not implementation details
- Sensei scope explicitly bounded to
.github/skills/ - Version bump guidance added as metadata comments
- PR description updated with clear "Not Included" section explaining what was removed and why
- Reference docs are thorough and well-organized across 4 files
- Smart token management via
{{ references/ }}includes
Overall Assessment: Approve — all 16 prior findings resolved, PR scope cleanly reduced. Nicely done!
Review performed with GitHub Copilot CLI
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add two new Copilot skills to the repository:
/azd-preflightfor running and auto-fixing preflight checks, and/senseifor evaluating skill quality with waza.Fixes #7565
What's Included
.github/skills/azd-preflight/.github/skills/sensei/.github/skills/using wazacli/azd/CONTRIBUTING.md,cli/azd/AGENTS.md/azd-preflightskillNot Included (follow-up PR)
The eval suite (waza test tasks) and CI workflow (
skill-eval.yml) were removed from this PR per review feedback. The evaluation infrastructure will be refined and submitted separately.How It Works
/azd-preflightin Copilotcd cli/azd && mage preflight