🎯 Waza Platform Roadmap - Tracking Issue

[spboyer]

Migrated from spboyer/waza#66

Waza Skills Development Platform

This is the tracking issue for the waza platform implementation based on the PRD and Squad Proposal.

Primary Phase (Core Features)

E1: Go CLI Foundation (P0)

• feat: Add --format flag to waza check command #24 - waza run command
• test: Add JSON output tests for waza check #25 - waza init command
• feat: Implement JSON output for waza check #26 - waza generate command
• fix: Standardize emoji spacing in waza check display #27 - waza compare command
• feat: Define JSON output structs for waza check #28 - All 8 grader types
• --discover doesn't find eval.yaml in evals/ subdirectory (inconsistent with waza init/new) #29 - Copilot SDK executor
• waza check rejects SKILL.md file path — only accepts directory #30 - Verbose mode
• waza suggest deadlocks — goroutine panic on copilot SDK stdio transport #31 - Transcript logging
• feat: Registry backend evaluation (Git/OCI/Releases/federated) #16 - JSON-RPC server for IDE integration
• Integration tests and documentation for .waza.yaml consolidation #21 - Session event logging and viewer

E2: Sensei Engine (P0)

• v0.11.0 release binary is stale — reports 'waza version 0.9.0', missing v0.10.0 fixes #32 - waza dev command (Sensei loop)
• feat: Migrate Squad from .ai-team/ to .squad/ (v0.5.3) #33 - Compliance scoring system
• chore(deps): Bump rollup from 4.57.1 to 4.59.0 in /web #34 - Improvement suggestions engine
• chore: bump version to 0.12.0 #35 - Target score option
• chore: Release v0.12.0 — registry and version sync #36 - Trigger accuracy tests
• fix: make release workflow resilient to enterprise token restrictions #37 - --skip-integration flag
• fix: make release workflow resilient to enterprise token restrictions #38 - --fast flag

E3: Evaluation Framework (P0)

• Images broken in dashboard page on gh-pages #39 - Multiple model execution
• fix: broken images on gh-pages due to missing base path configuration #40 - Task completion metrics
• Consolidating the keyword and regex graders into a single text grader. #41 - Trigger accuracy metrics
• azd waza run examples/code-explainer/eval.yaml fail with error #42 - Behavior quality metrics
• fix: regression test + changelog for waza suggest deadlock #43 - Trials for statistical confidence
• fix: --discover finds eval.yaml in project-root evals/{name}/ layout #44 - LLM-powered suggestions
• fix: Standardize emoji spacing in waza check display #45 - Parallel task execution
• fix: resolve dashboard image paths using Astro base URL #46 - Task filtering

E4: Token Management (P1)

• Copilot SDK usage display for waza run #47 - waza tokens count
• Review init code, make sure we always prompt if data needs to be overwritten #48 - waza tokens check
• images on docs are still broken #49 - --strict mode
• Fix broken dashboard-explore doc images across deployment bases #50 - waza tokens suggest
• Fix Docker build #51 - waza tokens compare
• feat: Add trigger heuristic grader #80 - BPE token counter

E5: Waza Skill (P1)

• Waza doesn't understand skills under the .github directory #52 - SKILL.md for microsoft/skills
• Working around an issue in the copilot SDK, Start() and contexts #53 - Guided requirements gathering
• Migrate all uses of copilot.Client to use execution's CopilotEngine instead #54 - Conversational readiness check
• Migrate prompt grader from raw copilot.Client to CopilotEngine #55 - Result interpretation
• fix: make site base path configurable + remove unused workflow #56 - CLI command invocation

Secondary Phase (Integration & Extensions)

E6: CI/CD Integration (P1)

• Fix config.schema.json mismatches + add parity test #57 - GitHub Actions workflow template
• Refactor waza new to use shared FileWriter #58 - CI exit codes
• Invert token limits priority: .waza.yaml first, .token-limits.json as legacy fallback #59 - GitHub PR comment reporter
• fix: discover skills under .github/skills/ directory #60 - microsoft/skills CI compatibility
• feat: add FileWriter for safe scaffold file creation #61 - Evaluation result caching

E7: AZD Extension (P2)

• chore: add MIT LICENSE file #62 - Package as AZD extension
• feat: add FileWriter service and refactor waza init inventory #48 #63 - azd waza commands
• feat: invert token limits priority to .waza.yaml first #59 #64 - IntelliSense metadata
• fix: align config.schema.json defaults with Go source of truth #57 #65 - azure.yaml integration (closed — shipped via extension.yaml)

E8: Getting Started Experience (P1)

• [E1] Decouple ExecutionResponse from Copilot SDK + Multi-Agent Engine Support #10 - Getting Started Experience umbrella
• #168 - Getting started documentation
• #169 - Redesign waza init
• #170 - waza new skill scaffolding
• #171 - Retrofit CLI commands for workspace awareness
• #172 - internal/workspace package

---

E3: Evaluation Framework — Azure ML Evaluator Integration

• #104 - Implement prompt (LLM-as-judge) grader
• #105 - Implement action_sequence grader
• #106 - Port Azure ML tool_call evaluation rubrics
• #107 - Port Azure ML task evaluation rubrics
• #108 - Create example eval YAMLs using new graders
• #109 - Document prompt and action_sequence grader types
• #138 - Multi-model evaluation with recommendation engine

E3: Multi-Skill Evaluation Support

• #142 - Wire skill_directories from eval YAML to Copilot SDK
• #143 - Add required_skills preflight validation
• #144 - Add skill_invocation grader for asserting dependent skill usage

E3: A/B Skill Impact Measurement

• #194 - --baseline flag for A/B skill impact comparison

E3: Extended Evaluation Features

• feat: add waza quality command — LLM-as-Judge skill quality scoring #98 - Behavior grader
• chore: add MIT LICENSE file #99 - Diff grader for workspace changes
• #184 - Retry/attempts mechanism
• #185 - Lifecycle hooks
• #186 - Template variable support
• #187 - CSV dataset support
• #188 - Result groupBy/categorization
• #189 - Custom inputs

E9: Competitive Positioning (P1)

• #195 - Multi-agent engine support (assigned: richardpark-msft)

E10: Web UI (P2)

• feat: Map OpenAI Evals YAML format → waza graders #14 - Web UI + Dashboard (competitive analysis)
• #201 - Scaffold React 19 + Vite + Tailwind CSS v4 (PR #212)
• #202 - Dashboard shell layout with DevEx-style dark theme (PR #215)
• #203 - HTTP web server for waza serve (PR #211)
• #204 - Phase 1 REST API endpoints (PR #210)
• #205 - KPI summary cards component (PR #214)
• #206 - Recent Runs sortable table (PR #216)
• #207 - Run Detail drill-down view (PR #217)
• #208 - Playwright E2E test infrastructure (PR #218)

v0.8.0: Advanced Features & MCP Integration (Shipped)

E11: MCP Server Integration (P0)

• #286 - Always-on waza serve with MCP transport
• #316 - MCP scoring validators integration
• #289 - 10 MCP tools (run, init, generate, compare, dev, tokens, serve, new, init-task, help)

E12: LLM-Powered Intelligence (P0)

• #287 - waza suggest command for AI-powered eval recommendations
• #309 - --judge-model flag for separate judge LLM configuration

E13: Advanced Skill Development (P1)

• #288 - Interactive skill for workflow orchestration
• #319 - Auto-generate trigger tests from skill triggers
• #311 - Skill profile with static token analysis

E14: Evaluation Enhancements (P1)

• #299 - Grader weighting for weighted composite scores
• #308 - Statistical confidence intervals via bootstrap
• #317 - Batch processing with waza dev multi-skill support
• #318 - Token budget enforcement with strict comparison mode

E15: Compliance & Validation (P2)

• #314 - agentskills.io specification compliance checks
• #315 - SkillsBench 5 advisory checks
• #312 - JUnit XML reporter for CI pipeline integration

v0.9.0: A/B Testing, Discovery & Competitive Features

E16: A/B Testing & Comparative Evaluation (P0)

• #307 - A/B baseline testing (--baseline flag)
• #310 - Pairwise LLM judging with bias mitigation
• #391 - Tool constraint assertions (expect_tools / reject_tools)
• #392 - Auto skill discovery (--discover flag)

E17: Documentation & Site (P1)

• #383 - Releases page on GitHub Pages site
• #381 - Convert ASCII diagrams to Mermaid

E18: Eval & Grader Registry (P2) — Design Complete

• #385 - Eval & Grader Registry design doc (parent epic)
• #386 - Map OpenAI Evals YAML format to waza graders
• #387 - Go-module-style grader/eval references
• #388 - Registry backend evaluation
• #389 - Composable eval construction
• #390 - Grader plugin extensibility

Summary

Epic	Total	Done	Open	Priority
E1: Go CLI Foundation	10	9	1	P0
E2: Sensei Engine	7	5	2	P0
E3: Evaluation Framework	23	23	0	P0
E4: Token Management	6	6	0	P1
E5: Waza Skill	5	5	0	P1
E6: CI/CD Integration	5	5	0	P1
E7: AZD Extension	4	4	0	P2
E8: Getting Started	6	6	0	P1
E9: Competitive Positioning	1	0	1	P1
E10: Web UI	9	9	0	P2
E11: MCP Server	3	3	0	P0
E12: LLM Intelligence	2	2	0	P0
E13: Skill Development	3	3	0	P1
E14: Evaluation Enhancements	4	4	0	P1
E15: Compliance & Validation	3	3	0	P2
E16: A/B Testing	4	4	0	P0
E17: Documentation & Site	2	2	0	P1
E18: Eval Registry	6	0	6	P2
Total	103	93	10

Related Documents

• PRD
• Squad Proposal
• Web UI Competitive Analysis
• SkillsBench Competitive Analysis
• Positioning Strategy

[spboyer]

Originally by @spboyer on 2026-02-06:

Recent Progress (2026-02-06)

Infrastructure Complete

• ✅ Removed Python CI - Go only going forward
• ✅ Branch protection configured (Go CI must pass)
• ✅ Auto-merge enabled
• ✅ AGENTS.md updated for Go-first development
• ✅ PRD published to docs/PRD.md

Ready to Start Development

The foundation is set. Priority work:

1. E1: Go CLI Foundation - Port remaining Python features to Go
2. E2: Sensei Engine - Implement compliance scoring

See individual issues for details.

[spboyer]

Originally by @spboyer on 2026-02-10:

Team Update

👤 Wallace Breza (@wbreza) has joined the team as a Go Engineer. Welcome Wallace! 🎉

He's available to pick up unblocked work. Current candidates once blockers clear:

• Images broken in dashboard page on gh-pages #39 (multi-model execution) — blocked on --discover doesn't find eval.yaml in evals/ subdirectory (inconsistent with waza init/new) #29
• fix: --discover finds eval.yaml in project-root evals/{name}/ layout #44 (LLM suggestions) — blocked on Images broken in dashboard page on gh-pages #39
• Migrate all uses of copilot.Client to use execution's CopilotEngine instead #54 (readiness check) — blocked on feat: Migrate Squad from .ai-team/ to .squad/ (v0.5.3) #33

@richardpark-msft @chlowell — Wallace can also help with any grader or token work if you'd like to split tasks.

[spboyer]

Originally by @spboyer on 2026-02-10:

Update — Feb 10

• ✅ Review init code, make sure we always prompt if data needs to be overwritten #48 waza tokens check — merged (PR #102 by @chlowell)
• 🔧 feat: add waza quality command — LLM-as-Judge skill quality scoring #98 behavior grader — PR docs: Add CI/CD integration guide (GitHub Actions, Azure DevOps) #100 open, CI fix pushed
• 📬 #103 AZD extension files — PR open by @wbreza (chore: add MIT LICENSE file #62)
• 📬 chore: add MIT LICENSE file #99 diff grader — issue created, assigned to @richardpark-msft

[spboyer]

Originally by @spboyer on 2026-02-13:

📋 Tracking Update — 2026-02-13

Updated 10 stale checkboxes to reflect completed work:

E2: Sensei Engine

• ✅ v0.11.0 release binary is stale — reports 'waza version 0.9.0', missing v0.10.0 fixes #32 — waza dev command (Charles, PR #117)
• ✅ feat: Migrate Squad from .ai-team/ to .squad/ (v0.5.3) #33 — Compliance scoring system (Charles)
• ✅ chore: bump version to 0.12.0 #35 — Target score option (Charles)

E4: Token Management — now fully complete!

• ✅ Review init code, make sure we always prompt if data needs to be overwritten #48 — waza tokens check (Charles)
• ✅ images on docs are still broken #49 — --strict mode (Charles)
• ✅ Fix broken dashboard-explore doc images across deployment bases #50 — waza tokens suggest (Charles, PR #114)
• ✅ Fix Docker build #51 — waza tokens compare (Charles, PR #111)

E6: CI/CD Integration

• ✅ Fix config.schema.json mismatches + add parity test #57 — GitHub Actions workflow template (@copilot, PR #132)

E3: Azure ML Evaluator Integration

• ✅ #105 — action_sequence grader (Squad, PR #110)
• ✅ #109 — Grader docs (@copilot, PR #133)

@chlowell @richardpark-msft — please keep this tracking issue updated when you close issues. Check the box for your issue when the PR merges. 🙏

Currently in progress:

• PR #134 — Example eval YAMLs (#108, @copilot working)
• PR #135 — CI exit codes (Refactor waza new to use shared FileWriter #58, @copilot working)

[spboyer]

Originally by @wbreza on 2026-02-17:

E8: Getting Started Experience (P1)

• [E1] Decouple ExecutionResponse from Copilot SDK + Multi-Agent Engine Support #10 — E8 Epic (Getting Started Experience)
• #169 — E8-01: Redesign waza init as project scaffolder
• #170 — E8-02: Implement waza new skill scaffolding command
• #172 — E8-03: Implement internal/workspace skill detection package
• #171 — E8-04: Retrofit CLI commands for workspace awareness
• #168 — E8-05: Getting started documentation

Dependency graph:
WI-1 (design/plan) → E8-01 + E8-02 + E8-03 (parallel) → E8-04 (depends on E8-03) → E8-05 (docs)
E8-01 depends on E8-02 being available (init calls new internally)

[spboyer]

Originally by @spboyer on 2026-02-19:

Web UI Status Update

E10: Web UI — 8/9 items complete (PR #232 merged all Phases 1-3):

• ✅ #201 - Scaffold React 19 + Vite + Tailwind CSS v4
• ✅ #202 - Dashboard shell layout with DevEx-style dark theme
• ✅ #203 - HTTP web server for waza serve
• ✅ #204 - Phase 1 REST API endpoints
• ✅ #205 - KPI summary cards component
• ✅ #206 - Recent Runs sortable table
• ✅ #207 - Run Detail drill-down view
• ⬜ #208 - Playwright E2E test infrastructure (remaining)

Additional work completed beyond original scope:

• CompareView (side-by-side run comparison with deltas)
• TrendChart + TrendsPage (SVG trend charts with model filter)
• LiveView with SSE streaming and TrajectoryViewer
• CSV export with formula injection protection
• API routes wired to real FileStore (was placeholder stubs)
• Bug fixes: outcome badge matching, pass rate display, ResizeObserver cleanup

[spboyer]

Originally by @spboyer on 2026-02-19:

📋 New Epic: E11 — Trajectory Replay Viewer

Added as a result of SkillsBench/Harbor/MSBench competitive research. This was identified as the biggest competitive gap — a full step-by-step replay viewer with raw tool calls, arguments, and outputs.

Key insight: This is a UI+API problem, not an engine problem. The data model (TranscriptEvent, SessionDigest) is already complete.

Issues Created

Issue	Title	Size	Assigned	Priority	Depends
#237	Expose transcript & session digest in web API	S	Linus	P1	—
#238	True trajectory replay viewer component	M	Rusty	P1	#237
#239	Trajectory diffing — compare runs step-by-step	L	Linus + Rusty	P2	#237
#240	E2E tests for trajectory features	S	Basher	P2	#238, #239

Sequencing

1. Clear #208 (Playwright E2E infra) first
2. TRV-1 (#237) — API plumbing — unblocks everything
3. TRV-2 (#238) + TRV-3 (#239) — can run in parallel once API ships
4. TRV-4 (#240) — tests after features land

Total estimate: ~9-15 days across 3 layers.