refactor(cli): minimal WAA CLI with vanilla image support

[abrichr]

Summary

Complete CLI refactor for WAA benchmark automation. Replaces the 6800-line CLI with a minimal 1300-line implementation that uses the vanilla Microsoft WAA image.

Key Changes

CLI Refactor (-5500 lines)

• Replaced complex vm subcommand structure with flat commands: create, run, probe, analyze, etc.
• Removed unused code paths and simplified command routing
• Added analyze command to parse and summarize benchmark results
• Added --num-tasks to limit the number of tasks to run

Vanilla WAA Image Support

• Uses official windowsarena/winarena:latest Docker image
• Custom Dockerfile copies Python 3.9 from vanilla image (fixes transformers compatibility)
• IP patches for dockurr/windows compatibility (172.30.0.2)

Python 3.9 Compatibility Fix

• GroundingDINO requires transformers 4.46.2 (not 5.x)
• Fixed by copying Python 3.9 and all packages from vanilla WAA image
• This resolves: AttributeError: 'BertModel' has no attribute 'get_head_mask'

Results Analysis

• New analyze command parses downloaded benchmark logs
• Shows success rate by domain
• Handles ANSI color codes in logs

Commands

# Create VM with Docker and WAA image
uv run python -m openadapt_ml.benchmarks.cli create

# Run benchmark (auto-downloads results)
uv run python -m openadapt_ml.benchmarks.cli run --num-tasks 5 --model gpt-4o

# Check WAA server status
uv run python -m openadapt_ml.benchmarks.cli probe --wait

# Analyze results
uv run python -m openadapt_ml.benchmarks.cli analyze

# View logs
uv run python -m openadapt_ml.benchmarks.cli logs --lines 100

# Clean up
uv run python -m openadapt_ml.benchmarks.cli deallocate -y

Files Changed

• openadapt_ml/benchmarks/cli.py - Complete refactor (6800 → 1300 lines)
• openadapt_ml/benchmarks/waa_deploy/Dockerfile - Python 3.9 compatibility
• docs/CLI_V2_DESIGN.md - Design documentation
• .gitignore - Coverage and analysis artifacts

Test Plan

• probe - Correctly detects WAA server status
• run --num-tasks 2 - Limits tasks correctly
• analyze - Parses benchmark logs and shows results by domain
• logs - Shows container logs
• navi agent runs successfully (Python 3.9 fix working)

🤖 Generated with Claude Code