feat: add YAML-based custom task evaluation without forking WAA

[abrichr]

Summary

• Simple YAML format for defining custom tasks with real end-to-end evaluation
• No WAA fork or Docker modification required — evaluator configs sent client-side
• Four check types: command (PowerShell/Python output), file (exists/contains), screenshot (VLM judge), python (arbitrary code)
• Milestone support for dense partial rewards (reward = milestones_passed / total)
• 5 example tasks: notepad hello, desktop folder, calc formula, clear browsing data (Chrome + Edge)
• 22 tests passing

How it works

Users write YAML, our code translates to WAA evaluator format at runtime:

name: "Open Notepad and type Hello World"
setup:
  - launch: "notepad.exe"
evaluate:
  - check: command
    run: "powershell -c 'Get-Process notepad | Measure | Select -Expand Count'"
    expect: "1"
  - check: screenshot
    description: "Notepad shows Hello World"
milestones:
  - name: "Notepad open"
    check: command
    run: "powershell -c 'Get-Process notepad | Measure | Select -Expand Count'"
    expect: "1"

Test plan

• 22 unit tests passing (YAML loading, WAA translation, VLM judge, milestones, examples)
• End-to-end test against live WAA VM with example tasks

🤖 Generated with Claude Code