Viewer consolidation, workflow segmentation, and VM monitoring

[abrichr]

Summary

Clean consolidation of three major feature sets from PR #7, with redundant commits removed. This PR adds:

1. Viewer Consolidation (Phase 1) - Foundation for migrating to reusable openadapt-viewer component library
2. Workflow Segmentation System - ML pipeline for extracting canonical workflows from recordings
3. VM Monitoring Enhancements - Comprehensive Azure VM usage visibility and cost tracking

This is a rebased and cleaned version of PR #7 that removes commits already merged via PR #6 (unified baseline adapters, CI workflow, linting fixes).

---

1. Viewer Consolidation (Phase 1)

Changes

• ✅ Add openadapt-viewer dependency via uv add openadapt-viewer
• ✅ Create adapter module openadapt_ml/training/viewer_components.py with ML-specific wrappers:
  • screenshot_with_predictions() - Screenshot display with human/AI action overlays
  • training_metrics() - Training statistics metrics grid
  • playback_controls() - Playback UI controls
  • correctness_badge() - Pass/fail badge component
  • generate_comparison_summary() - Model comparison summary
• ✅ Add screenshot tests tests/test_viewer_screenshots.py for component validation
• ✅ Create validation example openadapt_ml/training/viewer_migration_example.py
• ✅ Add search functionality to viewer with Ctrl+F / Cmd+F keyboard shortcut

Design Principles

1. Zero breaking changes - Existing viewer.py code remains unchanged
2. Adapter pattern - Wraps openadapt-viewer components with ML-specific context
3. Incremental adoption - Can be gradually adopted in future phases
4. Type-safe - Functions accept openadapt-ml data structures

Next Steps (Phase 2)

Once validated, Phase 2 will gradually migrate viewer.py functions to use these adapters.

---

2. Workflow Segmentation System

A complete ML pipeline for extracting canonical workflows from screen recordings for training data curation and demo retrieval.

Core Pipeline (4 Stages)

1. Stage 1 - Frame Description (VLM): Converts screenshots + actions into semantic descriptions

   • Supports Gemini, Claude, GPT-4o backends
   • Automatic caching for efficiency

2. Stage 2 - Episode Extraction (LLM): Identifies coherent workflow boundaries

   • Few-shot prompting for better quality
   • Confidence-based filtering

3. Stage 3 - Deduplication (Embeddings): Finds similar workflows across recordings

   • Agglomerative clustering with cosine similarity
   • Supports OpenAI or local HuggingFace embeddings

4. Stage 4 - Annotation (VLM Quality Control): Auto-annotates episodes for training quality

   • Detects failures, boundary issues, incompleteness
   • Human-in-the-loop review workflow

Integration Features

• CaptureAdapter: Loads recordings from openadapt-capture SQLite format
• Unified Pipeline: Run all stages with single API
• CLI Interface: Full command-line interface for all stages
• Comprehensive Documentation: 920-line README with 20+ code examples

Use Cases

1. Training Data Curation: Extract and filter high-quality demonstration episodes
2. Demo Retrieval: Build searchable libraries for demo-conditioned prompting
3. Workflow Documentation: Auto-generate step-by-step guides from recordings

Files Added

• openadapt_ml/segmentation/ (9 new modules, 5,313 lines)
• openadapt_ml/segmentation/README.md (920 lines of documentation)
• tests/test_segmentation_pipeline.py + tests/test_capture_adapter.py
• docs/SEGMENTATION_TEST_PLAN.md + docs/SEGMENTATION_TEST_RESULTS.md

---

3. VM Monitoring Enhancements

Enhanced the vm monitor command to provide comprehensive Azure VM usage visibility:

Features

• Real-time VM Status: VM state, size, IP, power state
• Activity Detection: What the VM is currently doing (idle, benchmark running, setup)
• Cost Tracking: Current uptime, hourly rate, total cost for session
• Azure ML Jobs: Recent jobs from last 7 days with status
• Evaluation History: Past benchmark runs and success rates (with --details flag)
• Mock Mode: Testing without VM (with --mock flag)

Usage

# Basic monitoring
uv run python -m openadapt_ml.benchmarks.cli vm monitor

# With detailed information
uv run python -m openadapt_ml.benchmarks.cli vm monitor --details

# With auto-shutdown after 2 hours
uv run python -m openadapt_ml.benchmarks.cli vm monitor --auto-shutdown-hours 2

# Mock mode for testing
uv run python -m openadapt_ml.benchmarks.cli vm monitor --mock

Files Changed

• openadapt_ml/benchmarks/vm_monitor.py - New monitoring functions
• openadapt_ml/benchmarks/cli.py - Enhanced vm monitor command
• openadapt_ml/cloud/local.py - Dashboard API endpoints
• README.md - Updated with VM monitor documentation

---

Documentation

New Documentation Files

• docs/REPOSITORY_HISTORY.md - Archive record of OpenAdapter repository
• docs/SEGMENTATION_TEST_PLAN.md - Test plan for workflow segmentation
• docs/SEGMENTATION_TEST_RESULTS.md - Test results from real captures
• docs/VM_MONITOR_SCREENSHOT_IMPLEMENTATION.md - VM screenshot feature plan
• docs/vm_monitor_screenshot_analysis.md - Screenshot capture approaches
• docs/screenshots/ - VM monitor dashboard screenshots

---

Test Plan

• All baseline tests pass (58 tests)
• Ruff linting passes
• Ruff formatting passes
• Viewer components import correctly
• Segmentation pipeline modules load
• VM monitor commands work (tested with --mock)

---

Relation to PR #7

This PR is a cleaned and rebased version of PR #7 that:

• Removes commits already merged via PR Add GitHub Actions CI workflow #6 (baseline adapters, CI, linting)
• Keeps only new features: viewer consolidation, segmentation system, VM monitoring
• Resolves all merge conflicts by rebasing on current main
• Adds linting fixes for the new code

The original PR #7 will be closed in favor of this cleaner version.

---

Breaking Changes

None - all changes are additive.

🤖 Generated with Claude Code