Repository navigation
SYS REQ 031
The #250 evaluation system shall score deterministic Windows 11 synthetic UI tasks using ground truth independent of the model.
Epic #250 must show that evidence-grounded assistance improves a real task outcome while LBA capture remains active. Model self-evaluation cannot establish either correct state recognition or successful task completion. P1 issue #253 specifies a reproducible Windows 11 UI workload, a harness-only oracle, and a distinct synthetic-UI MCP server.
A fixed workload version and scenario seed produce the same finite UI-state and evidence-event sequence. The model receives only captured UI evidence and approved tool responses; oracle state is available only to the test harness and is absent from model prompts, MCP results, and agent-visible errors. The test harness records capture-active state and monotonic evidence/response/action timestamps. It scores state grounding, action correctness, and oracle-confirmed task completion as separate outcomes. Calibration records the model digest, guest image and resource profile, host reserve, capture settings, mode, scenario seeds, latency definitions, sample count, and uncertainty. Mode-specific p50/p95 latency and grounding/task-success thresholds remain unset until a reviewed calibration dataset is analyzed and the thresholds are frozen before qualification.
TEST-124 and TEST-126 offline workload/oracle and calibration-scorer cases are implemented through PR #254, merge 61d6e6dd3ff435d6d30c5edc370aea4d61f2236f. The full suite passed 2,231 tests at 97% scripts coverage; all seven PR checks passed and independent agent review found no findings. No Windows VM, live capture, or model run has been performed; system-level qualification remains outstanding.
System-Requirements-Specification; epic #250; P1 #253
High
Offline workload/oracle fixtures and a calibration-data scorer are implemented. The scorer reports separate per-mode latency, grounding, task-success statistics, sample counts, and uncertainty. No calibration dataset, threshold values, or live qualification evidence exists yet.
Offline workload/oracle/calibration components are scripts/assisted_benchmark_synthetic_ui.py, scripts/assisted_benchmark_synthetic_oracle.py, and scripts/assisted_benchmark_calibration.py, integrated through PR #254. Capture-time event association, host/model calibration, and live workflow remain unimplemented.
Offline workload, oracle, and scorer are implemented through PR #254; no calibration dataset or system-level live qualification is claimed.