Repository navigation
REQ 123
The standalone Windows 11 synthetic UI workload shall provide a deterministic finite-state task and an independent harness-only oracle for its actual outcome.
Epic #250 evaluates assistance during active capture. A model-generated success statement cannot serve as ground truth. The standalone UI and oracle contract is tracked by P1 #253; LabVIEW/VIPM provisioning remains a separate epic concern.
For a fixed workload version and seed, the UI produces the same finite state sequence, visible controls, and ordered event/frame identities. The agent sees only the captured UI and approved action results. Oracle state is exposed only through a harness-only interface, never through model prompts, MCP tools/results, or agent-visible errors. A task passes only when the oracle observes the expected terminal state after the authorized UI action; model text alone cannot pass it. The oracle separately reports current-state/action correctness and terminal task completion.
TEST-124 executed deterministic-seed, invalid-action, oracle-isolation, and completion-scoring cases through PR #254, merge 61d6e6dd3ff435d6d30c5edc370aea4d61f2236f. The full suite passed 2,231 tests at 97% scripts coverage; all seven PR checks passed. The review was performed by an independent agent with no findings; the administrative missing-GitHub-review bypass is recorded on the PR and is not an approval. No VM, Ollama inference, or live LBA capture was performed.
SYS-REQ-031; epic #250; P1 #253
High
scripts/tests/test_assisted_benchmark_synthetic_ui.py executed test_seeded_ui_states_and_event_ids_are_repeatable, test_independent_oracle_scores_seeded_task, test_wrong_action_is_recorded_without_leaking_success_or_expected_target, test_oracle_detects_a_faulty_workload_scenario_generator, test_oracle_scores_model_state_claim_separately_from_task_success, test_mcp_handlers_reject_wrong_task_malformed_action_and_duplicate_submission, and test_stdio_mcp_server_exposes_only_exact_ui_action_and_returns_no_oracle_data. The integrated test_stdio_ui_action_flows_through_server_set_broker_router_and_oracle exercises the full local fixture. No live VM or model inference was used.
Implemented offline in scripts/assisted_benchmark_synthetic_ui.py, scripts/assisted_benchmark_synthetic_oracle.py, scripts/assisted_benchmark_synthetic_ui_server.py, scripts/assisted_benchmark_synthetic_ui_mcp_server.py, and tools/assisted-benchmark/synthetic-ui/index.html through PR #254. The oracle scores public observations and model state claims independently; it is not exposed to the UI/MCP server.
Implemented offline in develop through PR #254. No live Windows VM, LBA capture, or Ollama inference qualification is claimed.