Repository navigation
REQ 125
The live-assistance qualification procedure shall freeze per-mode latency, grounding, and task-success thresholds from a separate reviewed calibration dataset before qualifying runs.
Epic #250 compares on-demand, periodic, and continuous interaction. Thresholds chosen before representative observations risk rewarding one mode or hiding poor task outcomes. P1 issue #253 defines the Windows 11 synthetic task and calibration protocol.
Calibration records a fixed workload version and seed set, interaction mode, approved model digest and execution settings, Windows guest image and resource profile, host resource reserve, capture configuration, sample count, and uncertainty. Using a monotonic clock, it reports per-mode p50/p95 from evidence availability to first and final response token and from evidence availability to oracle-confirmed action; it separately reports state/action grounding and oracle-confirmed task success. Calibration runs are labeled non-qualifying and kept distinct from qualification data. No numeric threshold is claimed until the calibration data and method are reviewed; a versioned requirement freezes the selected thresholds before any qualification run, and qualification results cannot be used to tune them.
TEST-126 executed deterministic metric calculations, per-mode separation, uncertainty/sample-count reporting, and malformed/inconsistent-record rejection through PR #254, merge 61d6e6dd3ff435d6d30c5edc370aea4d61f2236f. The full suite passed 2,231 tests at 97% scripts coverage; all seven PR checks passed. No calibration dataset or qualification run exists, so numeric thresholds are not selected or frozen.
SYS-REQ-031; epic #250; P1 #253
High
scripts/tests/test_assisted_benchmark_synthetic_ui.py executed test_calibration_summarizes_each_mode_separately_and_never_qualifies, test_calibration_rejects_mixed_seeds_and_inconsistent_configuration, test_calibration_rejects_malformed_records, and test_calibration_single_sample_reports_unestimated_uncertainty. These validate the offline scorer only. No calibration dataset, selected model, numeric threshold, or qualifying run exists.
Offline record validation and summary metrics are implemented in scripts/assisted_benchmark_calibration.py through PR #254. Threshold selection and freezing still require a separately reviewed calibration dataset.
Partially implemented offline; the scorer is available, but this requirement is not satisfied until reviewed Windows 11 calibration data is collected and thresholds are frozen before qualification.