Evidence 0009-002: Local model quality and actor-policy benchmarks

[gidich]

Evidence gate: evidence-leadership-0009-002

Current benchmark status: not ready.

Top blockers from .agent-factory/benchmark-gate-report.json:

• local_model_quality:actor_policy:real_local_model_visible_fact_grounding_probe_failed
• local_model_quality:target_hardware:target_hardware_not_m4_profile

Acceptance posture:

• Run structured-output, hidden-truth, actor-policy, and target M4 Pro or M4 Max local model benchmarks.
• Keep hidden-truth leakage and actor-policy probes psychometrically explicit.
• Do not enable local dialogue in station runtime until the model quality gate clears.

[gidich]

Codex update: committed and pushed 484e02a fix: clarify visible actor policy probe. This adjusts the benchmark contract so visible_fact_grounding asks for the visible symptom description instead of onset, which was categorized as hidden in the prompt. Historical 2026-05-05 evidence remains blocked until a fresh approved local-model run on target hardware replaces it.

[gidich]

Committed and pushed fresh local model policy evidence in 14606ca. The corrected visible-fact probe was rerun through the approved offline Qwen/llama.cpp cache path. It still leaked hidden onset/cocaine facts, so evidence-leadership-0009-002 remains blocked by real_local_model_visible_fact_grounding_probe_failed plus the honest Apple M1 Max vs target M4 hardware blocker. Verification: focused local model/benchmark-gate tests; pnpm test:tools; pnpm typecheck; pnpm agent:benchmark-ids; pnpm agent:sources; pnpm security:audit-policy; pnpm security:licenses; git diff --check.

[gidich]

Local model actor-policy blocker narrowed in cdc96bd. Added a per-turn visible-fact-only disclosure boundary to the approved offline Qwen/llama.cpp actor-policy benchmark, reran the cached local model run with no cloud/API use, and refreshed the quality/gate evidence. The prior visible_fact_grounding hidden-fact leak is now cleared by external scanner evidence; the gate remains unresolved only on the honest target hardware blocker: current evidence is Apple M1 Max, while the acceptance target is Apple M4 Pro/Max. Verification: pnpm exec vitest run tools/openclinxr/local-model-quality-benchmark.test.ts tools/agent-factory/benchmark-gate-report.test.ts; pnpm test:tools; pnpm typecheck; pnpm agent:benchmark-ids; pnpm agent:sources; pnpm security:audit-policy; pnpm security:licenses; git diff --check.