QA integrity fixes: prompt-safety, diff_size, reference audits, clean admissions (Phase 0)

[ElbertePlinio]

From the 2026-07-08 multi-model audit (gpt-5.5 + opus-4.8 + glm-5.2 panel + two code scouts). These fixes block trusting any published number, so they come first.

Ordered checklist (dependency order matters):

• 1. Prompt-safety QA scans the resolved prompt (instruction.md / task prompt after template substitution) for hidden paths, verifier names, and reference literals. Today _checkPromptSafety (app/lib/runner/task_qa_runner.dart:878) scans only fixture context, never the prompt itself. Avoid naive substring matching (false positives) — scan hidden paths / verifier filenames / reference code explicitly.
• 2. Demote diff_size to diagnostic-only: remove it from pass/fail and aggregate blocking (app/lib/evaluators/diff_size_evaluator.dart k=20 threshold 0.3; weight 0.3 in app/lib/core/scoring.dart). Measure the full patch against the clean baseline, not one fixture path. Version the methodology change; check whether the published board depended on it.
• 3. Audit all 10 reference solutions against their own task contracts (Replace fixed-width breakpoints in ui.action_bar_overflow reference solution with measurement-based overflow #3 is instance one). Bump task version + regenerate admission whenever a reference changes.
• 4. Close the remaining QA/sandbox hardening: npm-installed Droid binds + sandboxed admission validation in tests #4 defect: official_file_backed_task_test validates admission reports unsandboxed while asserting generatedCodeSandbox.enforced == true. Re-baseline against already-landed sandbox commits 3984bb5 / 7cf7eb8 first so we don't redo landed work.
• 5. Regenerate all 10 admission reports from clean HEAD (today all record gitDirty=true @ 68321ee). Add admission+release gates failing on gitDirty or task-bundle digest mismatch (digest-based so docs-only commits don't overblock). Budget for tasks that only "passed" under stale evidence.
• 6. Relabel the current published 1-model leaderboard as provisional until regenerated evidence lands.

Refs #3 #4

[ElbertePlinio]

Parent plan: #10

[ElbertePlinio]

Progress update:

• Scope confirmed: checklist item 1 only
• Inspect prompt loading and current prompt-safety scan
• Add regression coverage for instruction.md hidden filename leak
• Implement resolved-prompt scan with token-aware hidden/reference matching
• Run requested validation

PR: pending
Blocked: none
Next: inspect runner/tests

[ElbertePlinio]

Progress update:

• Resolved task prompt is now included in prompt-safety scanning
• Hidden filenames/paths, verifier IDs, negative-case paths, reference paths, and distinctive reference snippets are covered with token-aware checks
• Added regression for instruction.md leaking a hidden test filename
• Committed: 3b18519

Validation:

• dart format --set-exit-if-changed lib test: pass
• flutter analyze: pass
• flutter test test/tasks/file_backed_task_test.dart: pass
• flutter test test/tasks/ test/runner/ 2>/dev/null || flutter test: first suite reached the existing 20-minute corpus_task_qa_test timeout after official file-backed checks completed; no prompt-safety leak reported in the 10 official tasks. Fallback full-suite rerun was stopped to avoid duplicating the same timeout.

PR: pending
Blocked: none for checklist item 1; broader corpus timeout is separate
Next: PR/review when requested

[ElbertePlinio]

Progress update:

• Resolved prompt is now included in prompt-safety scanning
• Added hidden filename leak regression
• Format, analyze, focused regression, and official 10-task admission test passed
• Committed on fix/qa-prompt-safety-scan

Commit: 3b18519
Real official prompt leaks found: none
Blocked: none

[ElbertePlinio]

Review fixes amended into c5dd7ce:

• prompt-safety leak logic moved to a fast-testable runner module
• hidden verifier content now uses distinctive snippet matching with public-fixture exclusion
• path boundary, bare hidden_tests marker, extensionless hidden stems, and short id/stem guards covered
• validation passed: format check, analyze, prompt_safety_test, file_backed_task_test, official_file_backed_task_test (30:34)

[ElbertePlinio]

Residual prompt-safety normalization fix amended into 2deb80a:

• distinctive hidden/reference snippet comparison now casefolds and collapses whitespace around punctuation
• added fast tests for case-variant and spacing-variant pasted snippets
• validation passed: format check, analyze, prompt_safety_test, file_backed_task_test
• official 30-minute regression skipped per instruction after grep found no private path markers or hidden/reference assertion/arrow quote patterns in official instruction.md files

[ElbertePlinio]

PR opened for item 1 (prompt-safety scan): #11 — in Codex review. Remaining items unblocked and next up: diff_size demotion, all-10 reference audit (#3), #4 test fix, clean admission regeneration.

[ElbertePlinio]

Progress update:

• Scope confirmed: checklist item 2 only
• Inspect diff_size evaluator, scoring, patch capture, and exports
• Demote diff_size to diagnostic-only and keep telemetry
• Measure full workspace patch instead of one generated path
• Update tests and schema/version metadata
• Run requested validation

PR: pending
Blocked: none
Next: inspect evaluator/scoring paths

[ElbertePlinio]

Reference-contract audit of all 10 tasks complete: 8 OK, 1 violation (ui.action_bar_overflow — #3, fixed-width thresholds), 1 smell (forms.email_validation regex accepts consecutive-dot domains). Fix branch fix/reference-contracts in progress for both, with task-version bumps + per-task admission regeneration.

[ElbertePlinio]

Progress update:

• Checklist item 2 complete: diff_size is diagnostic-only and always passes
• Full patch measurement implemented: agentic uses captured git patch; codegen falls back to all fixture paths in the workspace
• diff_size removed from aggregate scoring weights/ranking influence, telemetry/export columns retained
• Evaluator/scoring schema metadata bumped; official admission evidence metadata updated to evaluator schema 2
• Validation passed
• Committed: b235923

Validation:

• dart format --set-exit-if-changed lib test: pass
• flutter analyze: pass
• flutter test test/evaluators/ 2>/dev/null || flutter test test/ -x slow: pass (test/evaluators/, 47 tests)
• Focused diff_size/scoring/export tests: pass (133 tests)
• flutter test test/tasks/official_file_backed_task_test.dart: pass (11 tests, final run 33:56)

Admission evidence: no task admission checks changed status; checked-in admission reports only bump nested evaluator.schemaVersion 1 -> 2.
PR: pending
Blocked: none