test(eval): deterministic agent-efficacy harness (UAT-EVAL-1..5)

[kunaldhongade]

Summary

• Closes the first closable slice of test(eval): prove CodeDecay improves verified agent outcomes over the agent alone #683: deterministic fake-agent control/treatment harness covering UAT-EVAL-1..5 (integrity, contamination, failed/unavailable providers, schema report via pnpm eval:agent-efficacy).
• Adds versioned human UAT kit scaffolding under docs/evals/uat-kit/ for test(uat): validate the published AI workflow with independent developers #692 (script, rubric, consent, result schema). Does not close test(uat): validate the published AI workflow with independent developers #692 — that still needs independent human sessions.
• Real-agent efficacy baselines and epic(ai-native): make CodeDecay a verifiably AI-first engineering safety agent #675 10/10 claims remain out of scope; harness stays local-first with no hidden provider calls.

Test plan

• vitest run packages/knowledge/test/eval-harness.test.ts packages/knowledge/test/human-uat-kit.test.ts
• tsc -p tsconfig.base.json --noEmit
• CI green on PR
• Confirm test(eval): prove CodeDecay improves verified agent outcomes over the agent alone #683 closes on merge; test(uat): validate the published AI workflow with independent developers #692 stays open

Closes #683
Refs #692

Made with Cursor