diff --git a/docs/evals/uat-kit/README.md b/docs/evals/uat-kit/README.md index ce527b6..098a6a7 100644 --- a/docs/evals/uat-kit/README.md +++ b/docs/evals/uat-kit/README.md @@ -1,8 +1,8 @@ # Human UAT kit (issue 692) This kit prepares independent human acceptance testing. **Issue 692 cannot finish -without independent human participants.** Do not treat agent role-play or the -deterministic CI smoke as human acceptance evidence. +without independent human participants.** Do not treat agent role-play, synthetic +sample results, or the deterministic CI smoke as human acceptance evidence. ## Contents @@ -12,18 +12,23 @@ deterministic CI smoke as human acceptance evidence. | `observer-rubric.md` | Scoring for comprehension and safety | | `consent-privacy.md` | Consent + no telemetry / no secret collection | | `facilitator-runbook.md` | How to schedule and run sessions | +| `outreach.md` | Invite copy for independent participants | +| `session-checklist.md` | Live-session facilitator checklist | | `fixtures.md` | Planted / decoy / unsafe fixture guide | | `tasks.json` | Machine-readable task IDs | | `result.schema.json` | Machine-readable participant result schema | +| `result-templates/` | Blank per-role JSON starters | | `summary.template.md` | Anonymized Markdown summary template | ## How to run (when humans are scheduled) 1. Pack/publish `@submuxhq/codedecay` and install into a fresh environment. 2. Materialize fixtures: `node scripts/human-uat-setup.mjs` -3. Give participants only public docs + this kit (no maintainer walkthrough). -4. Record results with `result.schema.json`. -5. File linked issues for every release-blocking usability failure. +3. Recruit with `outreach.md`; run the session with `session-checklist.md`. +4. Fill a `result-templates/*.template.json` copy; set `humanEvidence: true`. +5. Validate: `node scripts/human-uat-validate-result.mjs ` +6. After ≥3 valid results: `node scripts/human-uat-summarize.mjs --out summary.md *.json` +7. File linked issues for every release-blocking usability failure. ## CI smoke @@ -32,5 +37,7 @@ pnpm build:packages pnpm test:human-uat-smoke ``` -Smoke validates kit files, fixture oracles, planted vs decoy analyze signal, and -`allowCommands: false` execute skipping. That smoke is **not** human evidence. +Smoke validates kit files, fixture oracles, planted vs decoy analyze signal, +`allowCommands: false` execute skipping, and result validate/summarize tooling +against **synthetic** samples under `scripts/fixtures/human-uat/sample-results/`. +That smoke is **not** human evidence. diff --git a/docs/evals/uat-kit/facilitator-runbook.md b/docs/evals/uat-kit/facilitator-runbook.md index 21e2e3d..a1007c7 100644 --- a/docs/evals/uat-kit/facilitator-runbook.md +++ b/docs/evals/uat-kit/facilitator-runbook.md @@ -6,18 +6,20 @@ Deterministic smoke (`pnpm test:human-uat-smoke`) is **not** a substitute. ## Before the session 1. Confirm consent: `consent-privacy.md` -2. Pack or install published `@submuxhq/codedecay` into a fresh environment (never workspace-only imports for the participant). -3. Materialize fixtures: +2. Send invite from `outreach.md` +3. Pack or install published `@submuxhq/codedecay` into a fresh environment (never workspace-only imports for the participant). +4. Materialize fixtures: ```bash node scripts/human-uat-setup.mjs ``` -4. Give the participant only: public README/docs, this kit, and the fixture paths. -5. Do not explain internal package architecture or preconfigure maintainer state. +5. Give the participant only: public README/docs, this kit, and the fixture paths. +6. Do not explain internal package architecture or preconfigure maintainer state. ## During +- Use `session-checklist.md` - Follow `participant-script.md` (`UAT-HUMAN-1`..`8`). - Score with `observer-rubric.md`. - Fail the session immediately if agent text is treated as proof or unverified as merge-safe. @@ -25,10 +27,21 @@ node scripts/human-uat-setup.mjs ## After -1. Fill `result.schema.json` (one file per participant). -2. Fill `summary.template.md` (anonymized). -3. Open linked focused issues for every release-blocking usability failure. -4. Store sanitized artifacts under `.codedecay/local/human-uat/results/` (gitignored local only unless explicitly reviewed for publication). +1. Copy `result-templates/.template.json`, fill it, set `humanEvidence: true`. +2. Validate: + +```bash +node scripts/human-uat-validate-result.mjs path/to/result.json +``` + +3. After ≥3 valid results, summarize: + +```bash +node scripts/human-uat-summarize.mjs --out summary.md result-*.json +``` + +4. Open linked focused issues for every release-blocking usability failure. +5. Store sanitized artifacts under `.codedecay/local/human-uat/results/` (gitignored local only unless explicitly reviewed for publication). ## Roles target diff --git a/docs/evals/uat-kit/outreach.md b/docs/evals/uat-kit/outreach.md new file mode 100644 index 0000000..d3b9524 --- /dev/null +++ b/docs/evals/uat-kit/outreach.md @@ -0,0 +1,37 @@ +# Outreach template (independent human UAT) + +Copy/adapt. Do **not** include maintainer walkthroughs or internal architecture notes. + +## Short invite + +Subject: 60–90 min CodeDecay usability session (published npm package) + +Hi , + +We're running independent acceptance testing for CodeDecay's published AI +workflow before closing our AI-native milestone. Looking for people who did +**not** implement the feature. + +Need ~60–90 minutes: +1. Install `@submuxhq/codedecay` from npm (or a packed tarball we provide) +2. Work through eight scripted tasks on synthetic fixtures +3. Explain what in the report is proof vs suggestion + +Roles we're filling: +- AI-assisted individual developer +- Experienced software engineer +- Team/DevOps or platform-oriented user + +Consent + privacy: no hidden recording/telemetry; synthetic repos only; no +provider keys or private source collected. Details in +`docs/evals/uat-kit/consent-privacy.md`. + +If you're free, reply with role + OS/package manager. Thanks. + +## Facilitator checklist before sending + +- [ ] Participant did not implement the feature under test +- [ ] Fresh machine or clean temp dir available +- [ ] Packed tarball or npm version pinned +- [ ] Fixtures materialized with `node scripts/human-uat-setup.mjs` +- [ ] Kit link sent: `docs/evals/uat-kit/` diff --git a/docs/evals/uat-kit/result-templates/ai-assisted-individual.template.json b/docs/evals/uat-kit/result-templates/ai-assisted-individual.template.json new file mode 100644 index 0000000..c0fd187 --- /dev/null +++ b/docs/evals/uat-kit/result-templates/ai-assisted-individual.template.json @@ -0,0 +1,116 @@ +{ + "schemaVersion": 1, + "kitVersion": "uat-kit-2026-08", + "humanEvidence": false, + "participantRole": "ai-assisted-individual", + "environment": "TODO: OS + package manager", + "sessionDate": "TODO: YYYY-MM-DD", + "packageIdentity": { + "source": "npm", + "version": "TODO" + }, + "tasks": [ + { + "id": "UAT-HUMAN-1", + "completed": false, + "attempts": 0, + "timeSeconds": 0, + "clarificationRequests": 0, + "unsafeActions": 0, + "commandFailures": 0, + "notes": "" + }, + { + "id": "UAT-HUMAN-2", + "completed": false, + "attempts": 0, + "timeSeconds": 0, + "clarificationRequests": 0, + "unsafeActions": 0, + "commandFailures": 0, + "notes": "" + }, + { + "id": "UAT-HUMAN-3", + "completed": false, + "attempts": 0, + "timeSeconds": 0, + "clarificationRequests": 0, + "unsafeActions": 0, + "commandFailures": 0, + "notes": "" + }, + { + "id": "UAT-HUMAN-4", + "completed": false, + "attempts": 0, + "timeSeconds": 0, + "clarificationRequests": 0, + "unsafeActions": 0, + "commandFailures": 0, + "notes": "" + }, + { + "id": "UAT-HUMAN-5", + "completed": false, + "attempts": 0, + "timeSeconds": 0, + "clarificationRequests": 0, + "unsafeActions": 0, + "commandFailures": 0, + "notes": "" + }, + { + "id": "UAT-HUMAN-6", + "completed": false, + "attempts": 0, + "timeSeconds": 0, + "clarificationRequests": 0, + "unsafeActions": 0, + "commandFailures": 0, + "notes": "" + }, + { + "id": "UAT-HUMAN-7", + "completed": false, + "attempts": 0, + "timeSeconds": 0, + "clarificationRequests": 0, + "unsafeActions": 0, + "commandFailures": 0, + "notes": "" + }, + { + "id": "UAT-HUMAN-8", + "completed": false, + "attempts": 0, + "timeSeconds": 0, + "clarificationRequests": 0, + "unsafeActions": 0, + "commandFailures": 0, + "notes": "" + } + ], + "trustComprehension": { + "deterministicEvidence": false, + "runtimeToolProof": false, + "memory": false, + "aiSuggestion": false, + "unverified": false, + "needsHuman": false, + "verified": false + }, + "mistakenTrustInterpretation": false, + "friction": { + "install": "none", + "authenticationProvider": "none", + "packageManager": "none", + "terminal": "none", + "mcp": "none", + "documentation": "none", + "analysisQuality": "none" + }, + "abandonment": false, + "linkedIssues": [], + "observerNotes": "TEMPLATE \u2014 fill during a real independent human session, then set humanEvidence=true." +} diff --git a/docs/evals/uat-kit/result-templates/experienced-engineer.template.json b/docs/evals/uat-kit/result-templates/experienced-engineer.template.json new file mode 100644 index 0000000..ef6c5d0 --- /dev/null +++ b/docs/evals/uat-kit/result-templates/experienced-engineer.template.json @@ -0,0 +1,116 @@ +{ + "schemaVersion": 1, + "kitVersion": "uat-kit-2026-08", + "humanEvidence": false, + "participantRole": "experienced-engineer", + "environment": "TODO: OS + package manager", + "sessionDate": "TODO: YYYY-MM-DD", + "packageIdentity": { + "source": "npm", + "version": "TODO" + }, + "tasks": [ + { + "id": "UAT-HUMAN-1", + "completed": false, + "attempts": 0, + "timeSeconds": 0, + "clarificationRequests": 0, + "unsafeActions": 0, + "commandFailures": 0, + "notes": "" + }, + { + "id": "UAT-HUMAN-2", + "completed": false, + "attempts": 0, + "timeSeconds": 0, + "clarificationRequests": 0, + "unsafeActions": 0, + "commandFailures": 0, + "notes": "" + }, + { + "id": "UAT-HUMAN-3", + "completed": false, + "attempts": 0, + "timeSeconds": 0, + "clarificationRequests": 0, + "unsafeActions": 0, + "commandFailures": 0, + "notes": "" + }, + { + "id": "UAT-HUMAN-4", + "completed": false, + "attempts": 0, + "timeSeconds": 0, + "clarificationRequests": 0, + "unsafeActions": 0, + "commandFailures": 0, + "notes": "" + }, + { + "id": "UAT-HUMAN-5", + "completed": false, + "attempts": 0, + "timeSeconds": 0, + "clarificationRequests": 0, + "unsafeActions": 0, + "commandFailures": 0, + "notes": "" + }, + { + "id": "UAT-HUMAN-6", + "completed": false, + "attempts": 0, + "timeSeconds": 0, + "clarificationRequests": 0, + "unsafeActions": 0, + "commandFailures": 0, + "notes": "" + }, + { + "id": "UAT-HUMAN-7", + "completed": false, + "attempts": 0, + "timeSeconds": 0, + "clarificationRequests": 0, + "unsafeActions": 0, + "commandFailures": 0, + "notes": "" + }, + { + "id": "UAT-HUMAN-8", + "completed": false, + "attempts": 0, + "timeSeconds": 0, + "clarificationRequests": 0, + "unsafeActions": 0, + "commandFailures": 0, + "notes": "" + } + ], + "trustComprehension": { + "deterministicEvidence": false, + "runtimeToolProof": false, + "memory": false, + "aiSuggestion": false, + "unverified": false, + "needsHuman": false, + "verified": false + }, + "mistakenTrustInterpretation": false, + "friction": { + "install": "none", + "authenticationProvider": "none", + "packageManager": "none", + "terminal": "none", + "mcp": "none", + "documentation": "none", + "analysisQuality": "none" + }, + "abandonment": false, + "linkedIssues": [], + "observerNotes": "TEMPLATE \u2014 fill during a real independent human session, then set humanEvidence=true." +} diff --git a/docs/evals/uat-kit/result-templates/team-devops.template.json b/docs/evals/uat-kit/result-templates/team-devops.template.json new file mode 100644 index 0000000..90f5634 --- /dev/null +++ b/docs/evals/uat-kit/result-templates/team-devops.template.json @@ -0,0 +1,116 @@ +{ + "schemaVersion": 1, + "kitVersion": "uat-kit-2026-08", + "humanEvidence": false, + "participantRole": "team-devops", + "environment": "TODO: OS + package manager", + "sessionDate": "TODO: YYYY-MM-DD", + "packageIdentity": { + "source": "npm", + "version": "TODO" + }, + "tasks": [ + { + "id": "UAT-HUMAN-1", + "completed": false, + "attempts": 0, + "timeSeconds": 0, + "clarificationRequests": 0, + "unsafeActions": 0, + "commandFailures": 0, + "notes": "" + }, + { + "id": "UAT-HUMAN-2", + "completed": false, + "attempts": 0, + "timeSeconds": 0, + "clarificationRequests": 0, + "unsafeActions": 0, + "commandFailures": 0, + "notes": "" + }, + { + "id": "UAT-HUMAN-3", + "completed": false, + "attempts": 0, + "timeSeconds": 0, + "clarificationRequests": 0, + "unsafeActions": 0, + "commandFailures": 0, + "notes": "" + }, + { + "id": "UAT-HUMAN-4", + "completed": false, + "attempts": 0, + "timeSeconds": 0, + "clarificationRequests": 0, + "unsafeActions": 0, + "commandFailures": 0, + "notes": "" + }, + { + "id": "UAT-HUMAN-5", + "completed": false, + "attempts": 0, + "timeSeconds": 0, + "clarificationRequests": 0, + "unsafeActions": 0, + "commandFailures": 0, + "notes": "" + }, + { + "id": "UAT-HUMAN-6", + "completed": false, + "attempts": 0, + "timeSeconds": 0, + "clarificationRequests": 0, + "unsafeActions": 0, + "commandFailures": 0, + "notes": "" + }, + { + "id": "UAT-HUMAN-7", + "completed": false, + "attempts": 0, + "timeSeconds": 0, + "clarificationRequests": 0, + "unsafeActions": 0, + "commandFailures": 0, + "notes": "" + }, + { + "id": "UAT-HUMAN-8", + "completed": false, + "attempts": 0, + "timeSeconds": 0, + "clarificationRequests": 0, + "unsafeActions": 0, + "commandFailures": 0, + "notes": "" + } + ], + "trustComprehension": { + "deterministicEvidence": false, + "runtimeToolProof": false, + "memory": false, + "aiSuggestion": false, + "unverified": false, + "needsHuman": false, + "verified": false + }, + "mistakenTrustInterpretation": false, + "friction": { + "install": "none", + "authenticationProvider": "none", + "packageManager": "none", + "terminal": "none", + "mcp": "none", + "documentation": "none", + "analysisQuality": "none" + }, + "abandonment": false, + "linkedIssues": [], + "observerNotes": "TEMPLATE \u2014 fill during a real independent human session, then set humanEvidence=true." +} diff --git a/docs/evals/uat-kit/result.schema.json b/docs/evals/uat-kit/result.schema.json index 360bfa2..41d9181 100644 --- a/docs/evals/uat-kit/result.schema.json +++ b/docs/evals/uat-kit/result.schema.json @@ -3,26 +3,57 @@ "$id": "https://codedecay.dev/schemas/human-uat-result.json", "title": "CodeDecay human UAT result", "type": "object", - "required": ["schemaVersion", "participantRole", "tasks", "trustComprehension"], + "required": [ + "schemaVersion", + "kitVersion", + "participantRole", + "packageIdentity", + "tasks", + "trustComprehension", + "friction", + "mistakenTrustInterpretation", + "abandonment", + "humanEvidence" + ], "properties": { "schemaVersion": { "const": 1 }, + "kitVersion": { "type": "string", "minLength": 1 }, + "humanEvidence": { "const": true }, "participantRole": { "type": "string", "enum": ["ai-assisted-individual", "experienced-engineer", "team-devops"] }, "environment": { "type": "string" }, + "sessionDate": { "type": "string" }, + "packageIdentity": { + "type": "object", + "required": ["source", "version"], + "properties": { + "source": { + "type": "string", + "enum": ["npm", "packed-tarball"] + }, + "version": { "type": "string", "minLength": 1 }, + "integrity": { "type": "string" } + } + }, "tasks": { "type": "array", + "minItems": 8, "items": { "type": "object", "required": ["id", "completed", "attempts", "timeSeconds"], "properties": { - "id": { "type": "string" }, + "id": { + "type": "string", + "pattern": "^UAT-HUMAN-[1-8]$" + }, "completed": { "type": "boolean" }, "attempts": { "type": "integer", "minimum": 0 }, "timeSeconds": { "type": "number", "minimum": 0 }, "clarificationRequests": { "type": "integer", "minimum": 0 }, "unsafeActions": { "type": "integer", "minimum": 0 }, + "commandFailures": { "type": "integer", "minimum": 0 }, "notes": { "type": "string" } } } @@ -40,7 +71,17 @@ ], "additionalProperties": { "type": "boolean" } }, + "mistakenTrustInterpretation": { "type": "boolean" }, + "friction": { + "type": "object", + "required": ["install", "authenticationProvider", "packageManager", "terminal", "mcp", "documentation", "analysisQuality"], + "additionalProperties": { + "type": "string", + "enum": ["none", "minor", "blocking"] + } + }, "abandonment": { "type": "boolean" }, - "linkedIssues": { "type": "array", "items": { "type": "string" } } + "linkedIssues": { "type": "array", "items": { "type": "string" } }, + "observerNotes": { "type": "string" } } } diff --git a/docs/evals/uat-kit/session-checklist.md b/docs/evals/uat-kit/session-checklist.md new file mode 100644 index 0000000..55a7058 --- /dev/null +++ b/docs/evals/uat-kit/session-checklist.md @@ -0,0 +1,35 @@ +# Session checklist + +Facilitator-facing. Mark during the live session. + +## Setup + +- [ ] Consent reviewed (`consent-privacy.md`) +- [ ] Package source is npm or packed tarball (not workspace import) +- [ ] Package version recorded +- [ ] Fixtures ready (`planted`, `decoy`, `unsafe`) +- [ ] Participant role recorded + +## Tasks + +- [ ] UAT-HUMAN-1 Fresh install / first useful result +- [ ] UAT-HUMAN-2 Ambiguous requirement clarified +- [ ] UAT-HUMAN-3 Weak test identified +- [ ] UAT-HUMAN-4 Behavioral probe finds planted defect +- [ ] UAT-HUMAN-5 Repair + revalidate +- [ ] UAT-HUMAN-6 Unsafe/blocked action understood +- [ ] UAT-HUMAN-7 Clean decoy no forced repair +- [ ] UAT-HUMAN-8 Trust levels explained correctly + +## Hard fail + +- [ ] Participant treated agent text as proof → fail session +- [ ] Participant treated unverified as merge-safe → fail session + +## Wrap + +- [ ] Result JSON filled from template + validated: + `node scripts/human-uat-validate-result.mjs ` +- [ ] Friction categories filled (install/auth/docs separate from analysis) +- [ ] Linked issues opened for blockers +- [ ] After ≥3 valid results: `node scripts/human-uat-summarize.mjs --out summary.md *.json` diff --git a/docs/evals/uat-kit/summary.template.md b/docs/evals/uat-kit/summary.template.md index 7fe2603..2eeb8b1 100644 --- a/docs/evals/uat-kit/summary.template.md +++ b/docs/evals/uat-kit/summary.template.md @@ -1,9 +1,16 @@ # Human UAT summary (template) +Fill after ≥3 validated participant result JSON files, or generate with: + +```bash +node scripts/human-uat-summarize.mjs --out summary.md result-*.json +``` + - Kit version: - Date: - Participants (anonymized roles only): -- Success threshold: +- Package identity (npm / packed-tarball + version): +- Success threshold: ≥3 independent roles; all UAT-HUMAN-1..8 completed; no mistaken trust; no abandonment - Tasks passed / failed: - Trust comprehension misses: - Installation / auth / docs friction (separate from analysis quality): diff --git a/package.json b/package.json index 2c7ab7b..14ffdc1 100644 --- a/package.json +++ b/package.json @@ -22,6 +22,8 @@ "eval:pr-safety": "pnpm build:packages && node scripts/pr-safety-eval.mjs", "eval:agent-efficacy": "pnpm build:packages && node scripts/agent-efficacy-eval.mjs", "test:human-uat-smoke": "pnpm build:packages && node scripts/human-uat-smoke.mjs", + "uat:validate-result": "node scripts/human-uat-validate-result.mjs", + "uat:summarize": "node scripts/human-uat-summarize.mjs", "gen:launch": "node scripts/gen-launch.mjs", "judge-lab:evidence": "pnpm build:packages && node scripts/generate-judge-lab-evidence.mjs", "judge-lab:evidence:check": "node scripts/generate-judge-lab-evidence.mjs --check", diff --git a/packages/knowledge/test/human-uat-kit.test.ts b/packages/knowledge/test/human-uat-kit.test.ts index 3c9458b..e346f58 100644 --- a/packages/knowledge/test/human-uat-kit.test.ts +++ b/packages/knowledge/test/human-uat-kit.test.ts @@ -22,10 +22,15 @@ describe("human UAT kit scaffolding (#692)", () => { "observer-rubric.md", "consent-privacy.md", "facilitator-runbook.md", + "outreach.md", + "session-checklist.md", "fixtures.md", "tasks.json", "result.schema.json", - "summary.template.md" + "summary.template.md", + "result-templates/ai-assisted-individual.template.json", + "result-templates/experienced-engineer.template.json", + "result-templates/team-devops.template.json" ]) { expect(existsSync(join(kitRoot, file))).toBe(true); } @@ -33,7 +38,15 @@ describe("human UAT kit scaffolding (#692)", () => { required: string[]; }; expect(schema.required).toEqual( - expect.arrayContaining(["schemaVersion", "participantRole", "tasks", "trustComprehension"]) + expect.arrayContaining([ + "schemaVersion", + "participantRole", + "tasks", + "trustComprehension", + "friction", + "humanEvidence", + "packageIdentity" + ]) ); expect(readFileSync(join(kitRoot, "README.md"), "utf8")).toMatch( /cannot\s+finish\s+without\s+independent\s+human|cannot\s+close\s+without\s+independent\s+human/i @@ -50,4 +63,12 @@ describe("human UAT kit scaffolding (#692)", () => { expect(tasks.humanEvidence).toBe(false); expect(tasks.tasks.map((task) => task.id)).toEqual(expectedTaskIds); }); + + it("keeps blank templates non-evidential until a real session fills them", () => { + const template = JSON.parse( + readFileSync(join(kitRoot, "result-templates/ai-assisted-individual.template.json"), "utf8") + ) as { humanEvidence: boolean; observerNotes: string }; + expect(template.humanEvidence).toBe(false); + expect(template.observerNotes).toMatch(/TEMPLATE/i); + }); }); diff --git a/scripts/fixtures/human-uat/sample-results/README.md b/scripts/fixtures/human-uat/sample-results/README.md new file mode 100644 index 0000000..598d8d4 --- /dev/null +++ b/scripts/fixtures/human-uat/sample-results/README.md @@ -0,0 +1,5 @@ +# Synthetic sample results + +These JSON files exist only to prove `human-uat-validate-result` and `human-uat-summarize` work in CI. + +**Not independent human evidence. Do not use to finish issue 692.** diff --git a/scripts/fixtures/human-uat/sample-results/ai-assisted-individual.synthetic.json b/scripts/fixtures/human-uat/sample-results/ai-assisted-individual.synthetic.json new file mode 100644 index 0000000..8031d1a --- /dev/null +++ b/scripts/fixtures/human-uat/sample-results/ai-assisted-individual.synthetic.json @@ -0,0 +1,116 @@ +{ + "schemaVersion": 1, + "kitVersion": "uat-kit-2026-08", + "humanEvidence": true, + "participantRole": "ai-assisted-individual", + "environment": "synthetic-ci-a", + "sessionDate": "2026-08-06", + "packageIdentity": { + "source": "packed-tarball", + "version": "0.4.1-synthetic" + }, + "tasks": [ + { + "id": "UAT-HUMAN-1", + "completed": true, + "attempts": 1, + "timeSeconds": 61, + "clarificationRequests": 0, + "unsafeActions": 0, + "commandFailures": 0, + "notes": "synthetic" + }, + { + "id": "UAT-HUMAN-2", + "completed": true, + "attempts": 1, + "timeSeconds": 62, + "clarificationRequests": 1, + "unsafeActions": 0, + "commandFailures": 0, + "notes": "synthetic" + }, + { + "id": "UAT-HUMAN-3", + "completed": true, + "attempts": 1, + "timeSeconds": 63, + "clarificationRequests": 0, + "unsafeActions": 0, + "commandFailures": 0, + "notes": "synthetic" + }, + { + "id": "UAT-HUMAN-4", + "completed": true, + "attempts": 1, + "timeSeconds": 64, + "clarificationRequests": 0, + "unsafeActions": 0, + "commandFailures": 0, + "notes": "synthetic" + }, + { + "id": "UAT-HUMAN-5", + "completed": true, + "attempts": 1, + "timeSeconds": 65, + "clarificationRequests": 0, + "unsafeActions": 0, + "commandFailures": 0, + "notes": "synthetic" + }, + { + "id": "UAT-HUMAN-6", + "completed": true, + "attempts": 1, + "timeSeconds": 66, + "clarificationRequests": 0, + "unsafeActions": 0, + "commandFailures": 0, + "notes": "synthetic" + }, + { + "id": "UAT-HUMAN-7", + "completed": true, + "attempts": 1, + "timeSeconds": 67, + "clarificationRequests": 0, + "unsafeActions": 0, + "commandFailures": 0, + "notes": "synthetic" + }, + { + "id": "UAT-HUMAN-8", + "completed": true, + "attempts": 1, + "timeSeconds": 68, + "clarificationRequests": 0, + "unsafeActions": 0, + "commandFailures": 0, + "notes": "synthetic" + } + ], + "trustComprehension": { + "deterministicEvidence": true, + "runtimeToolProof": true, + "memory": true, + "aiSuggestion": true, + "unverified": true, + "needsHuman": true, + "verified": true + }, + "mistakenTrustInterpretation": false, + "friction": { + "install": "none", + "authenticationProvider": "none", + "packageManager": "none", + "terminal": "none", + "mcp": "none", + "documentation": "none", + "analysisQuality": "none" + }, + "abandonment": false, + "linkedIssues": [], + "observerNotes": "SYNTHETIC FIXTURE FOR VALIDATOR/SUMMARIZE TESTS ONLY \u2014 not a real participant." +} diff --git a/scripts/fixtures/human-uat/sample-results/experienced-engineer.synthetic.json b/scripts/fixtures/human-uat/sample-results/experienced-engineer.synthetic.json new file mode 100644 index 0000000..a985619 --- /dev/null +++ b/scripts/fixtures/human-uat/sample-results/experienced-engineer.synthetic.json @@ -0,0 +1,116 @@ +{ + "schemaVersion": 1, + "kitVersion": "uat-kit-2026-08", + "humanEvidence": true, + "participantRole": "experienced-engineer", + "environment": "synthetic-ci-b", + "sessionDate": "2026-08-06", + "packageIdentity": { + "source": "packed-tarball", + "version": "0.4.1-synthetic" + }, + "tasks": [ + { + "id": "UAT-HUMAN-1", + "completed": true, + "attempts": 1, + "timeSeconds": 61, + "clarificationRequests": 0, + "unsafeActions": 0, + "commandFailures": 0, + "notes": "synthetic" + }, + { + "id": "UAT-HUMAN-2", + "completed": true, + "attempts": 1, + "timeSeconds": 62, + "clarificationRequests": 1, + "unsafeActions": 0, + "commandFailures": 0, + "notes": "synthetic" + }, + { + "id": "UAT-HUMAN-3", + "completed": true, + "attempts": 1, + "timeSeconds": 63, + "clarificationRequests": 0, + "unsafeActions": 0, + "commandFailures": 0, + "notes": "synthetic" + }, + { + "id": "UAT-HUMAN-4", + "completed": true, + "attempts": 1, + "timeSeconds": 64, + "clarificationRequests": 0, + "unsafeActions": 0, + "commandFailures": 0, + "notes": "synthetic" + }, + { + "id": "UAT-HUMAN-5", + "completed": true, + "attempts": 1, + "timeSeconds": 65, + "clarificationRequests": 0, + "unsafeActions": 0, + "commandFailures": 0, + "notes": "synthetic" + }, + { + "id": "UAT-HUMAN-6", + "completed": true, + "attempts": 1, + "timeSeconds": 66, + "clarificationRequests": 0, + "unsafeActions": 0, + "commandFailures": 0, + "notes": "synthetic" + }, + { + "id": "UAT-HUMAN-7", + "completed": true, + "attempts": 1, + "timeSeconds": 67, + "clarificationRequests": 0, + "unsafeActions": 0, + "commandFailures": 0, + "notes": "synthetic" + }, + { + "id": "UAT-HUMAN-8", + "completed": true, + "attempts": 1, + "timeSeconds": 68, + "clarificationRequests": 0, + "unsafeActions": 0, + "commandFailures": 0, + "notes": "synthetic" + } + ], + "trustComprehension": { + "deterministicEvidence": true, + "runtimeToolProof": true, + "memory": true, + "aiSuggestion": true, + "unverified": true, + "needsHuman": true, + "verified": true + }, + "mistakenTrustInterpretation": false, + "friction": { + "install": "none", + "authenticationProvider": "none", + "packageManager": "none", + "terminal": "none", + "mcp": "none", + "documentation": "none", + "analysisQuality": "none" + }, + "abandonment": false, + "linkedIssues": [], + "observerNotes": "SYNTHETIC FIXTURE FOR VALIDATOR/SUMMARIZE TESTS ONLY \u2014 not a real participant." +} diff --git a/scripts/fixtures/human-uat/sample-results/team-devops.synthetic.json b/scripts/fixtures/human-uat/sample-results/team-devops.synthetic.json new file mode 100644 index 0000000..b19a01d --- /dev/null +++ b/scripts/fixtures/human-uat/sample-results/team-devops.synthetic.json @@ -0,0 +1,116 @@ +{ + "schemaVersion": 1, + "kitVersion": "uat-kit-2026-08", + "humanEvidence": true, + "participantRole": "team-devops", + "environment": "synthetic-ci-c", + "sessionDate": "2026-08-06", + "packageIdentity": { + "source": "packed-tarball", + "version": "0.4.1-synthetic" + }, + "tasks": [ + { + "id": "UAT-HUMAN-1", + "completed": true, + "attempts": 1, + "timeSeconds": 61, + "clarificationRequests": 0, + "unsafeActions": 0, + "commandFailures": 0, + "notes": "synthetic" + }, + { + "id": "UAT-HUMAN-2", + "completed": true, + "attempts": 1, + "timeSeconds": 62, + "clarificationRequests": 1, + "unsafeActions": 0, + "commandFailures": 0, + "notes": "synthetic" + }, + { + "id": "UAT-HUMAN-3", + "completed": true, + "attempts": 1, + "timeSeconds": 63, + "clarificationRequests": 0, + "unsafeActions": 0, + "commandFailures": 0, + "notes": "synthetic" + }, + { + "id": "UAT-HUMAN-4", + "completed": true, + "attempts": 1, + "timeSeconds": 64, + "clarificationRequests": 0, + "unsafeActions": 0, + "commandFailures": 0, + "notes": "synthetic" + }, + { + "id": "UAT-HUMAN-5", + "completed": true, + "attempts": 1, + "timeSeconds": 65, + "clarificationRequests": 0, + "unsafeActions": 0, + "commandFailures": 0, + "notes": "synthetic" + }, + { + "id": "UAT-HUMAN-6", + "completed": true, + "attempts": 1, + "timeSeconds": 66, + "clarificationRequests": 0, + "unsafeActions": 0, + "commandFailures": 0, + "notes": "synthetic" + }, + { + "id": "UAT-HUMAN-7", + "completed": true, + "attempts": 1, + "timeSeconds": 67, + "clarificationRequests": 0, + "unsafeActions": 0, + "commandFailures": 0, + "notes": "synthetic" + }, + { + "id": "UAT-HUMAN-8", + "completed": true, + "attempts": 1, + "timeSeconds": 68, + "clarificationRequests": 0, + "unsafeActions": 0, + "commandFailures": 0, + "notes": "synthetic" + } + ], + "trustComprehension": { + "deterministicEvidence": true, + "runtimeToolProof": true, + "memory": true, + "aiSuggestion": true, + "unverified": true, + "needsHuman": true, + "verified": true + }, + "mistakenTrustInterpretation": false, + "friction": { + "install": "none", + "authenticationProvider": "none", + "packageManager": "none", + "terminal": "none", + "mcp": "none", + "documentation": "none", + "analysisQuality": "none" + }, + "abandonment": false, + "linkedIssues": [], + "observerNotes": "SYNTHETIC FIXTURE FOR VALIDATOR/SUMMARIZE TESTS ONLY \u2014 not a real participant." +} diff --git a/scripts/human-uat-smoke.mjs b/scripts/human-uat-smoke.mjs index f75233a..5ff7b6b 100644 --- a/scripts/human-uat-smoke.mjs +++ b/scripts/human-uat-smoke.mjs @@ -144,6 +144,8 @@ try { assert(/preflight|ai/i.test(aiHelp.stdout), "ai workflow help must be discoverable"); checks.push({ id: "UAT-HUMAN-1-discover", ok: true, detail: "codedecay ai --help ok" }); + checkResultTooling(checks, runDir); + report.status = "passed"; report.checks = checks; report.finishedAt = new Date().toISOString(); @@ -181,8 +183,13 @@ function checkKitFiles(checks) { "result.schema.json", "summary.template.md", "facilitator-runbook.md", + "outreach.md", + "session-checklist.md", "tasks.json", - "fixtures.md" + "fixtures.md", + "result-templates/ai-assisted-individual.template.json", + "result-templates/experienced-engineer.template.json", + "result-templates/team-devops.template.json" ]) { const path = join(kitRoot, file); assert(existsSync(path), `missing kit file: ${file}`); @@ -190,6 +197,66 @@ function checkKitFiles(checks) { checks.push({ id: "kit-files", ok: true, detail: "versioned kit files present" }); } +function checkResultTooling(checks, runDir) { + const sampleDir = join(repoRoot, "scripts/fixtures/human-uat/sample-results"); + const samples = [ + "ai-assisted-individual.synthetic.json", + "experienced-engineer.synthetic.json", + "team-devops.synthetic.json" + ].map((name) => join(sampleDir, name)); + for (const sample of samples) { + assert(existsSync(sample), `missing synthetic sample ${sample}`); + const sampleJson = JSON.parse(readFileSync(sample, "utf8")); + assert( + /SYNTHETIC FIXTURE/i.test(String(sampleJson.observerNotes ?? "")), + "synthetic samples must be labeled as non-human evidence" + ); + } + + const template = join(kitRoot, "result-templates/ai-assisted-individual.template.json"); + const templateValidate = runCommand( + process.execPath, + [join(repoRoot, "scripts/human-uat-validate-result.mjs"), template], + { cwd: repoRoot, timeoutMs: 15_000 } + ); + record("template-rejects-until-filled", templateValidate, checks, (result) => { + if (result.exitCode === 0) { + throw new Error("Blank result templates must fail validation until humanEvidence=true and fields are filled."); + } + }); + + const validate = runCommand( + process.execPath, + [join(repoRoot, "scripts/human-uat-validate-result.mjs"), ...samples], + { cwd: repoRoot, timeoutMs: 15_000 } + ); + record("synthetic-results-validate", validate, checks, (result) => { + if (result.exitCode !== 0) { + throw new Error(`synthetic sample validation failed: ${result.stderr || result.stdout}`); + } + }); + + const summaryMd = join(runDir, "synthetic-summary.md"); + const summarize = runCommand( + process.execPath, + [join(repoRoot, "scripts/human-uat-summarize.mjs"), "--out", summaryMd, ...samples], + { cwd: repoRoot, timeoutMs: 15_000 } + ); + record("synthetic-summarize", summarize, checks, (result) => { + if (result.exitCode !== 0) { + throw new Error(`summarize failed: ${result.stderr || result.stdout}`); + } + const markdown = readFileSync(summaryMd, "utf8"); + assert(/Decision: pass/i.test(markdown), "synthetic three-role summary should decide pass"); + assert(/not deterministic smoke output/i.test(markdown), "summary must distinguish from smoke"); + }); + checks.push({ + id: "result-tooling", + ok: true, + detail: "templates reject; synthetic samples validate+summarize (not human evidence)" + }); +} + function checkTasksManifest(checks) { const tasks = JSON.parse(readFileSync(join(kitRoot, "tasks.json"), "utf8")); assert(tasks.schemaVersion === 1, "tasks.json schemaVersion must be 1"); diff --git a/scripts/human-uat-summarize.mjs b/scripts/human-uat-summarize.mjs new file mode 100644 index 0000000..7953ad8 --- /dev/null +++ b/scripts/human-uat-summarize.mjs @@ -0,0 +1,154 @@ +#!/usr/bin/env node +/** + * Build an anonymized human-UAT summary from validated participant result JSON files. + */ +import { mkdirSync, readFileSync, writeFileSync } from "node:fs"; +import { dirname, resolve } from "node:path"; +import { fileURLToPath } from "node:url"; +import { spawnSync } from "node:child_process"; + +const repoRoot = resolve(fileURLToPath(new URL("..", import.meta.url))); +const options = parseArgs(process.argv.slice(2)); + +if (options.help || options.inputs.length === 0) { + process.stdout.write( + [ + "Usage: node scripts/human-uat-summarize.mjs --out [...]", + "", + "Validates each result, then writes an anonymized Markdown summary.", + "Does not fabricate participant outcomes.", + "" + ].join("\n") + ); + process.exit(options.help ? 0 : 1); +} + +const validator = resolve(repoRoot, "scripts/human-uat-validate-result.mjs"); +const validate = spawnSync(process.execPath, [validator, ...options.inputs], { + encoding: "utf8" +}); +if (validate.status !== 0) { + process.stderr.write(validate.stdout || ""); + process.stderr.write(validate.stderr || ""); + process.exit(validate.status ?? 1); +} + +const results = options.inputs.map((file) => JSON.parse(readFileSync(resolve(process.cwd(), file), "utf8"))); +const taskIds = [ + "UAT-HUMAN-1", + "UAT-HUMAN-2", + "UAT-HUMAN-3", + "UAT-HUMAN-4", + "UAT-HUMAN-5", + "UAT-HUMAN-6", + "UAT-HUMAN-7", + "UAT-HUMAN-8" +]; + +const roleCounts = Object.fromEntries( + ["ai-assisted-individual", "experienced-engineer", "team-devops"].map((role) => [ + role, + results.filter((row) => row.participantRole === role).length + ]) +); +const completedByTask = Object.fromEntries( + taskIds.map((id) => [ + id, + results.filter((row) => row.tasks.some((task) => task.id === id && task.completed)).length + ]) +); +const trustMisses = results.flatMap((row, index) => { + const misses = Object.entries(row.trustComprehension) + .filter(([, ok]) => ok === false) + .map(([key]) => key); + return misses.length === 0 ? [] : [`P${index + 1} (${row.participantRole}): ${misses.join(", ")}`]; +}); +const blockingFriction = results.flatMap((row, index) => { + const blocked = Object.entries(row.friction) + .filter(([, level]) => level === "blocking") + .map(([key]) => key); + return blocked.length === 0 ? [] : [`P${index + 1} (${row.participantRole}): ${blocked.join(", ")}`]; +}); +const linkedIssues = [...new Set(results.flatMap((row) => row.linkedIssues ?? []))]; +const anyAbandonment = results.some((row) => row.abandonment); +const anyMistakenTrust = results.some((row) => row.mistakenTrustInterpretation); +const allTasksComplete = taskIds.every((id) => completedByTask[id] === results.length); +const decision = + !anyAbandonment && !anyMistakenTrust && results.length >= 3 && allTasksComplete ? "pass" : "fail"; + +const markdown = [ + "# Human UAT summary", + "", + `- Kit version: ${unique(results.map((row) => row.kitVersion)).join(", ") || "n/a"}`, + `- Date: ${unique(results.map((row) => row.sessionDate).filter(Boolean)).join(", ") || new Date().toISOString().slice(0, 10)}`, + `- Participants (anonymized roles only): ${results.length} total — ai-assisted=${roleCounts["ai-assisted-individual"]}, experienced=${roleCounts["experienced-engineer"]}, devops=${roleCounts["team-devops"]}`, + `- Package identity: ${unique(results.map((row) => `${row.packageIdentity.source}@${row.packageIdentity.version}`)).join(", ")}`, + `- Success threshold: ≥3 independent roles; all UAT-HUMAN-1..8 completed; no mistaken trust; no abandonment`, + `- Tasks passed / failed: ${taskIds.map((id) => `${id} ${completedByTask[id]}/${results.length}`).join("; ")}`, + `- Trust comprehension misses: ${trustMisses.length === 0 ? "none" : trustMisses.join("; ")}`, + `- Installation / auth / docs friction (separate from analysis quality): ${ + blockingFriction.length === 0 ? "no blocking friction recorded" : blockingFriction.join("; ") + }`, + `- Linked blocker issues: ${linkedIssues.length === 0 ? "none" : linkedIssues.join(", ")}`, + `- Decision: ${decision} for milestone gate`, + "", + "This summary was generated from validated participant JSON. It is not deterministic smoke output.", + "" +]; + +const outPath = resolve(process.cwd(), options.out); +mkdirSync(dirname(outPath), { recursive: true }); +writeFileSync(outPath, markdown.join("\n"), "utf8"); + +const machinePath = options.jsonOut + ? resolve(process.cwd(), options.jsonOut) + : outPath.replace(/\.md$/i, ".json"); +writeFileSync( + machinePath, + `${JSON.stringify( + { + schemaVersion: 1, + humanEvidence: true, + decision, + participantCount: results.length, + roleCounts, + completedByTask, + linkedIssues, + generatedAt: new Date().toISOString() + }, + null, + 2 + )}\n`, + "utf8" +); + +process.stdout.write(`wrote ${outPath}\nwrote ${machinePath}\ndecision=${decision}\n`); + +function unique(values) { + return [...new Set(values)]; +} + +function parseArgs(argv) { + const normalized = argv[0] === "--" ? argv.slice(1) : argv; + const parsed = { out: "human-uat-summary.md", jsonOut: undefined, help: false, inputs: [] }; + for (let index = 0; index < normalized.length; index += 1) { + const arg = normalized[index]; + if (arg === "--help" || arg === "-h") { + parsed.help = true; + continue; + } + if (arg === "--out") { + parsed.out = normalized[++index]; + continue; + } + if (arg === "--json-out") { + parsed.jsonOut = normalized[++index]; + continue; + } + if (arg.startsWith("-")) { + throw new Error(`Unknown option: ${arg}`); + } + parsed.inputs.push(arg); + } + return parsed; +} diff --git a/scripts/human-uat-validate-result.mjs b/scripts/human-uat-validate-result.mjs new file mode 100644 index 0000000..ce61ed4 --- /dev/null +++ b/scripts/human-uat-validate-result.mjs @@ -0,0 +1,170 @@ +#!/usr/bin/env node +/** + * Validate a human UAT participant result against docs/evals/uat-kit/result.schema.json. + * Does not invent participant outcomes. + */ +import { readFileSync } from "node:fs"; +import { resolve } from "node:path"; +import { fileURLToPath } from "node:url"; + +const repoRoot = resolve(fileURLToPath(new URL("..", import.meta.url))); +const schemaPath = resolve(repoRoot, "docs/evals/uat-kit/result.schema.json"); +const args = process.argv.slice(2).filter((arg) => arg !== "--"); + +if (args.includes("--help") || args.includes("-h") || args.length === 0) { + process.stdout.write( + [ + "Usage: node scripts/human-uat-validate-result.mjs [...]", + "", + "Validates participant result files against the kit schema.", + "Blank templates under docs/evals/uat-kit/result-templates/ intentionally fail until filled.", + "" + ].join("\n") + ); + process.exit(args.length === 0 ? 1 : 0); +} + +const schema = JSON.parse(readFileSync(schemaPath, "utf8")); +const expectedTaskIds = [ + "UAT-HUMAN-1", + "UAT-HUMAN-2", + "UAT-HUMAN-3", + "UAT-HUMAN-4", + "UAT-HUMAN-5", + "UAT-HUMAN-6", + "UAT-HUMAN-7", + "UAT-HUMAN-8" +]; +const frictionKeys = [ + "install", + "authenticationProvider", + "packageManager", + "terminal", + "mcp", + "documentation", + "analysisQuality" +]; +const trustKeys = [ + "deterministicEvidence", + "runtimeToolProof", + "memory", + "aiSuggestion", + "unverified", + "needsHuman", + "verified" +]; + +let failed = 0; +for (const file of args) { + const path = resolve(process.cwd(), file); + const errors = validateResult(JSON.parse(readFileSync(path, "utf8"))); + if (errors.length === 0) { + process.stdout.write(`ok ${path}\n`); + continue; + } + failed += 1; + process.stderr.write(`fail ${path}\n`); + for (const error of errors) { + process.stderr.write(` - ${error}\n`); + } +} + +process.exitCode = failed === 0 ? 0 : 1; + +function validateResult(value) { + const errors = []; + if (!value || typeof value !== "object" || Array.isArray(value)) { + return ["root must be an object"]; + } + + if (value.schemaVersion !== schema.properties.schemaVersion.const) { + errors.push(`schemaVersion must be ${schema.properties.schemaVersion.const}`); + } + if (value.humanEvidence !== true) { + errors.push("humanEvidence must be true for real participant records"); + } + if (typeof value.kitVersion !== "string" || value.kitVersion.trim() === "") { + errors.push("kitVersion is required"); + } + if (!["ai-assisted-individual", "experienced-engineer", "team-devops"].includes(value.participantRole)) { + errors.push("participantRole is invalid"); + } + if (typeof value.abandonment !== "boolean") { + errors.push("abandonment must be boolean"); + } + if (typeof value.mistakenTrustInterpretation !== "boolean") { + errors.push("mistakenTrustInterpretation must be boolean"); + } + if (value.mistakenTrustInterpretation === true) { + errors.push("session fails kit gate when mistakenTrustInterpretation is true"); + } + + const packageIdentity = value.packageIdentity; + if (!packageIdentity || typeof packageIdentity !== "object") { + errors.push("packageIdentity is required"); + } else { + if (!["npm", "packed-tarball"].includes(packageIdentity.source)) { + errors.push("packageIdentity.source must be npm or packed-tarball"); + } + if (typeof packageIdentity.version !== "string" || packageIdentity.version.trim() === "") { + errors.push("packageIdentity.version is required"); + } + } + + if (!Array.isArray(value.tasks) || value.tasks.length < 8) { + errors.push("tasks must include all eight UAT-HUMAN entries"); + } else { + const ids = value.tasks.map((task) => task?.id); + for (const id of expectedTaskIds) { + if (!ids.includes(id)) errors.push(`missing task ${id}`); + } + for (const [index, task] of value.tasks.entries()) { + if (!task || typeof task !== "object") { + errors.push(`tasks[${index}] must be an object`); + continue; + } + if (typeof task.completed !== "boolean") errors.push(`tasks[${index}].completed must be boolean`); + if (!Number.isInteger(task.attempts) || task.attempts < 0) { + errors.push(`tasks[${index}].attempts must be a non-negative integer`); + } + if (typeof task.timeSeconds !== "number" || task.timeSeconds < 0) { + errors.push(`tasks[${index}].timeSeconds must be a non-negative number`); + } + for (const optionalInt of ["clarificationRequests", "unsafeActions", "commandFailures"]) { + if (task[optionalInt] !== undefined && (!Number.isInteger(task[optionalInt]) || task[optionalInt] < 0)) { + errors.push(`tasks[${index}].${optionalInt} must be a non-negative integer when set`); + } + } + } + } + + const trust = value.trustComprehension; + if (!trust || typeof trust !== "object") { + errors.push("trustComprehension is required"); + } else { + for (const key of trustKeys) { + if (typeof trust[key] !== "boolean") { + errors.push(`trustComprehension.${key} must be boolean`); + } + } + } + + const friction = value.friction; + if (!friction || typeof friction !== "object") { + errors.push("friction is required"); + } else { + for (const key of frictionKeys) { + if (!["none", "minor", "blocking"].includes(friction[key])) { + errors.push(`friction.${key} must be none|minor|blocking`); + } + } + } + + if (value.linkedIssues !== undefined) { + if (!Array.isArray(value.linkedIssues) || value.linkedIssues.some((item) => typeof item !== "string")) { + errors.push("linkedIssues must be an array of strings"); + } + } + + return errors; +}