Add Ranjan2113/doctask-ranjan-jamnis-review-eval — a human review-cost evaluation harness for AI-proposed edits

[Ranjan2113]

Ranjan Jamnis

A measurement harness that tests how the presentation of AI-proposed document edits changes human review — decision time, inter-reviewer agreement, and error rate against a pre-registered answer key — across four conditions: batch vs sequential presentation, crossed with section-diff vs whole-document-diff.

Folder: use-cases/Ranjan2113/doctask-ranjan-jamnis-review-eval/

What's in it

A corpus builder that freezes a reproducible set of proposed edits from the SuperDocs API, a FastAPI backend that serves batches and records timestamped decision events, a React reviewer UI implementing all four conditions off one shared timer hook, and an analysis script computing Fleiss' κ, decision-time distributions and accuracy against ground truth.

176 tests (152 Python, 24 frontend). None require an API key or network access — the SuperDocs client is mocked, so the whole suite runs offline.

SuperDocs features used

• POST /v1/chat/async with approval_mode: "ask_every_time"
• GET /v1/jobs/{job_id} polling through the job lifecycle
• Human-in-the-loop metadata.pending_changes capture, used to freeze the corpus without ever applying an edit

Budget: 9 operations against a 15-operation cap declared before any call. $0 — free tier. Every call prints its running total, and the client raises before dispatch rather than after overspending.

On the results — read this before the numbers

This is an N=1 pilot and it does not rank the four conditions. The sole reviewer returned a single verdict throughout, so accuracy and Fleiss' κ are suppressed in the report rather than printed — at this sample neither is a measurement, and the report says so above the numbers, not in a footnote. No panel was recruited; that is the stated next step, and the harness runs a real panel at zero additional SuperDocs cost.

Two instrumentation failures, documented rather than buried

1. Labels were derived from the requested intent instead of the delivered diff. A change pre-registered as APPROVE was in fact a REJECT — the model had quietly raised a contract's total price to keep its arithmetic consistent. Fixed structurally: a label now cannot enter a batch unless it has been confirmed against that change's actual old_html/new_html.
2. The batch-condition timer started at render, not at engagement. It produced a clean, plausible, entirely fictitious 40× gap between conditions. Nine passing tests did not catch it, because the specification itself was wrong.

Both would have yielded quotable, wrong numbers. They're written up in PROGRESS.md, GROUND_TRUTH_SAFETY.md and as dated amendments to the pre-registered PROTOCOL.md.

Findings about the API worth keeping

Explanations become generic when edits are batched; changes arrive chunk-level rather than intent-level, so one instruction can fan out into several; no-op edits get proposed and need filtering; section references by title succeed where "Clause N" returns nothing; and the model declined to delete a protective clause when asked to.

Built for the SuperDocs task. No secrets in the tree — .env is gitignored, and job UUIDs in the published study data are redacted to stable placeholders.