Method and scope

Method and scope

Research question

When a business document becomes a development specification, do its obligations, permissions, prohibitions, owners, conditions and exceptions survive? How much active human correction is needed before the specification can be accepted?

A source saying that a manager may approve can become an output saying they must approve. This illustrative change can coexist with a valid source citation and valid JSON. Citation existence, schema compliance and semantic fidelity therefore need separate evidence.

Unit of comparison

The unit is a frozen system configuration: model revision, provider/runtime, reasoning and generation settings, prompt, adapter, parser/retriever where applicable, tools, context and corpus. Different settings cannot be treated as a clean model-only comparison.

Track	Boundary
Interpretation	Human-verified source units to annotated propositions
Document to blueprint	Original PDF/XLSX through reviewed export
Control conformance	Faults, boundaries and actual permitted effects

First proposed study

Procurement is the initial domain. The 150-200 proposition target is a planning estimate, not a statistical sufficiency guarantee. Two blinded domain annotators and adjudication are proposed before inspecting system outputs. Independent source families and valid alternative interpretations must be recorded. Repeating a task three times does not create three independent families.

The three original repositories above 100 MB are development cases. File size alone is not a measure of reasoning difficulty, and these cases cannot become unseen tests by relabelling them.

Study RFC | Full protocol | Qualification and roadmap