Repository navigation
Home
11 November audit notice: This is a research programme with zero qualified results. Evaluator 0.2 has a confirmed non-claim exclusion flaw and must not be used for qualification. Historical model records are adapter smoke tests on 4,994 bytes of document text. See the audit response.
Specification fidelity: preserving business meaning when AI turns documents into development blueprints.
This handbook explains the research question, the executable evaluator and how to participate. The project is a public research pilot. There are zero qualified held-out results and no assigned PALO Blueprint Index score.
| I want to... | Start here |
|---|---|
| Understand the benchmark | [[Method and scope |
| Run the synthetic evaluator | [[Quick start |
| Interpret the charts | [[Explorer and results |
| Inspect metrics and denominators | [[Metrics |
| Audit the evidence contract | [[Evidence contract |
| Reproduce the archived checks | [[Reproducing results |
| Contribute expertise or cases | [[Contributing |
| Understand qualification gates | [[Qualification and roadmap |
| Compare other benchmarks | [[Benchmark references |
| Find a short answer | [[FAQ |
The evaluator has an executable JSON Schema, raw-output binding, annotation-consistency checks and calculated task acceptance. Evaluator 0.3.0 is distributed in internal prerelease v0.3.0-internal.1. Historical releases retain their original test receipts; consult the new release verification for current checks. These establish bounded software behaviour, not semantic validity or an independent ranking.
Live explorer | Research repository | Release v0.3.0-pilot.1
The wiki source is versioned in wiki/. The repository's executable schema and versioned protocol remain the authoritative technical references. Wiki edits must be reconciled with that source before the next sync.
The unified catalogue contains 18 model IDs, with 28 cohort/configuration records. The procurement campaigns have 324 assigned slots, 306 dispatched requests, 299 complete responses, 7 failed or incomplete responses and 18 pre-call blocks. These are internal controls with assistant-draft labels, not independent validation.
Internal test results · Project identity and release
Inspect source cases, workflow drafts and internal observations. This separate cycle preserves the original cohort and evaluator release. Automated advisory reviews of P002/P013/P036 do not substitute for human adjudication.