The Evidence Standard

The evidence standard

The README says the bias is toward what survived production. This is how that is judged.

---

Why this page exists

Every other page here makes claims. This one says what a claim has to clear before it gets made, which is the only reason to believe any of the others.

The repository states its provenance on the front page: hundreds of case studies and production implementations reviewed, dozens of practitioner interviews, direct delivery. That is a volume claim. Volume is not evidence, and reviewing a large number of bad case studies produces a confident survey of nothing.

So there is a filter, and this is it.

The rule, in one sentence

A number is publishable when someone else could re-derive it from a source you name.

That is stricter than "cited" and looser than "peer-reviewed". An analyst press release is a source you can name. It is not a source anyone can re-derive, so a figure resting on one gets reported as an attributed claim rather than as a finding.

Everything on these pages that fails the rule is either dropped or recorded. A figure that could not be re-derived from a named source is not on these pages, and what every page rests on, page by page, is set out in what is not verified.

That is one page rather than a footer on twenty-five, because a caveat repeated on every page stops being read on the second one.

Phase 1 · The wilderness test

Most AI fails because it was demonstrated against a sanitised mirage.

The messy reality audit. If the demonstration uses clean exports and digital documents, it has not been tested. The question is how it handles handwriting that overlaps the box, tables that are rotated, and the knowledge debt of four versions of the same document disagreeing on a shared drive.

One example worth keeping without its name: the most convincing system in this category ingested decades-old digitised training video, multilingual manuals, and the handwritten notebooks of retiring engineers. That is the standard. Clean CSVs are not a data pipeline, they are a photograph of one.

The contamination check. Models often score well because they have seen the test. Ask for an evaluation against a private benchmark. If performance drops materially on data the model could not have encountered publicly, you are looking at a memorisation engine rather than a reasoning one.

The self-consumption check. Is the model being trained on its own output? Model collapse is a documented and terminal risk. If no human curation sits in that loop, the system decays on a schedule nobody set.

Phase 2 · Useful versus smart

A smart model can write a poem about your invoice. A useful model extracts the line item accurately and flags the one-cent discrepancy.

The last-five-percent question. You can reach most of the way to accuracy with a prompt. The remainder costs an order of magnitude more and requires expert time.

Ask: what is the specific recovery protocol for the failure rate?

If the answer is that the model does not have one, the conversation is over.

Deterministic backbones. Credible systems use the model for fuzzy extraction and keep business logic in deterministic code. Tax, entitlement, eligibility and pricing rules do not belong in a probability distribution. The best example in the source encoded its governance rules, uniqueness and hierarchy limits, in ordinary code and used the model only for the parts that needed language.

The latency-to-value ratio. If a thirty-second wait for a better answer does not save meaningfully more than thirty seconds of human work, the return is negative and no amount of accuracy fixes it.

Phase 3 · Source triage

Not all signal is equal. Claims get weighted by who is making them.

Source	Reliability	What to watch for
Academic, peer-reviewed	High	Often silent on real-world latency and cost
Open-source community	Medium-high	"It works on my machine"
Executive or sales	Low	Seamless. Magic. Infinite.
Engineering or operations lead	Critical	Nothing. They volunteer failure modes and infrastructure costs, which is the tell

The bottom row is the whole matrix. Someone who opens with what broke is describing a system they ran. Someone who opens with what it enables is describing a system they sold.

The interviews behind this repository were weighted toward the bottom row. The people are not named here: they spoke for a specific piece of work, and naming them on a public page is a permission question rather than a citation question.

Phase 4 · The security and privacy trap

A system that passes the first three phases can still be unpublishable.

The questions are ordinary and rarely asked in a demonstration: what leaves the building, where does it go, on what legal basis, who else can reach it, and what happens to it afterwards. An architecture that works and cannot answer those is not a candidate, and finding that out at procurement is expensive.

Confidence, labelled

Not every useful number clears the bar. The answer is to label rather than to discard.

Label	What it means	How it may be used
Verifiable	Traceable to a named source anyone can retrieve and check	State it as a fact, with the citation
Directional	Reported consistently by vendors, associations or consultancies, with a shared interest in the answer	State it as a range, say who reports it, and say it is directional
Composite	Assembled from a pattern that recurs, not from a single case	Label it a composite on the page, every time
Illustrative	An example number chosen to show a shape	Mark it as illustrative, or remove it
Retired	Found to be wrong after publication	Correct it visibly and keep the record of the correction

The failure this prevents is the quiet promotion, where a directional figure loses its qualifier in the second retelling and becomes a fact in the third. Every framework here has drifted that way at least once.

Corrections, in public

Being wrong in public is a cost of publishing. Hiding it is a different and larger one.

The practice is: correct it visibly, keep the original visible, and record what caused it. A page that quietly changes a number teaches its readers nothing. A page that says this said 1.7 times, the source did not support it, here is what happened teaches them how to read the rest.

Every correction this repository has had to make is on one page: Corrections. A spine that named nine frameworks and defined three. A front page that advertised five tools where two other pages said seven. A template placeholder that shipped, unfilled, inside a public clone command.

None of those were found by a reader. All of them were found by checking, which is the argument for checking.

What this standard costs

It is not free and it is worth saying so.

It removes the most quotable material, because the most quotable material is usually a named company attached to a large number through a citation that does not resolve. Roughly two dozen such cases were available for these pages and none of them are here.

It makes the writing more hedged. Some of that is honest and some of it is just less confident prose, and the line between the two is a judgement made page by page.

What it buys is the only thing that matters at this altitude: a reader who checks one claim and finds it holds has a reason to believe the next one.

_{From the practice canon. Developed through client delivery and refined in collaboration; the frameworks stated here are the author's own. Practitioner sources are deliberately unnamed.}

_{Back to Home · The framework spine · Reading list}