Three Proof Gates

The three proof gates

Programmes die in one of two places: before they start, or after the pilot.

---

Death by committee, death by scale

Before they start: the business case circulates for five months, acquires seven reviewers, and quietly stops moving. Nobody killed it. It just never got a decision.

After the pilot: the demo worked, everyone applauded, and then it met real volume, real data, real users, and a cost line nobody modelled.

The gates exist to make both of those into decisions rather than outcomes. Each one is a deliberate checkpoint with an entry cost, an exit criterion, and a named person who can say no.

The framework spine names the three proofs as dimensions. This page is the gates. Same framework, two altitudes: the spine says what evidence you need, this says how you actually get it and in what order.

Gate 1 · Proof of Value

Does this matter?

Roughly two to four weeks. Minimal investment. No code.

The goal is to establish that the business problem is real, measurable, and worth solving this way. Most of what kills a programme at month nine was knowable in this fortnight.

The checklist:

• A named executive sponsor who owns the KPI this affects. Not informed. Accountable
• A baseline measured, not remembered. Current handling time, error rate, cycle time
• An opportunity score above the threshold
• Regulatory and security review initiated, not deferred
• Data access confirmed in the environment the pilot will actually use
• A failure mode defined: what does "this failed" look like, and who decides to stop
• A conservative value model. Assume well under the projected saving in year one
• A dignity clause review: what happens to the people whose work this affects, with the reskilling path defined before kickoff

Exit: proceed if the case is approved at director level or above, data access is confirmed, and the failure mode is agreed. Otherwise park it or kill it. Do not proceed to build with an open item on this list.

The item everyone skips is data access, and it is the one that ends programmes. If you hit a wall there, that wall is your real first project, not the agent.

Gate 2 · Proof of Competence

Can we build it?

Roughly four to eight weeks. One or two developers, a product owner, a domain expert.

A working prototype against real data in a sandbox. Not a demo. The distinction is that a demo is chosen and a prototype is sampled.

The checklist:

• Processes real, de-identified data, not synthetic inputs
• A golden dataset built by the domain expert, not the developer. Around a hundred cases
• Accuracy and hallucination rate measured against that dataset
• Integration tested against the actual target system, not a mock
• Sensitive-data handling reviewed by the compliance function
• The business metric moved in a controlled test, not just the technical one
• At least one failure mode triggered deliberately and handled gracefully
• User acceptance with an actual operator, not the project team

Exit: proceed if accuracy clears the threshold you set in advance, integration is stable, users accept it, and compliance has signed. Redesign if accuracy is short or integration is unstable. Kill if the problem turns out not to be solvable with the data that exists.

The golden-dataset rule is the load-bearing line in this gate. A test set written by the people who built the system tests whether the system does what it does. Written by the domain expert, it tests whether the system does what the work needs.

Gate 3 · Proof of Technology

Can we run it safely, at scale?

Roughly six to twelve weeks. Full team: engineering, operations, change management, security.

The checklist:

• Shadow mode deployed: runs in parallel, outputs compared, nothing acted on
• Observability live: latency, error rate, hallucination rate, cost per transaction
• Circuit breaker defined: automatic suspension on a threshold breach
• Change communication sent. The affected teams know what is changing and why, before it changes
• Escalation path documented: when it fails, who is alerted, how fast
• Shadow results reviewed before any production traffic
• Canary release to a small share of volume before full rollout
• Rollback plan tested, not just written

The experiment template

Every gate-3 submission states a hypothesis you could be wrong about.

Hypothesis. If we run this agent in shadow mode, it will match human accuracy within a stated tolerance, on a stated input type, within a stated window.

Test. Run it in parallel with named human operators, on a defined volume.

Metric. The acceptance rate, the handling time, the downstream rejection rate.

Learning. Within tolerance, proceed to canary. Outside it, identify the specific failure pattern and redesign. Not "try harder."

The tolerance and the window are set before the run. A tolerance chosen after seeing the result is not a test, it is a description.

The pre-mortem: four ways this fails

Run with the full team before the gate-3 submission, not after the incident.

1 · The data problem

The sandbox data was clean. Production data is not. Extraction failure rates on real documents run far above the estimate, and accuracy collapses on inputs nobody sampled.

Mitigation: shadow mode before any production volume, and source-specific validation rules built before go-live.

2 · The people problem

The team sees the agent as a threat. They stop supplying complete inputs. Accuracy drops, not because the agent broke, but because humans are withholding context.

Mitigation: the dignity clause, in effect and communicated before shadow mode. Bring the domain leads into building the golden dataset so the agent is visibly theirs.

This is the failure that looks exactly like a model problem in the logs, and the one a technical post-mortem will never find.

3 · The silent-failure problem

An upstream system changes its interface. The agent breaks with no error and simply starts being wrong. Nobody notices for weeks. The downstream metric moves, then the exposure follows.

Mitigation: continuous observability on the business outcome, not just the technical one, plus a circuit breaker on the downstream rejection rate.

4 · The cost problem

It works. It scales. Inference cost was underestimated by a large multiple, and the programme is cancelled six months after go-live by someone who never saw a unit-economics number.

Mitigation: a token budget from day one, cost per transaction reported alongside productivity in every update, and unit economics approved by finance before this gate closes.

All four are organisational failures wearing technical clothes, and all four are cheaper to prevent than to diagnose.

_{Adapted from the AI CoE and Agent Factory Playbook, based on The Augmented Enterprise framework. The gate structure and the pre-mortem are the author's own; vertical specifics have been generalised.}

_{Back to Home · The deployment ladder · The framework spine}