Skip to content

North Star Business Case

sergio velderrain edited this page Nov 17, 2026 · 1 revision

North-Star Business Case

This is the public summary of the north star (VITechnologies/projectbuddy#328): one measured lbabus wiki change, an improvement to the standard, and a rerun. Figures are relative. The full case, with absolute figures and the stakeholder deck, is private.

Result

Under standard @4, implementing the change cost 17.6% less AI to acceptance than the mean of two control runs under @2, in 12.6% less wall-clock time. The defect the improvement targeted did not recur. It was missing scope on the test pages of allocated requirements, and it caused every control-run gate resume.

Run Standard AI cost ÷ control mean Wall-clock ÷ control mean Gate resumes
A @2 1.13 1.16 1
A2 @2 0.87 0.84 2
B @3 1.12 1.48 0
B2 @4 0.82 0.87 1

How It Was Measured

  • Pre-registered in VITechnologies/projectbuddy#337: the same baseline commit, a frozen prompt, the same model, and a frozen acceptance gate (the reference checker, plus an independent review).
  • Unattended: each run is a scripted Copilot CLI session with no operator and no route to any remote. A run that fails its gate is resumed with the findings until it passes. Its cost is the total.
  • Metered in AI Units, the unit the quota is billed in. Each run's AI Units come from its own session records, and the account quota measured around each step agreed with them, to between −0.04% and +8.8% (VITechnologies/projectbuddy#339).

What Was Learned

  • A rule that is ambiguous can cost more than it saves. Standard @3 said that a test page "names the member that verifies" a condition it leaves unverified, and a run read that as "prove it". It audited another repository and cost 12% more than the control. Standard @4 takes the verifying member from the allocating specification instead. That fixed the problem and paid back its own cost after about 2.5 changes.
  • Operator time is the largest lever. It was 70.7% of the workstation's baseline cost, and the measured runs needed none of it.
  • Caveat: the two control runs differ from each other by 25%. With one run per treatment, the 17.6% is not statistically significant. The effect on the targeted defect was the clearest result.

Next

  • Adopt @4 across the ecosystem.
  • Iteration 3: an observation claims only what its stimulus shows.
  • At least 3 runs per arm, for significance.

Clone this wiki locally