Knowing When to Re-evaluate

[joshua-temple]

The Proposal

Define explicit signals that trigger a re-evaluation of our tooling choices, so the decision to look again is made by a tripwire we agreed on in advance, not by drift or by whatever benchmark chart is loudest that week.

Why This Matters

Both failure modes are silent, and they pull in opposite directions.

If our stack falls behind, nothing visibly breaks. Every task still completes; the work is just a bit slower, a bit more expensive, a bit more error-prone. No single day feels wrong, so no single day prompts the question. Falling behind looks exactly like normal.

Churn is the mirror image. A team that re-litigates its model choice on every release announcement never compounds on anything. Prompts, skills, and workflow investments reset with each switch, and evaluation itself becomes a standing tax.

Pre-agreed triggers solve both at once: they guarantee we look when something meaningful changes, and they license us to not look when nothing has.

What a Trigger Could Look Like

External events (cheap to watch):

• A frontier model release in a class we use daily
• A pricing change beyond an agreed percentage on a tool we pay for
• A pattern or protocol reaching real adoption among teams we benchmark against

Internal signals (visible in our own work):

• Retry rate on agent tasks climbing
• Humans re-doing agent output more often
• Token spend per shipped change trending up over an agreed window

Each trigger needs two levels: "interesting, log it" and "stop and re-evaluate now."

Open Questions

1. Baseline first. Can we state today's cost per shipped change and first-pass success rate as numbers? If not, what is the smallest instrumentation that gets us there within two weeks?
2. External watch list. Which two or three external events deserve a standing watch, and who owns watching each one?
3. First internal metric. Which internal signal do we trust enough to alert on today, and where does its data already live?
4. Thresholds. What delta, on which metric, moves us from "noted" to "re-evaluate now"?
5. Cost of the re-evaluation itself. What is a re-evaluation time-boxed to (a day, a week?), so thresholds can be set against that cost rather than picked arbitrarily?

[joshua-temple]

Accepted, merged with #3 as Deliberate Currency. Documented in #13.