Reproducibility

[joshua-temple]

The Proposal

Decide what level of reproducibility each part of the system requires, and build the mechanisms (pinning, seeding, logging, evaluation) to achieve it. Same inputs should produce predictably similar outputs, where "similar" is defined per use case rather than assumed.

Why This Matters

LLMs are non-deterministic by design: sampling temperature, seeds, and provider-side model updates all introduce variance. Without a stated reproducibility target, every downstream practice quietly breaks:

• Debugging: a bug you cannot reproduce is a bug you cannot fix, or prove fixed.
• Testing: assertions need a stable baseline; without one, tests flake and get deleted.
• Auditing: answering "why did the system do that?" requires replaying the decision with the same inputs.
• Trust: users who get different answers to the same question stop believing any of them.

Chasing exact reproducibility everywhere is the wrong fix. It is often unachievable (provider updates) and sometimes undesirable (creative tasks benefit from variance). The real work is choosing the right level per use case.

Levels of Reproducibility

• Exact: byte-identical output every run. Requires pinned model versions, temperature 0, fixed seeds, and still breaks when providers update. Reserve for narrow cases.
• Statistical: outputs drawn from the same distribution. Verified by running N samples and asserting on aggregates (pass rate, score distribution).
• Behavioral: the same kind of output (correct answer, right tool call, valid schema) with phrasing free to vary. Verified by property assertions or an evaluator.

Most production needs are behavioral. Most debugging needs are exact, achieved by replaying logged inputs and outputs rather than regenerating them.

Open Questions

• Which of our current workflows require which level? Propose behavioral as the default and enumerate the exceptions.
• What must we log per call (model version, parameters, seed, full prompt) so any past output can be replayed or explained?
• How do we test against behavioral targets: property assertions, LLM-as-judge, N-run pass thresholds? What flake rate is acceptable before a suite loses trust?
• Which paths treat variance as a feature (brainstorming, exploration), and how do we mark them exempt from reproducibility checks?
• How do we detect provider model updates that shift our baselines, and what is the response when one lands?

[joshua-temple]

Accepted. Documented in #13.