Repository navigation
DeciShift: behavioral regression testing for versioned ML decision systems #1838
Replies: 1 comment 1 reply
|
For an agent, I would make the comparison unit a task episode rather than only the final decision. Each episode needs a versioned receipt: initial goal and inputs; resolved prompt/tool/policy versions; canonicalized tool actions; retries and budgets; terminal outcome; and the evidence that supports that outcome. Then you can compare two useful things independently:
That matters because an unchanged final answer can hide a real regression: twice the tool calls, a weaker source, or a recovery loop doing the work a first attempt used to do. I would also let each task declare its own allowed change contract. For example, a retrieval agent may allow a different source only if the cited claim remains supported; an action agent may allow a different path only if no protected action is added. Keep test inputs and the measurement policy fixed, then report the transition matrix and the evidence for each changed episode. The hard part is separating intentional behavior change from drift. A declared component version and an explicit expected-change set give reviewers somewhere to start; anything outside that set becomes a regression candidate rather than an unexplained aggregate score movement. |
Uh oh!
There was an error while loading. Please reload this page.
I’ve been exploring a form of evaluation focused not only on model-level metrics but on behavioral changes in the final decisions produced by a complete system.
I recently open-sourced DeciShift, a CPU-first Python framework that compares two executable versions of an ML decision system.
It asks:
The framework currently focuses mainly on conventional/tabular ML systems and includes examples using scikit-learn, XGBoost and LightGBM.
GitHub: https://github.com/sauravsingla/DeciShift
The broader question I’m interested in is whether decision-level behavioral regression testing could also become a useful eval abstraction for systems composed of multiple versioned components.
I’d be interested in feedback on what additional abstractions would be necessary for this approach to become useful for agent/LLM systems as well.
All reactions