Repository navigation
Decision-level regression testing between ML pipeline versions with DeciShift #5314
Replies: 2 comments
|
This looks like a very powerful addition to the ML lifecycle! Regression testing at the decision level is often overlooked but critical for ensuring that model updates don't introduce silent failures in business logic. I'm curious about the integration with ZenML's artifact store—does DeciShift leverage the existing artifact lineage to compare decision outputs between pipeline versions, or does it require a separate data export for the regression analysis? Looking forward to seeing how this evolves! |
|
Thanks, Nidhal — great question. This is actually the integration boundary I’m working through. Today, DeciShift is framework-agnostic and does not yet have a native ZenML artifact-store adapter. The intended integration, though, is not to create a second artifact store or require a duplicate data export. The pattern I’m aiming for is: ZenML artifact versions + lineage → DeciShift comparison step → DeciShift evidence / Decision Contract → ZenML artifact or downstream release gate ZenML would remain the source of truth for the baseline/candidate artifact versions and their lineage. A DeciShift step could consume those specific artifact versions, materialize the required evaluation records/components through ZenML, and then replay the baseline and candidate decision systems on the same records. I’d also preserve the ZenML artifact/version IDs in the DeciShift comparison metadata, so the decision-change evidence can be traced back to the exact pipeline artifacts that produced it. The resulting comparison, transition evidence, and Decision Contract result could then be persisted back into ZenML and consumed by a deployment/release step. One distinction is important: artifact lineage tells us which versions and upstream artifacts are involved, but lineage alone is not enough for decision-level attribution. For full software-counterfactual attribution, DeciShift needs replayable components on the same records. If only precomputed predictions/actions are available, DeciShift can still identify record-level decision transitions, but it deliberately reports component attribution as insufficient evidence rather than inventing an explanation. I’ve opened the ZenML integration work here as well: DeciShift issue #16. So the goal is to reuse ZenML’s lineage and artifact lifecycle, not duplicate it, with DeciShift adding the behavioral regression/attribution layer on top. |
Uh oh!
There was an error while loading. Please reload this page.
I’ve been working on an open-source project called DeciShift that explores an important question beyond model metrics:
When an ML pipeline version changes, which final operational decisions actually changed, and why?
DeciShift compares two executable versions of an ML decision system and identifies changed decisions, action transitions, contributions from features/models/thresholds/policies/rules, cohort-level changes, and whether those changes remain within predefined decision contracts.
A workflow I’m exploring is:
Train/evaluate candidate → compare decision behavior → apply decision contract → release/block
In a reproducible public example, candidate model metrics improved while 30 of 719 final actions still changed. The overall shift stayed within the configured contract, but one cohort exceeded its threshold and caused the release gate to block.
GitHub: https://github.com/sauravsingla/DeciShift
I’d be interested in feedback from ZenML users:
I’d particularly welcome anyone willing to test it against two versions of an existing ML pipeline.
All reactions