ModelDiff-ML

Your candidate model improved AUC from 0.84 to 0.85. But did it get better for everyone?

ModelDiff-ML compares two machine learning models one observation at a time. It finds readable regions where a candidate behaves differently from a baseline, then validates those regions on independent data. The result is a practical answer that an average metric cannot provide:

The candidate improved overall, but regressed for customers with high utilization and low income.

The library is deliberately small. It works with fitted models or precomputed predictions, depends only on NumPy, pandas, and SciPy, and produces plain Python objects, JSON, and standalone HTML. It is not an MLOps platform and does not require a model registry, service, or database.

ModelDiff-ML requires Python 3.11 or newer.

pip install modeldiff-ml

The package is published as modeldiff-ml and imported as modeldiff.

Compare two fitted models on a shared evaluation set:

import modeldiff

report = modeldiff.compare(
    baseline=old_model,
    candidate=new_model,
    X=X_test,
    y=y_test,
    task="binary_classification",
    metric="log_loss",
    random_state=42,
)

print(report.summary())
report.to_json("modeldiff-report.json")
report.to_html("modeldiff-report.html")

Binary classifiers must provide predict_proba(X) with two columns. ModelDiff-ML does not turn an uncalibrated decision_function into a probability. Regression estimators only need predict(X).

If predictions already exist, use the model-independent API:

from modeldiff import ModelDiffConfig, compare_predictions

config = ModelDiffConfig(
    max_depth=3,
    min_region_fraction=0.02,
    n_bootstrap=1000,
    n_permutations=2000,
    random_state=42,
)

report = compare_predictions(
    baseline_predictions=old_probabilities,
    candidate_predictions=new_probabilities,
    X=X_test,
    y=y_test,
    task="binary_classification",
    metric="log_loss",
    config=config,
)

for region in report.regressions(significant_only=True):
    print(region.description, region.validation_delta)

X can be a pandas DataFrame or a two-dimensional NumPy array. DataFrames produce more useful condition names. Array columns are named x0, x1, and so on.

A comparison includes the overall metric change, prediction disagreement diagnostics, validated regression and improvement regions, confidence intervals, p-values, q-values adjusted for the false discovery rate, and warnings about small or unstable samples. Reports can be viewed in the terminal, serialized to JSON, or saved as a self-contained HTML file.

Every delta follows the same convention:

delta = candidate loss - baseline loss

Positive values mean the candidate is worse. Negative values mean it is better.

For rmse, the overall metric is square rooted after aggregation. Region discovery and saved checks continue to operate on per-observation squared error, so their deltas are expressed as mean squared error rather than a difference between aggregate RMSE values.

One discovery split can still produce a slice that depends on a lucky threshold or a handful of rows. The stability API repeats region discovery on subsamples of the discovery partition, groups similar regions by their membership, and keeps only those selected often enough. It then evaluates the surviving conditions once on the untouched validation partition.

from modeldiff import ModelDiffConfig, StabilityConfig, compare_stable_predictions

report = compare_stable_predictions(
    old_predictions,
    new_predictions,
    X,
    y,
    task="regression",
    metric="mae",
    config=ModelDiffConfig(random_state=42),
    stability_config=StabilityConfig(
        n_resamples=30,
        subsample_fraction=0.75,
        min_selection_frequency=0.50,
        random_state=42,
    ),
)

for region in report.regressions(significant_only=True):
    print(region.description, region.selection_frequency, region.validation_delta)

Each stable region records its selection frequency, sign consistency, similarity to other members of its cluster, and the ranges of discovery effects and support across resamples. Random and explicit splits resample rows, group splits resample whole groups, and time splits use contiguous blocks. In every case, resampling stays entirely inside discovery data.

Pass an existing CheckSuite through known_checks= to distinguish newly discovered regressions from behavior the project already tracks. A stable regression is labeled known when saved checks cover at least 80% of its validation membership, partially_covered when coverage is lower but nonzero, and new when there is no overlap. Without a suite, triage remains unclassified.

ModelDiff-ML searches for regions on one partition and evaluates them unchanged on another. A random split is the default, while three additional strategies help prevent leakage in structured data.

For example, a group split keeps every row for a customer in one partition:

report = compare_predictions(
    old_predictions,
    new_predictions,
    X,
    y,
    task="regression",
    metric="mae",
    validation_strategy="group",
    groups=customer_ids,
)

A chronological comparison uses earlier observations for discovery and later observations for validation:

report = compare_predictions(
    old_predictions,
    new_predictions,
    X,
    y,
    task="regression",
    metric="mae",
    validation_strategy="time",
    time_values=event_timestamps,
)

Group and time boundaries stay as close as possible to discovery_fraction without splitting a group or equal timestamp. Explicit partitions must assign every observation exactly once. Split metadata is available through report.split and is included in serialized reports.

A discovered region can become a fixed regression check for future model releases. The saved check keeps the exact structured conditions, metric settings, allowed loss increase, minimum sample count, and enough provenance to trace it back to a comparison report.

checks = report.export_checks(
    regions=report.regressions(significant_only=True),
    max_loss_increase=0.01,
    min_samples=100,
    names=["High utilization applicants"],
)
checks.to_json("model-checks.json")

result = modeldiff.evaluate_checks(
    checks="model-checks.json",
    baseline_predictions=production_predictions,
    candidate_predictions=new_predictions,
    X=X_fresh,
    y=y_fresh,
)

print(result.summary())
result.to_json("check-results.json")
result.to_html("check-results.html")

Omitting regions exports significant regression regions by default. Passing an empty list creates a valid empty suite. When names is provided, it must contain one name for each selected region. Check IDs are deterministic, and policy settings can be refined without mutating the original suite:

check_id = checks.checks[0].id
checks = checks.with_check_settings(
    check_id,
    name="Priority production segment",
    max_loss_increase=0.005,
    min_samples=250,
)

Observed checks receive pass, fail, or insufficient_data. A loss increase equal to the tolerance passes. A slice with fewer than min_samples, including a slice with zero matches, is never allowed to pass silently.

Check tolerances use the region loss units shown above. An accuracy tolerance is an increase in misclassification rate. An rmse tolerance is an increase in mean squared error. Classification thresholds and the numerical epsilon used for log loss are preserved in the saved suite.

Saved checks are release policies, not causal or hypothesis-test conclusions. Fresh evaluation data is strongly recommended. Reusing discovery data is descriptive, while repeatedly tuning against the same evaluation set can overfit that set. Exported checks also represent selected findings rather than complete coverage of every regression a future model might introduce.

For uncertainty-aware release gates, use a confidence decision policy. A check passes only when its entire interval is within tolerance, fails only when the entire interval exceeds tolerance, and is otherwise inconclusive. Bonferroni correction is the default so the requested confidence level applies to the suite rather than to each check in isolation.

from modeldiff import CheckDecisionPolicy

result = modeldiff.evaluate_checks(
    checks,
    production_predictions,
    new_predictions,
    X_fresh,
    y_fresh,
    decision_policy=CheckDecisionPolicy(
        method="confidence",
        confidence_level=0.95,
        correction="bonferroni",
    ),
)

result.to_junit_xml("modeldiff-checks.xml")

The complete workflow is available in examples/saved_checks_example.py.

The CLI works with CSV files containing features, the target, and both prediction columns:

modeldiff-ml compare \
  --data validation.csv \
  --target y \
  --baseline-pred old_prediction \
  --candidate-pred new_prediction \
  --task binary_classification \
  --metric log_loss \
  --html report.html \
  --json report.json

Columns other than the target and prediction columns are treated as features. Repeat --exclude COLUMN to omit metadata. Group, time, and explicit partitions are also available:

modeldiff-ml compare ... --validation-strategy group --group-column customer_id
modeldiff-ml compare ... --validation-strategy time --time-column event_time
modeldiff-ml compare ... --validation-strategy explicit --split-column partition

An explicit partition column must contain only discovery and validation. Columns used to control a split are automatically excluded from region search.

The existing comparison command can fail a CI job for a significant region or excessive overall regression:

modeldiff-ml compare ... --fail-on-significant-regression
modeldiff-ml compare ... --max-overall-regression 0.01

Stable discovery has its own command and accepts the same split controls:

modeldiff-ml stability \
  --data validation.csv \
  --target y \
  --baseline-pred old \
  --candidate-pred new \
  --task regression \
  --metric mae \
  --n-resamples 30 \
  --min-selection-frequency 0.5 \
  --known-checks model-checks.json \
  --json stability-report.json \
  --html stability-report.html

To save discovered regions and evaluate them later:

modeldiff-ml compare \
  --data discovery.csv \
  --target y \
  --baseline-pred old \
  --candidate-pred new \
  --task regression \
  --metric mae \
  --export-checks model-checks.json \
  --check-max-loss-increase 0.01 \
  --check-min-samples 100

modeldiff-ml check \
  --checks model-checks.json \
  --data fresh-predictions.csv \
  --target y \
  --baseline-pred production_prediction \
  --candidate-pred new_prediction \
  --decision confidence \
  --correction bonferroni \
  --junit check-results.xml \
  --json check-results.json \
  --html check-results.html

The check command reads the task, metric, classification threshold, and epsilon from the suite. Exit code 0 means every check in a nonempty suite passed. Exit code 1 covers a failed or inconclusive check, insufficient data, or an empty suite. Exit code 2 indicates invalid usage, inputs, or suite data. Requested reports are written before a valid evaluation returns exit code 1.

ModelDiff-ML keeps selection and evaluation separate.

1. It computes a loss difference for every observation.
2. It creates independent discovery and validation partitions.
3. Numeric conditions come from empirical quantiles. Categorical conditions use frequent values.
4. A bounded beam search builds short conjunctions and ranks them by effect and support.
5. Highly overlapping regions are removed using Jaccard similarity.
6. Selected conditions are applied unchanged to the validation partition.
7. Validation effects receive percentile bootstrap intervals, paired sign flip p-values, and Benjamini–Hochberg adjusted q-values.

This separation matters. Choosing a threshold and reporting a p-value from the same rows would understate selection bias. Independent validation is a practical first defense, although it does not turn exploratory subgroup discovery into a causal conclusion.

ModelDiffConfig controls partition size, beam width, search depth, support limits, overlap deduplication, bootstrap and permutation counts, confidence and significance levels, classification threshold, numerical epsilon, and random seed. Defaults favor readable regions and avoid tiny slices. Missing feature values are never imputed; a missing value does not match a condition.

Standard comparison reports retain schema 1.0. Stability reports, newly saved check suites, and check evaluation results use schema 1.1. Version 1.0 check suites remain loadable and use the observed decision policy. Serialized artifacts contain no model objects, raw observations, or original sample IDs. Unavailable numeric results are represented as null, never NaN or infinity.

The classification example plants a local classification regression inside a candidate that improves overall. The regression example does the same for continuous predictions. The saved checks example discovers a region, saves it, and evaluates both a failing and a passing candidate on fresh data.

ModelDiff-ML currently supports binary classification and regression. Discovered subgroups remain exploratory. Independent validation and false discovery rate correction reduce false findings but cannot eliminate them, especially with small data, distribution shift, correlated observations, or repeated human analysis. A fresh final test set is appropriate before high-stakes deployment.

The project does not make causal claims or provide fairness certification. It also does not include multiclass evaluation, distributed execution, monitoring infrastructure, model registries, dashboards, or automated remediation.

git clone https://github.com/ecklein/modeldiff-ml.git
cd modeldiff-ml
python -m venv .venv
. .venv/bin/activate
pip install -e '.[dev]'
pytest
ruff check .
mypy src

Bug reports and focused contributions are welcome through GitHub Issues.

ModelDiff-ML is available under the MIT License.