ShiftWatch: reproducible ML evaluation and dataset drift monitoring

[pralav-25]

ShiftWatch brings model evaluation and numerical data monitoring into one inspectable workflow: run the Python experiment, inspect its report in the dashboard, and follow the analysis in an executed notebook.

Try the dashboard · Executed analysis notebook · Source and setup

What you can explore

• A dummy baseline, logistic regression, and random forest compared using training-only cross-validation and a separate holdout.
• Synthetic feature shifts and missing-data scenarios, with PSI, KS tests, adjusted p-values, and missingness alerts.
• Your own numerical CSV comparisons through the Python CLI, with JSON reports that load into the dashboard.
• Reproduction checks, automated tests, and a notebook with exploration, model comparison, holdout errors, and drift interpretation.

A useful example

A feature can develop more missing values without a dramatic model-accuracy change. Load the demo's missingness scenario and compare the data-quality alerts with the prediction metrics. The two signals answer different questions.

The included experiment uses the small UCI Wine dataset and synthetic shifts. Its scores describe this experiment; they do not establish production performance or tell you automatically when to retrain. The methodology and model card explain the assumptions and limits.

Feedback welcome: what would make a drift report more useful to you—a clearer explanation, another comparison view, or a particular input format? Reproducible examples using synthetic or public data are especially helpful.