fix: WeakspotsDiagnosis handles multiclass and non-{0,1} labels

[hunner]

Pull Request Description

What and why?

WeakspotsDiagnosis bins each feature into slices and scores every slice with sklearn's precision_score/recall_score/f1_score called bare, so sklearn's defaults average='binary', pos_label=1 apply to every slice. Those defaults are only valid when a slice's labels are binary and include 1. For anything else the test crashes before producing a result:

• a slice whose labels collapse to two values not containing 1 (e.g. {0, 4}) fails label validation → ValueError: pos_label=1 is not a valid label. It should be one of [0, 4] — the exact error reported by the customer on ZD-704 (v2.13.6, multiclass logistic regression);
• a slice retaining ≥3 labels fails with ValueError: Target is multiclass but average='binary'.

The test tags itself multiclass_classification, so multiclass is in-scope by its own metadata — the averaging strategy just never adapted. This is the same class of bug fixed for MinimumF1Score in #529 (shipped in v2.13.6, confirmed working by the same customer), here multiplied across every feature slice. Even a purely binary model crashes if its two labels aren't {0, 1} (e.g. {0, 4} or string labels).

After this change, the averaging strategy is decided once from the global label set — the union of the target and prediction columns of both input datasets — and bound into the default metric registry via functools.partial:

• > 2 classes: average="macro", labels=<global labels>, zero_division=0 — macro for consistency with fix: MinimumF1Score multiclass detection uses y_true and y_pred #529; passing the global label set keeps per-slice scores comparable when a bin lacks some classes, and zero_division=0 suppresses the resulting undefined-metric warnings instead of emitting one per empty class per slice;
• binary but 1 not a label (e.g. {0, 4}): pos_label=labels.max() — the larger value is treated as the positive class;
• conventional {0, 1} binary: untouched — sklearn defaults, zero behavior change for the vast majority of existing usage.

Deciding globally rather than per slice matters: per-slice detection would silently mix binary and macro semantics across bars of the same chart, and would still crash on {0, 4}-style slices.

The sibling diagnosis tests OverfitDiagnosis and RobustnessDiagnosis share the identical bare-call weakness in their metric registries. Their default classification metric is auc, so default runs don't hit it, but explicitly selecting metric="f1"/"precision"/"recall" crashes under the same conditions. The second commit applies the same global-label-set guard there, closing out this bug class for the ticket (sibling hardening was also the pattern asked for in #529's review).

User-supplied custom metrics are left alone in all three tests — custom callables own their own kwargs.

How to test

Reproduce the customer's failure on main, then confirm it on this branch:

import numpy as np, pandas as pd
from sklearn.linear_model import LogisticRegression
import validmind as vm

rng = np.random.default_rng(0)
X = pd.DataFrame(rng.normal(size=(400, 3)), columns=["a", "b", "c"])
y = rng.choice([0, 2, 4], size=400)          # multiclass, non-contiguous labels
df = X.assign(target=y)
model = LogisticRegression().fit(X, y)

vm_model = vm.init_model(model, input_id="logistic")
vm_train = vm.init_dataset(df.iloc[:300], target_column="target", input_id="train")
vm_test = vm.init_dataset(df.iloc[300:], target_column="target", input_id="test")
vm_train.assign_predictions(vm_model); vm_test.assign_predictions(vm_model)

vm.tests.run_test(
    "validmind.model_validation.sklearn.WeakspotsDiagnosis",
    inputs={"datasets": [vm_train, vm_test], "model": vm_model},
)

On main this raises pos_label=1 is not a valid label... (or the multiclass/binary-average variant depending on which slice is scored first); on this branch it completes. Swapping [0, 2, 4] for [0, 4] exercises the non-{0,1} binary path, and [0, 1] confirms the untouched conventional path. The same shapes with metric="f1" cover OverfitDiagnosis/RobustnessDiagnosis.

Unit tests: python -m unittest discover -s tests/unit_tests/model_validation/sklearn → 44 passed, 2 skipped (pre-existing skips). New coverage includes the three averaging branches plus end-to-end no-raise runs for {0, 2, 4}, {0, 4}, and {0, 1} across all three tests.

What needs special review?

• pos_label=labels.max() for non-{0,1} binary is a judgment call (macro averaging would also unblock it). Treating the larger label as the positive class matches the common encoding, and it's documented in the docstring.
• Weakspots structural move: df_1/df_2 are now built before metric preparation so the global label set exists when defaults are bound. They're pure column selections, so no behavior change — but it's the one reordering in the diff.
• Overfit gate: the rebinding only engages on the classification path (detected via the probability column), so the regression tests let the model produce probabilities rather than injecting bare predictions — worth a glance to confirm that matches real usage.
• Out of scope, flagged for completeness: the siblings' auc path has a separate multiclass gap (roc_auc_score without multi_class=), not touched here.

Dependencies, breaking changes, and deployment notes

None. Every changed path currently raises ValueError, so behavior only changes from "crash" to "scored"; conventional {0, 1} binary paths are byte-for-byte unaffected.

Release notes

Fixed WeakspotsDiagnosis (and OverfitDiagnosis/RobustnessDiagnosis with explicit f1/precision/recall metrics) failing with ValueError: pos_label=1 is not a valid label or Target is multiclass but average='binary' for multiclass models and for binary models whose labels are not {0, 1}.

Checklist

• What and why
• Screenshots or videos (Frontend)
• How to test
• What needs special review
• Dependencies, breaking changes, and deployment notes
• Labels applied
• PR linked to Shortcut
• Unit tests added (Backend)
• Tested locally
• Documentation updated (if required)
• Environment variable additions/changes documented (if required)

Closes ZD #704 / sc-17091

🤖 Generated with Claude Code

[github-actions]

PR Summary

This pull request introduces improvements to how classification metrics (f1, precision, and recall) are handled in model diagnosis tools. The changes adjust the averaging strategy by binding the appropriate keyword arguments based on the global set of labels encountered in both training and testing datasets. This ensures that when using metrics from scikit-learn, scenarios such as multiclass tasks or non-standard binary cases (for example, labels such as {0, 4} where the default binary behavior of pos_label=1 would fail) are correctly managed.

Key functional changes include:

• Introduction and enhancement of a helper function (_classification_metric_fn) in multiple diagnosis modules (OverfitDiagnosis, RobustnessDiagnosis, and WeakspotsDiagnosis) that chooses the averaging method (macro averaging for multiclass or adapting pos_label for non-standard binary) from the union of true and predicted labels.
• Addition of a new helper (_averaged_default_metrics) within the weakspots diagnosis module to bind default metric functions with the correct averaging parameters based on the global label set.
• Updates to the diagnosis functions to compute the global label set before finalizing the metric function. This avoids inconsistent averaging semantics across different slices of the data and prevents errors when unexpected label configurations occur.
• Extended regression tests and unit tests across the new and modified diagnosis modules to verify the correct behavior of the enhanced averaging logic. These tests ensure that both conventional and non-conventional binary, as well as multiclass classification scenarios, are handled correctly.

The modifications improve the robustness and reliability of the diagnosis tools by ensuring that metric calculations do not fail due to mismatches between dataset labels and scikit-learn default behaviors.

Test Suggestions

• Add tests for edge cases with highly imbalanced classes to ensure that the averaging logic holds.
• Verify behavior when the global label set is derived from more than two datasets or when there are missing classes in one of the datasets.
• Perform integration tests with larger synthetic datasets that mimic real-world label distributions to validate performance under different scenarios.
• Include tests that deliberately provide datasets with unconventional binary labels (e.g., labels other than {0, 1} or {1, 2}) to ensure the pos_label and averaging settings are correctly applied.

[hunner]

Closing in favor of #534, which subsumes this fix: same averaging convention (global label set, macro for multiclass, pos_label=max for non-{0,1} binary) applied via a shared helper module, plus the multiclass default-auc path in OverfitDiagnosis/RobustnessDiagnosis that this PR left out of scope, a CalibrationCurve multiclass skip, and a SHAPGlobalImportance fix. I verified empirically that #534 covers every scenario this branch fixes (15-scenario matrix run against both branches — details in my review there).

Tracking stays on sc-17091 / ZD-704.

🤖 Automated reply