Skip to content

v0.4.0

Choose a tag to compare

@github-actions github-actions released this 09 Oct 13:33
· 9 commits to main since this release
c5e59ba

0.4.0

Added

  • Multiclass and multilabel models. metadata.json gains task_type
    (binary | multiclass | multilabel) and output.{name, classes, thresholds}. simulate, evaluate, and compare take --task, and
    --positive-group-id is repeatable (one cohort per model output, in
    output.classes order). All sklearn-onnx probability formats are read:
    [n, K] tensors, ZipMaps with int or string keys (matched to classes by
    name), and MultiOutputClassifier's per-label tensor sequence.
  • Every metric is computed per class/label; headline fields are macro
    averages. A new multi section in the result JSON reports micro ROC AUC,
    worst-label resiliency, a jitter decision-flip rate, the share of flipper
    pairs that never flip, the multiclass detour rate, labels excluded for
    having one class, and per-label scores. Binary output has no multi key.
  • Multiclass jitter stability sums the per-class prediction variances and
    halves them, so it equals the binary score for 2 classes and does not
    depend on how many classes the model has. Multilabel averages per label.
  • Multiclass flipper finds where the pairwise margin p[b] − p[a] crosses 0;
    multilabel flipper samples pairs per label (new pair_label grid column)
    and crosses each label's own threshold.
  • Single-target regression models. task_type: "regression" in
    metadata.json (default output variable, then predictions), and
    --task regression with true values from --targets <parquet> --target-col <name>, joined by ID as text. baselines are rmse, mae
    and r2; resiliency is blackout R² ÷ clean R² (the share of skill kept)
    under metric_based.r2; jitter variance is scaled by the target's variance.
    Flipper pairs patients across a clinical cutoff (--flip-threshold, in
    target units, passed to both simulate and evaluate) and finds where the
    prediction crosses it. A new regression section reports these in target
    units; classifier JSON is unchanged.

Changed

  • Breaking (library): Model::predict_proba returns a DataFrame with
    one column per output, SimulationVariant.y is a DataFrame label matrix,
    EvaluationConfig gains task, and OnnxModel::from_file takes
    positive_class_index: Option<usize>. TaskType gains Regression and
    EvaluationResult gains an optional regression field. CLI usage for
    binary models is unchanged.
  • evaluate/compare --labels is no longer required at parse time; it is
    required for classifiers and refused for regression. --flip-threshold
    still defaults to 0.5 for classifiers.
  • ROC AUC handles tied scores. Ties count half (Mann–Whitney) instead of
    depending on row order. Results without ties agree with the previous
    implementation to floating-point rounding.
  • The probability output is resolved when the model loads. A model without
    output_probability/probabilities (or the contract's output.name) now
    fails at load with the list of its real outputs, instead of at first
    prediction.
  • compare refuses two --model files with the same file name instead of
    panicking on duplicate prediction columns.

Docs

  • New pages for hospital IT and model reviewers: security and data handling
    (data flow, network behaviour, offline install, PHI in outputs), metric
    definitions and caveats, and an end-to-end model-selection walkthrough.
  • One-page usage overview PDF for IT review.
  • Fixed broken schema links and removed internal references from the READMEs
    and --help text.

What's Changed

New Contributors

Full Changelog: v0.3.0...v0.4.0