Repository navigation
v0.4.0
0.4.0
Added
- Multiclass and multilabel models.
metadata.jsongainstask_type
(binary|multiclass|multilabel) andoutput.{name, classes, thresholds}.simulate,evaluate, andcomparetake--task, and
--positive-group-idis repeatable (one cohort per model output, in
output.classesorder). All sklearn-onnx probability formats are read:
[n, K]tensors, ZipMaps with int or string keys (matched toclassesby
name), andMultiOutputClassifier's per-label tensor sequence. - Every metric is computed per class/label; headline fields are macro
averages. A newmultisection in the result JSON reports micro ROC AUC,
worst-label resiliency, a jitter decision-flip rate, the share of flipper
pairs that never flip, the multiclass detour rate, labels excluded for
having one class, and per-label scores. Binary output has nomultikey. - Multiclass jitter stability sums the per-class prediction variances and
halves them, so it equals the binary score for 2 classes and does not
depend on how many classes the model has. Multilabel averages per label. - Multiclass flipper finds where the pairwise margin
p[b] − p[a]crosses 0;
multilabel flipper samples pairs per label (newpair_labelgrid column)
and crosses each label's own threshold. - Single-target regression models.
task_type: "regression"in
metadata.json(default outputvariable, thenpredictions), and
--task regressionwith true values from--targets <parquet> --target-col <name>, joined by ID as text.baselinesarermse,mae
andr2; resiliency is blackout R² ÷ clean R² (the share of skill kept)
undermetric_based.r2; jitter variance is scaled by the target's variance.
Flipper pairs patients across a clinical cutoff (--flip-threshold, in
target units, passed to bothsimulateandevaluate) and finds where the
prediction crosses it. A newregressionsection reports these in target
units; classifier JSON is unchanged.
Changed
- Breaking (library):
Model::predict_probareturns aDataFramewith
one column per output,SimulationVariant.yis aDataFramelabel matrix,
EvaluationConfiggainstask, andOnnxModel::from_filetakes
positive_class_index: Option<usize>.TaskTypegainsRegressionand
EvaluationResultgains an optionalregressionfield. CLI usage for
binary models is unchanged. evaluate/compare--labelsis no longer required at parse time; it is
required for classifiers and refused for regression.--flip-threshold
still defaults to 0.5 for classifiers.- ROC AUC handles tied scores. Ties count half (Mann–Whitney) instead of
depending on row order. Results without ties agree with the previous
implementation to floating-point rounding. - The probability output is resolved when the model loads. A model without
output_probability/probabilities(or the contract'soutput.name) now
fails at load with the list of its real outputs, instead of at first
prediction. comparerefuses two--modelfiles with the same file name instead of
panicking on duplicate prediction columns.
Docs
- New pages for hospital IT and model reviewers: security and data handling
(data flow, network behaviour, offline install, PHI in outputs), metric
definitions and caveats, and an end-to-end model-selection walkthrough. - One-page usage overview PDF for IT review.
- Fixed broken schema links and removed internal references from the READMEs
and--helptext.
What's Changed
- Docs: security, metrics, and model-selection guides for hospital IT by @jeremy-wayland in #1
- Multiclass and multilabel model evaluation by @jeremy-wayland in #2
- Single-target regression model evaluation by @jeremy-wayland in #3
- Simplify agent docs and bring them up to date with #2 and #3 by @jeremy-wayland in #4
- Prepare v0.4.0 release by @jeremy-wayland in #5
New Contributors
- @jeremy-wayland made their first contribution in #1
Full Changelog: v0.3.0...v0.4.0