Skip to content

Releases: probkit/probmetrics

v1.3.0

27 May 16:14
8e8d190

Choose a tag to compare

v1.3.0 by @eugeneberta:

  • Splitted calibrators.py into several subfiles and implemented many new calibrators,
    most of which are benchmarked and ranked in the CalArena leaderboard.
    Among others, added Binary histogram binning, Scaling-binning (from
    uncertainty-calibration), BBQ (from netcal), Beta calibration (from betacal), ENIR
    (from netcal), binary and multiclass Kernel calibration (using beta and dirichlet
    kernels from ece_kde), Spline calibration (from splinecalib), CDF-Spline calibration,
    Ensemble Temperature Scaling, tree based calibration with CatBoost, LightGBM and
    XGBoost.
  • Re-structured the base Calibrator class to differentiate
    _predict_proba_torch_impl from _predict_proba_impl.
  • Added Kuiper and Kolmogorov-Smirnov binary calibration metrics.
  • Deprecated python 3.9, added python 3.13 and 3.14 support.

What's Changed

v1.2.0

03 Mar 09:57

Choose a tag to compare

v1.2.0 by @elsacho: Added new proper loss functions:

  • ProperLpLoss(p=p): Metrics to evaluate $E[ \Vert f(X) - E[Y|f(X)] \Vert_p ]$ where $f(X)$ are the
    predictions of the classifier, $p >= 1$, including p=float("inf")
  • TopClassLoss: A wrapper to variationally evaluate top-class errors.
  • OverConfidenceLoss & UnderConfidenceLoss: Wrappers to variationally evaluate
    over/under-confidence in binary predictors.
  • MetricsWithCalibration can now handle arbitrary classifiers and Lp-type losses.
  • New classifiers: Added WS_CatboostClassifier and WS_LGBMClassifier for
    evaluating calibration errors.
  • removed sklearn < 1.7 constraint.

v1.1.0

09 Jan 17:22

Choose a tag to compare

v1.1.0 by @eugeneberta: Improvements to the SVS and SMS calibrators:

  • logit pre-processing with 'ts-mix' is now automatic,
    and the global scaling parameter $\alpha$ is fixed to 1. This yields:
    • improved performance on our tabular and computer vision benchmarks
      (see the arxiv v2 of the SMS paper, coming soon).
    • faster convergence.
    • ability to compute the duality gap in closed form for stopping SAGA solvers,
      which we implement in this version.
  • improved L-BFGS solvers, much faster than in the previous version.
    Now the solver for default SVS and SMS.
  • the default binary calibrator in LogisticCalibrator is now quadratic scaling
    instead of affine scaling, this can be changed back by using
    LogisticCalibrator(binary_type='affine').

v1.0.0

05 Nov 16:39

Choose a tag to compare

What's Changed

  • New post-hoc calibrators including SMS, SVS, affine and quadratic scaling. by @eugeneberta in #1

New Contributors

Full Changelog: v0.0.2...v1.0.0