Skip to content

Releases: m-sanchez/calibrated

calibrated 2.0.1

Choose a tag to compare

@m-sanchez m-sanchez released this 03 Oct 08:57
5ebef32

Numerical corrections for temperature scaling. npm registry publication is pending; the corrected source is available in PR #1.

  • Compute NLL without probability clipping and preserve raw-logit decisions when probabilities round to ties.
  • Handle extreme finite logits, validate inputs and search settings, and report fit status and iteration count.
  • Evaluate the seeded demo on separate calibration and test data; clarify the limits of ECE resampling and null simulations.
  • Compatibility: empty NLL returns NaN; empty fitting and empty softmax throw. Invalid temperatures, dimensions and search settings throw. Code constructing TemperatureFit objects must supply the new status and iterations fields.
  • Validation: 46 tests, typecheck, build and package-install checks passed on Node 22, 24 and 26 in CI. Independent local SciPy checks passed for 403 probability/loss fixtures and three fits.

Detailed changes and limits.

v2.0.0

Choose a tag to compare

@m-sanchez m-sanchez released this 01 Sep 19:45

ECE was not a function of the data. Equal-mass binning - the strategy the README recommends for exactly the top-heavy regime where it broke - split runs of tied confidences across bin boundaries. 1,000 predictions at confidence 0.9 with 60% correct (a true gap of 0.30) read 0.4146, and MCE swung 0.1111 to 0.1570 across 50 shuffles of the same rows. Cuts now snap to value boundaries, and a permutation-invariance property test locks it.

  • effectiveBins reports how many bins actually carried data, so a reliability diagram cannot be labelled with a bin count it does not have.
  • New eceInterval() (seeded bootstrap CI) and nullEce(), the noise floor. ECE is positively biased: a perfectly calibrated n=100 at 15 bins reads a median of 0.0874. The README publishes that table, because a bare ECE of 0.08 means nothing without it.
  • calibrationError([]) and brier([]) now return NaN rather than 0 - no data was quietly passing an ece <= 0.1 ship bar as perfect calibration.
  • fitTemperature reports atBound when the optimum is pinned to its own bracket, and validates labels and logits instead of returning NaN.
  • softmax no longer blows the stack on a real vocabulary (tested at 200,000 classes).

36 tests.

v1.0.1

Choose a tag to compare

@m-sanchez m-sanchez released this 01 Sep 15:16

Packaging release.

  • Package renamed to @m-sanchez/calibrated; install with npm install @m-sanchez/calibrated once published, or from this tag: github:m-sanchez/calibrated#v1.0.1.
  • CI packs the scoped tarball and proves it installs and imports on Node 22, 24 and 26.
  • README states provenance: where the code came from and when it was first published.

No behaviour changes.