Adds mechanism for calibrating probabilities for category and binary features

[dantreiman]

Implements:
temperature scaling for binary and category outputs
matrix scaling for category outputs

Example using Twitter Bots w categorical output feature:
On this example, matrix scaling does not improve ECE, though it improves NLL (probably due to overfitting) so matrix scaling gets rolled back and we get the original uncalibrated probs. On much larger datasets matrix scaling may yield better results.

Validation Set (used to determine softmax temperature):

Test set:

[github-actions]

Unit Test Results

       6 files  ±  0         6 suites  ±0   2h 24m 4s ⏱️ + 10m 50s
2 837 tests +  5  2 803 ✔️ +  5    34 💤 ±0  0 ❌ ±0 
8 511 runs  +15  8 405 ✔️ +15  106 💤 ±0  0 ❌ ±0 

Results for commit aaea9d8. ± Comparison against base commit 8c6e189.

♻️ This comment has been updated with latest results.

[w4nderlust]

In the image I can't see the blue points, do they overlap with the green ones?

[dantreiman]

In the image I can't see the blue points, do they overlap with the green ones?

Thats right, in this example the matrix scaling got rolled back to the original uncalibrated probabilities.

[w4nderlust]

Not 100% sure, but it seems to me that in some cases calibration is the act of calibrating outputs, in some cases calibration is the object/function that can calibrate and in some other cases it is the act of training the calibrator.

We probably should be a bit more explicit to avoid confusion.
Wdyt about train_calibrator, get_calibrated_probabilities_from_logits or something else?
Those may be too verbose, but we can brainstorm a bit on this

[dantreiman]

I like train_calibrator, that is clear. get_calibrated_probabilities_from_logits is a good name IMO, where are you thinking that should go (which method would be renamed to this)? Right now it happens in the PredictModule which just calls the forward method of the calibration module.