Metrics 0.2.0 classification

Lodestar.Metrics 0.2.0. This page is frozen at that release. Read the current documentation for what main says now. A link to a decision or a migration page follows main, and leaves the archive.

Classification metrics — Lodestar.Metrics

Your model looked at some things and put a label on each one. How well did it do? Every type on this page answers that, and they disagree — not about the arithmetic, but about what "well" is worth measuring. One number can hide a model that never predicts the rare class; another can be near zero for a model that is right nine times in ten. Reporting the wrong one is the usual reason a model looks fine on a slide and useless in production.

Almost everything here is built on one object, so it is worth reading first.

A confusion matrix is a table with one row per true class and one column per predicted class, and each cell holds how many samples fell there. The diagonal is what the model got right; every other cell is a specific mistake — "this class, mistaken for that one". For two classes the table has four cells, and they have names: true positives (said yes, was yes), false positives (said yes, was no), false negatives (said no, was yes) and true negatives (said no, was no). Precision, recall and F1 are three different divisions of those four numbers.

flowchart LR
    subgraph M["The four cells, for one class"]
      direction TB
      TP["<b>TP</b><br/>said yes, was yes"]
      FP["<b>FP</b><br/>said yes, was no"]
      FN["<b>FN</b><br/>said no, was yes"]
      TN["<b>TN</b><br/>said no, was no"]
    end
    TP --> P["<b>Precision</b> = TP / (TP + FP)<br/><i>of what I flagged, how much belonged</i>"]
    FP --> P
    TP --> R["<b>Recall</b> = TP / (TP + FN)<br/><i>of what belonged, how much I found</i>"]
    FN --> R
    P --> F["<b>F1</b><br/>harmonic mean of the two"]
    R --> F
    TP -.-> A["<b>Accuracy</b> = (TP + TN) / everything"]
    TN -.-> A
    TN -.-x|"never read"| P
    TN -.-x|"never read"| R

Loading

The dotted lines are the point: TN is invisible to precision, recall and F1. A model that says "no" to everything scores perfectly on the true-negative cell and zero on all three of them, which is why a rare-disease detector can be 99% accurate and worthless.

Three conventions run through the whole namespace.

• Every metric has two ways in. One overload takes yTrue and yPred and counts the matrix on the way; the other takes a ConfusionMatrix you already have. They give the same number, and the second is what you want when you are reporting five metrics over one dataset — the counting happens once. The one place they can differ is an explicit labels subset, and each entry says so.
• Labels are int. A string class name is the caller's mapping to make, and ClassificationReport's targetNames is where readable names go back on.
• Undefined is a real answer, not a crash. A class nothing was predicted into has no precision; a class with no true samples has no recall. ZeroDivision says what comes back, and the default reproduces scikit-learn's 0.0.

Regression metrics — how far a number is from another number — are on the regression page, not here.

Which one do I report?

flowchart TD
    A["What are you reporting?"] --> B{"Are you scoring a decision,<br/>or a ranking?"}
    B -->|a ranking, or a probability| C["RocAuc"]
    B -->|a decision| D{"Are the classes<br/>roughly balanced?"}
    D -->|yes, and every mistake costs the same| E["Accuracy"]
    D -->|no| F{"Is one class the one<br/>you actually care about?"}
    F -->|yes| G{"Which mistake hurts more?"}
    G -->|a false alarm| H["Precision"]
    G -->|a miss| I["Recall"]
    G -->|both, and equally| J["F1"]
    G -->|both, unequally| K["FBeta"]
    F -->|no, every class matters| L{"Should a rare class count<br/>as much as a common one?"}
    L -->|yes| M["BalancedAccuracy,<br/>or Averaging.Macro"]
    L -->|no| N["Averaging.Weighted"]
    A --> O{"Do you want one number<br/>that already discounts luck?"}
    O -->|against chance agreement| P["CohenKappa"]
    O -->|as a correlation| Q["MatthewsCorrelation"]
    A --> R["Looking rather than reporting:<br/>ConfusionMatrix, then ClassificationReport"]

Loading

Type	What it is
Accuracy	The share of samples the model got right.
AverageRow	One averaged line of a ClassificationReport.
Averaging	How per-class scores are reduced to one number.
BalancedAccuracy	Accuracy that counts every class equally, however rare.
ClassificationReport	The per-class table, structured and as printable text.
ClassRow	One class's line of a ClassificationReport.
CohenKappa	Agreement between two raters, with chance agreement subtracted.
ConfusionMatrix	Predictions counted against truth — the table everything else reads.
F1	The harmonic mean of precision and recall.
FBeta	The same, with the balance between the two turned by hand.
KappaWeighting	How far apart two classes count as being, for CohenKappa.
MatthewsCorrelation	The correlation between prediction and truth, in [-1, 1].
MultiClassRocOptions	The optional settings of multiclass ROC-AUC.
MultiClassStrategy	One class against the rest, or every pair.
Normalization	Which sum a confusion matrix's cells are divided by.
Precision	Of everything flagged as a class, how much belonged there.
Recall	Of everything that belonged to a class, how much was found.
RocAuc	How well the scores rank a positive above a negative.
UndefinedMetricException	Thrown when a metric is undefined and you asked to be told.
ZeroDivision	What an undefined metric returns instead of throwing.