Skip to content

v1.2.0 — teaching mode

Choose a tag to compare

View all tags
@cattolatte cattolatte released this 27 Jul 08:59
v1.2.0
2d04deb

bench --explain used to print generic prose about each check kind — useful once, useless after. A paragraph on what a confidence interval is does not tell you why yours is that wide.

It now explains the run that just finished, using the folds that actually ran.

  Reading these numbers

    The dummy baseline scores 0.5 without looking at a single feature — it is
    what you get for free. linear scores 0.892, so the features are worth
    0.392 of balanced_accuracy here. That gap, not the headline score, is what
    the modelling bought.

    linear scored between 0.87 and 0.9338 across the 5 folds — a spread of
    0.06383 on the same data and the same settings, differing only in which
    120 rows were held out. That spread is the reason the interval has the
    width it does; it is measuring how much of the score is the fold rather
    than the model.

    rf scored 0.0141 below linear on average, but the two swapped places by
    ±0.0244 from fold to fold. The gap is smaller than the disagreement about
    the gap, which is what p=0.589 is reporting.

    Back-of-envelope: separating them at this effect size would take roughly
    3,200 rows, against the 600 here. That assumes the difference you measured
    is real and that error falls with the square root of the sample — both
    optimistic, so read it as an order of magnitude, not a target.

Four things, all computed from this run rather than looked up:

  • What the features bought, as a gap over the dummy baseline rather than a headline score.
  • Why the interval is that wide — the real fold-to-fold spread, and how many rows each fold held out to produce it.
  • Why two tied models could not be separated — the measured gap set against the measured disagreement about that gap, which is what the p-value reports.
  • Roughly how many rows would separate them, from this run's own effect size.

The line it does not cross

Every sentence states a fact about the result or arithmetic on it. "Separating these would take roughly 3,200 rows" is a calculation you act on however you like; "you should collect more data" would be michi making your decision. A test scans the generated notes for advice-shaped phrasing, so the distinction is enforced rather than merely intended.

The sample-size figure is labelled back-of-envelope where it appears — it assumes the measured difference is real and that error falls with √n, both optimistic. An unqualified number gets quoted.

Sections with nothing specific to say are omitted: identical fold scores get no paragraph explaining a spread of zero, and only one tie is worked through, because one example teaches and five are a wall of text.

Install: pip install komichi

Assets 2
Loading