Skip to content

v1.10.0 — diff, threshold, errors

Choose a tag to compare

View all tags
@cattolatte cattolatte released this 27 Jul 13:15
v1.10.0
67d9f81

Three questions michi couldn't answer, each of which someone was answering by hand with the same twenty lines of pandas.

michi diff — has this data changed?

michi diff baseline.profile.json production.csv --fail-on high
  high   cabin    1 column(s) present in the baseline are gone: cabin
  warn   salary   salary mean moved +5.35 baseline SD (3.936e+04 → 7.478e+04)
  warn   region   region has value(s) absent from the baseline: NORTH_NEW
  info   extra    1 new column(s): extra

The profile artifact was always half of this — inspect has written schemas and distributions since v0.1, so a drift check needs no new measurement. That's exactly why it belongs in michi rather than in a monitoring service michi would have to run.

Severity follows breakage, not effect size: a removed column is high whatever the distributions did; a new column is only information, because a fitted model never asked for it. Shift is measured in baseline standard deviations, so one threshold means the same thing for dollars and millimetres.

michi threshold — the cutoff nobody chose

A classifier emits a probability. Almost every tool turns it into a label at 0.5, silently. On imbalanced data, or when a miss costs more than a false alarm, that's just wrong.

michi threshold model.pkl data.csv --target churned --cost fn=10,fp=1 --objective cost
      cutoff   precision   recall      F1   flagged   missed   false alarms    cost
  ─────────────────────────────────────────────────────────────────────────────────
  ▸     0.16       0.774    0.996   0.871       305        1             69      79
        0.50       0.936    0.924   0.930       234       18             15     195

0.16, not 0.50 — and 0.50 costs 195 against 79. michi marks the best cutoff for an objective you named, never one it picked. Whether a miss really is worth ten false alarms isn't something michi can know.

michi errors — the rows behind the score

  33 of 600 rows wrong (5.5%)  ·  ordered by how sure the model was

  sure   predicted   age   salary      region   fare
   84%   0           30    —           south    11.7
   82%   1           59    4.562e+04   west     1e+04

  Where they concentrate
    · region = 'west': 10.7% wrong (6 of 56), against 5.5% overall

Ordered by confidence, because that separates the two kinds: a confident error is a mislabelled row, a leak, or a region the features don't describe — an unsure one is the model behaving correctly at the edge. Patterns are described, never diagnosed: michi sees that the west region fails twice as often and cannot see why. --out writes every mistake.

602 tests. Install: pip install komichi

Assets 2
Loading