v1.11.0 — text, time, out-of-fold, calibration

Five gaps, all of them places a practitioner had to leave michi to do something ordinary.

Text

A free-text column could previously only be dropped or one-hot encoded — useless for free text.

michi clean reviews.csv --target rating --text-length note --tfidf note

Op	What it does
text-length	Characters and words. Often most of the signal — how much someone wrote predicts more than any single word
tfidf	Term frequencies. Fitted, so it runs inside the fold where the test rows can't vote on which words exist

Time

michi clean sales.csv --target churned \
  --order-by date --lag revenue=1 --rolling revenue=7:mean

Both require --order-by, because "earlier" needs a definition and michi won't guess a sort column. Both accept a group so one entity's history never leaks into another's.

Both are classified deterministic despite depending on other rows — which looks wrong until you see why: they read only earlier rows in a stated order, so no future value can reach a past one. That's exactly the property that makes them safe outside the fold.

bench --oof

Out-of-fold predictions, one column per model:

purchased,linear,rf,dummy
1,1.0,1.0,0.0

Every value was predicted by a fold that did not train on that row. That's what makes them safe to stack on — and what separates them from predictions of the training data, which would give a meta-model the base models' memory rather than their generalisation.

bench --balance

Weights classes by inverse frequency where the estimator accepts it. A model without class_weight isn't an error — the request simply doesn't apply, and saying so would be noise on a leaderboard where other models did honour it.

fit --calibrate

Closes a loop open since v0.2: eval could report that a model says 0.9 and is right 0.7 of the time, and offered no way to fix it.

michi fit data.csv --target churned --calibrate isotonic -o model.joblib

Fitted by internal cross-validation, so the score→probability mapping is learned on folds the base model didn't train on. isotonic is flexible and wants more data; sigmoid assumes a shape and survives small samples.

Fixed

• bench --oof was declared and never threaded into run_benchmark — it silently wrote nothing. Found by running it, now five for five on how the real bugs here surface.
• The roadmap listed its 1.x milestones in descending order after 1.2, and still named deep-learning training loops a deliberate non-goal six releases after torch-mlp shipped.

---

618 tests. Install: pip install komichi

v1.10.0 — diff, threshold, errors

Three questions michi couldn't answer, each of which someone was answering by hand with the same twenty lines of pandas.

michi diff — has this data changed?

michi diff baseline.profile.json production.csv --fail-on high

  high   cabin    1 column(s) present in the baseline are gone: cabin
  warn   salary   salary mean moved +5.35 baseline SD (3.936e+04 → 7.478e+04)
  warn   region   region has value(s) absent from the baseline: NORTH_NEW
  info   extra    1 new column(s): extra

The profile artifact was always half of this — inspect has written schemas and distributions since v0.1, so a drift check needs no new measurement. That's exactly why it belongs in michi rather than in a monitoring service michi would have to run.

Severity follows breakage, not effect size: a removed column is high whatever the distributions did; a new column is only information, because a fitted model never asked for it. Shift is measured in baseline standard deviations, so one threshold means the same thing for dollars and millimetres.

michi threshold — the cutoff nobody chose

A classifier emits a probability. Almost every tool turns it into a label at 0.5, silently. On imbalanced data, or when a miss costs more than a false alarm, that's just wrong.

michi threshold model.pkl data.csv --target churned --cost fn=10,fp=1 --objective cost

      cutoff   precision   recall      F1   flagged   missed   false alarms    cost
  ─────────────────────────────────────────────────────────────────────────────────
  ▸     0.16       0.774    0.996   0.871       305        1             69      79
        0.50       0.936    0.924   0.930       234       18             15     195

0.16, not 0.50 — and 0.50 costs 195 against 79. michi marks the best cutoff for an objective you named, never one it picked. Whether a miss really is worth ten false alarms isn't something michi can know.

michi errors — the rows behind the score

  33 of 600 rows wrong (5.5%)  ·  ordered by how sure the model was

  sure   predicted   age   salary      region   fare
   84%   0           30    —           south    11.7
   82%   1           59    4.562e+04   west     1e+04

  Where they concentrate
    · region = 'west': 10.7% wrong (6 of 56), against 5.5% overall

Ordered by confidence, because that separates the two kinds: a confident error is a mislabelled row, a leak, or a region the features don't describe — an unsure one is the model behaving correctly at the edge. Patterns are described, never diagnosed: michi sees that the west region fails twice as often and cannot see why. --out writes every mistake.

---

602 tests. Install: pip install komichi

v1.9.0 — split, and column importance

Both of these exist because a default lies.

michi split

A random split is right until the rows aren't independent — and then it's confidently wrong.

michi split data.csv --target churned --group customer_id
michi split data.csv --time signup_date --test-size 0.25

Strategy	When
stratified	Default for a classification target — an unstratified split on imbalanced data can starve a fold
group	Rows share an entity. Four of a customer's rows in training and one in test isn't generalisation, it's memory
time	A series. A test set drawn from before the training rows asks the model to predict the past
random	Everything else — and it now says plainly what it cannot promise

The summary states the property the split was chosen to give, then verifies it:

  No value of customer_id appears on both sides — the leak a random split
  would have caused is absent.

If a group does span both sides, it says so loudly rather than printing a guarantee that didn't hold.

eval --importance

  column        drop when shuffled         ±
  ─────────────────────────────────────────────────────────
  age                      +0.2932   0.00899
  salary                     +0.16    0.0118
  region                  +0.04764    0.0092
  signup_date            +0.000405   0.00282   within noise

  What this model uses, not what matters: a column it ignores may still drive
  the outcome, and two correlated columns split the credit between them.

Measured through predict and nothing else — so a PyTorch network and a random forest are measured identically, with no per-model introspection for michi to keep working forever.

Each column is shuffled several times and the spread reported, because an importance smaller than its own noise is not a finding. Those rows are marked rather than hidden: a column measured as unimportant is a result.

The wording is load-bearing. Reporting a bare ranking has talked people into deleting a feature that mattered, so the caveat sits under the table rather than in the docs.

Also lands as a chart in michi ui, and 分 split joins the CLI stage map.

Fixed

--importance was declared as a flag and never passed to the evaluator — it silently did nothing. Caught by running it, not by a test.

Install: pip install komichi

v1.8.0 — bayesian search

tune sampled at random, halved, or exhausted a grid — all three treat every configuration as equally worth trying. A model-based optimiser does something different in kind: it fits a model of the objective and proposes the next configuration from its own beliefs. A thing that forms opinions and acts on them, inside a tool whose first principle is that it does not.

pip install 'komichi[bayes]'
michi tune data.csv --target y --model hist-gbm --strategy bayes --save-params best.yaml

Why it's allowed — ADR-0004

The obvious argument for ("finds better hyperparameters faster") is true and insufficient — AutoML also finds better everything faster.

The argument that decides it is narrower. Ask what the optimiser has an opinion about. Not the model. Not the space. Not the metric, when to stop, or what to ship. It chooses the order in which candidates from your own space are evaluated — the same category of decision as which rows land in which fold. A mechanic, not a judgement.

The hazard is the opposite of the obvious one

A model-based search is better at overfitting the inner folds. That's the mechanism working, not failing: it spends its budget near whatever looked good on the inner split, including where that split was lucky.

So the optimism gap grows with the optimiser's strength. Measured on the example dataset:

Strategy	Gain over defaults	Optimism gap
random	0.02019	0.00519
bayes	0.02303	0.00828

It found the better configuration and flattered itself more — exactly as the ADR reasoned. A tool reporting the inner score gets more wrong the better its search gets, which is the worst possible direction for an error to move. michi already printed both numbers adjacently; that decision is now load-bearing rather than merely correct.

Five constraints, each tested

1. The space stays printable (--list-space) and the optimiser may not widen it
2. Nested scoring is not configurable — there's no flag to report the flattering number
3. The optimism gap is always shown
4. Reproducible under a seed
5. A missing dependency names the extra and never falls back silently — asking for bayes and getting random would compare two runs that were never the same experiment

Fixed

Counting evaluated configurations assumed sklearn's cv_results_["params"]. Optuna's wrapper has no such key — it records trials_. The first Bayesian search to finish a fold raised KeyError. Found by running it, not by a test.

Install: pip install 'komichi[bayes]'

v1.7.0 — charts in the viewer

The viewer was ninety-nine lines listing runs. Every manifest michi has written since v0.2 carried a confusion matrix, calibration bins, an ECE, per-subgroup scores, and ten metrics with intervals — and none of it was ever drawn.

A terminal can rank models and print a confusion table. It cannot draw a reliability diagram, and that's the whole reason the viewer exists.

pip install 'komichi[ui]'
michi ui

Four charts

Chart	What it answers
Metrics with intervals	Which numbers this dataset actually pinned down. A wide bar is a metric you shouldn't quote to three decimals.
Confusion matrix	Which errors, not how many. Shaded by share of each true class.
Calibration	Whether a probability means what it says, with the expected calibration error beside it.
Score by subgroup	The group the average hides. Worst-first, only the worst one coloured.

Every chart renders from the manifest and computes nothing — a chart that recomputed a number could disagree with the terminal, and then two of michi's surfaces would describe the same run differently.

Inline SVG throughout: no plotting library at render time, no CDN, no JavaScript. The viewer still works air-gapped. A chart that can't be drawn honestly isn't drawn — past ten classes the grid is unreadable, so the table is shown instead.

Three faults found by looking at the page

• The charts were invisible. They shipped with a hardcoded near-black for text and rules — exactly the viewer's background colour. Text and rules now inherit the page's own colour, so one chart is legible in both the dark viewer and a light printed report. A test asserts it for all four.
• Confusion shading was per matrix. On an imbalanced problem that makes the majority class the only visible cell, hiding the failure the chart is read to find. Now per row.
• Subgroup and metric labels collided with their own bars.

Also

michi --help gains a stage map naming each verb by the kanji for what it does:

道 the path — 見 inspect · 整 clean · 比 bench · 確 eval · 探 tune · 作 fit · 記 report · 出 export

And the CLI module docstring stopped claiming the console "will become" available in v0.5.

Install: pip install 'komichi[ui]'

v1.6.0 — ensemble

Every tool will report an ensemble's score. Almost none report it beside the models it combines — which is the only comparison that answers whether combining bought you anything.

michi ensemble data.csv --target churned --models linear,rf,hist-gbm

  model      balanced_accuracy      95% interval   vs leader                     fit
  ──────────────────────────────────────────────────────────────────────────────────
  linear                 0.884   0.8402 – 0.9285   leader                       0.0s
  rf                     0.884    0.8144 – 0.954   tied with leader (p=0.99)    0.5s
  ensemble               0.881   0.8368 – 0.9261   tied with leader (p=0.667)   7.1s
  hist-gbm               0.869    0.807 – 0.9309   tied with leader (p=0.478)   1.0s
  dummy                    0.5         0.5 – 0.5   worse (p=0.0071)             0.0s

That output is the feature. The stack tied its best member and cost 7.1 seconds against 0.0. Reported alone, it would have looked like a result.

Two ways to combine

--method	What it does
stack (default)	Trains a meta-learner on out-of-fold member predictions. --final chooses it.
vote	Averages members. Soft voting where every member can express confidence, hard voting otherwise.

Stacking leaks unless the meta-learner is fed out-of-fold predictions. sklearn's stacking estimators cross-validate internally to build the meta-features, and michi wraps that in its own outer folds — so the reported number comes from data neither layer was fitted on.

Each member carries its own preparation: standardisation is right for a linear model and pointless for a tree, and sharing one pipeline would impose one model's needs on all of them.

What it will not do

michi picks no members, prunes none for scoring poorly, and weights none by validation score. Those are modelling judgements, and a tool that makes them silently is an AutoML system wearing a different hat.

---

An ensemble is assembled from what you named, so it joins the catalogue for the duration of one command and leaves cleanly — including when the run raises. That's why run_benchmark could treat it as an ordinary model, and why nothing downstream needed a branch asking whether a row is an ensemble.

Install: pip install komichi

v1.5.0 — neural networks, and the scaling bug they exposed

A training loop is part of the toolkit. michi already owned one — bench, tune, and fit all call .fit(), and for hist-gbm that's 500 boosting iterations with early stopping. But the catalogue had fourteen models and not one network, so the loop an engineer actually retypes was the one michi didn't write.

Two networks, no new architecture

michi bench data.csv --target y --models rf,hist-gbm,mlp     # no extra install
michi tune  data.csv --target y --model mlp --save-params best.yaml
michi fit   data.csv --target y --model mlp --params best.yaml -o model.joblib

Model	Needs	What it is
mlp	nothing	Feed-forward net on scikit-learn's loop
torch-mlp	pip install 'komichi[torch]'	Epochs, Adam, mini-batches, validation split, early stopping, best-weight restore

Both are ordinary catalogue entries. Because they satisfy fit/predict, every verb already works on them with no special case anywhere in michi — a network is cross-validated, significance-tested, and held against the dummy baseline exactly like a random forest. michi never presents "we used deep learning" as a result.

tune --list-space --model torch-mlp shows layer sizes, dropout, learning rate, and batch size as what they are: hyperparameters, not defaults michi picked for your data.

The bug the networks found

mlp scored exactly the dummy baseline on a dataset where linear regression scored 0.89. The cause wasn't the network.

When a recipe names a column, transformer_specs replaced michi's handling of it — including the standardisation scale-sensitive models depend on. An imputed salary reached the estimator in the tens of thousands while every other feature sat near zero.

This was never only about neural networks. linear, ridge, lasso, knn, and svm were all quietly degraded whenever a recipe had a fitted step, and had been since recipes and bench were first composed.

	Before	After
Max feature magnitude	56,542	7.0
mlp balanced accuracy	0.5000	0.8757

The recipe still decides what happens to a column; scaling is appended after it rather than instead of it. Two tests pin it — one that a claimed column arrives scaled, one that a scale-invariant model is still left alone, because the fix must not start standardising for trees.

Where the line is now

"No training loops" was too broad and the roadmap now says so properly. What michi still won't do is ship a per-architecture zoo: convolutional and sequence models need data michi doesn't model, and that would turn the catalogue into a framework. Bring those through module:object or a plugin — eval and predict already accept them.

Install: pip install komichi · with networks: pip install 'komichi[torch]'

v1.4.0 — the last mile: tune, fit, predict

michi could compare models and explain the comparison, then stopped one step before the thing anyone actually hands in. A student at a hackathon could choose a model with michi and could not produce a submission with it. Three verbs close that.

The whole loop, now

michi inspect  train.csv --target target --explain
michi clean    train.csv --target target -o recipe.yaml
michi bench    train.csv --target target --recipe recipe.yaml --models linear,rf,xgb --explain
michi tune     train.csv --target target --recipe recipe.yaml --model hist-gbm --save-params best.yaml
michi fit      train.csv --target target --recipe recipe.yaml --model hist-gbm --params best.yaml -o model.joblib
michi predict  model.joblib test.csv --recipe recipe.yaml --id id --proba -o submission.csv

michi tune — hyperparameter search

The space is printable before anything runs, and replaceable with your own YAML:

michi tune --model rf --list-space

Random, successive-halving, and grid strategies. Built-in spaces for twelve models.

The search is nested or it is a lie. Configurations are chosen by cross-validation inside each training fold; the reported score comes from folds the search never touched.

                           balanced_accuracy
  ──────────────────────────────────────────
  tuned (held-out folds)              0.8891
  defaults (same folds)                0.869
  search's own best                   0.8943

  Verdict  tuning gained 0.02019 of balanced_accuracy over the defaults,
  measured on folds the search never saw.
  The search's own best score was 0.00519 better than the held-out result.
  That gap is what reporting an inner score as performance would have hidden.

Most tools print the third number and call it performance. michi prints it next to the honest one, because the gap is the lesson — and reporting an inner search score as performance is the second most common silent leak in tabular ML, after target encoding.

michi fit — train and save

Trains one model on every row and writes a joblib file. Reports no accuracy on purpose: a score measured on the rows a model trained on is the most confidently wrong number a tool can print.

michi predict — the file you submit

Needs no label column. Writes CSV, TSV, or parquet with an optional id column and class probabilities.

id,prediction,proba_0,proba_1
0,1,0.0112,0.9888
1,0,0.8517,0.1483

Deep learning works here

predict accepts anything eval does, including module:object:

michi predict model.joblib test.csv -o preds.csv   # sklearn
michi predict mynet:model  test.csv -o preds.csv   # PyTorch, TF, ONNX, yours

michi calls predict/predict_proba and never inspects what is behind them, so a torch module wrapped in a class works identically to a random forest — with no per-framework loader for michi to maintain and get wrong. Verified against a non-sklearn model through the protocol.

---

tune --save-params writes YAML that fit --params reads, so the two hand off without you joining them by hand. A test asserts the round trip.

Install: pip install komichi

v1.3.0 — target encoding, and a menu for choosing what to engineer

Target encoding

Replacing a category with the target's mean is the strongest encoding available for a high-cardinality column, and the easiest way to destroy a model. Encoded naively, a unique id maps one-to-one onto its own label.

Measured on 600 random labels against 600 unique ids — data containing no signal whatsoever:

Encoding	Cross-validated accuracy	Truth
Naive (fit on everything)	1.000	0.500
michi (out of fold)	0.517	0.500

The naive number isn't a bug in the model; it's the model faithfully reporting that it memorised the answer key. A test asserts both halves of that table, because a trap is only instructive if you can watch it spring.

michi clean data.csv --target churned --target-encode city,postcode

michi ships its own encoder rather than scikit-learn's. sklearn's TargetEncoder is not reproducible on any version michi supports — the parameter that made it deterministic is deprecated in 1.9 and removed in 1.11, and the replacement needs a newer floor than >=1.4. Bumping the floor to fix one operation would have dropped users. export writes the encoder class into the generated file rather than importing it, so exported pipelines still depend on pandas and scikit-learn alone, and you can read exactly what the encoding did.

The feature menu

The five operations added in 1.1 were reachable only by flag. The wizard walks findings, and nothing is wrong with a column that could be logged — so no finding ever raised it, and the interactive path couldn't reach feature engineering at all.

michi clean now follows the findings triage with a column picker:

  Feature engineering — optional, and michi has no opinion here.
  Nothing is preselected; space to pick, enter to move on.

  [2/4]  Compress a long tail
         These columns are skewed enough that a few large values dominate
         every distance and every coefficient.
  ? Which columns?
   ◯ fare   (skew +6.87, 467 distinct)
   ◯ salary (skew +0.31, 511 distinct)

An operation appears only where the shape fits: datepart with timestamps, log when something is actually skewed, target-encode when a categorical column is too wide for one-hot. michi lists shapes and stays quiet about worth — whether a product term helps your problem is domain knowledge michi doesn't have, and a preselected checkbox would be a recommendation wearing a checkbox.

Also

• examples/sweep.yaml was the last file in the repo calling itself an example without ever having been run — and it named a target the example dataset doesn't have. It's now the verified plan: 12 cells in 6.1s, with the ranked output in the examples README. The seeds make the point the whole project rests on: linear spans 0.8907–0.8988 across four seeds and rf spans 0.8779–0.8967, so the gap between the best of each is smaller than the spread within either.
• The recipe file header had listed seven operations since v1.0. It now lists all thirteen, and the leakage note names every fitted step.

---

The scikit-learn floor stays at >=1.4. Install: pip install komichi

v1.2.0 — teaching mode

bench --explain used to print generic prose about each check kind — useful once, useless after. A paragraph on what a confidence interval is does not tell you why yours is that wide.

It now explains the run that just finished, using the folds that actually ran.

  Reading these numbers

    The dummy baseline scores 0.5 without looking at a single feature — it is
    what you get for free. linear scores 0.892, so the features are worth
    0.392 of balanced_accuracy here. That gap, not the headline score, is what
    the modelling bought.

    linear scored between 0.87 and 0.9338 across the 5 folds — a spread of
    0.06383 on the same data and the same settings, differing only in which
    120 rows were held out. That spread is the reason the interval has the
    width it does; it is measuring how much of the score is the fold rather
    than the model.

    rf scored 0.0141 below linear on average, but the two swapped places by
    ±0.0244 from fold to fold. The gap is smaller than the disagreement about
    the gap, which is what p=0.589 is reporting.

    Back-of-envelope: separating them at this effect size would take roughly
    3,200 rows, against the 600 here. That assumes the difference you measured
    is real and that error falls with the square root of the sample — both
    optimistic, so read it as an order of magnitude, not a target.

Four things, all computed from this run rather than looked up:

• What the features bought, as a gap over the dummy baseline rather than a headline score.
• Why the interval is that wide — the real fold-to-fold spread, and how many rows each fold held out to produce it.
• Why two tied models could not be separated — the measured gap set against the measured disagreement about that gap, which is what the p-value reports.
• Roughly how many rows would separate them, from this run's own effect size.

The line it does not cross

Every sentence states a fact about the result or arithmetic on it. "Separating these would take roughly 3,200 rows" is a calculation you act on however you like; "you should collect more data" would be michi making your decision. A test scans the generated notes for advice-shaped phrasing, so the distinction is enforced rather than merely intended.

The sample-size figure is labelled back-of-envelope where it appears — it assumes the measured difference is real and that error falls with √n, both optimistic. An unqualified number gets quoted.

Sections with nothing specific to say are omitted: identical fold scores get no paragraph explaining a spread of zero, and only one tie is worked through, because one example teaches and five are a wall of text.

---

Install: pip install komichi