Skip to content

v1.4.0 — the last mile: tune, fit, predict

Choose a tag to compare

View all tags
@cattolatte cattolatte released this 27 Jul 11:55
v1.4.0
6e42040

michi could compare models and explain the comparison, then stopped one step before the thing anyone actually hands in. A student at a hackathon could choose a model with michi and could not produce a submission with it. Three verbs close that.

The whole loop, now

michi inspect  train.csv --target target --explain
michi clean    train.csv --target target -o recipe.yaml
michi bench    train.csv --target target --recipe recipe.yaml --models linear,rf,xgb --explain
michi tune     train.csv --target target --recipe recipe.yaml --model hist-gbm --save-params best.yaml
michi fit      train.csv --target target --recipe recipe.yaml --model hist-gbm --params best.yaml -o model.joblib
michi predict  model.joblib test.csv --recipe recipe.yaml --id id --proba -o submission.csv

michi tune — hyperparameter search

The space is printable before anything runs, and replaceable with your own YAML:

michi tune --model rf --list-space

Random, successive-halving, and grid strategies. Built-in spaces for twelve models.

The search is nested or it is a lie. Configurations are chosen by cross-validation inside each training fold; the reported score comes from folds the search never touched.

                           balanced_accuracy
  ──────────────────────────────────────────
  tuned (held-out folds)              0.8891
  defaults (same folds)                0.869
  search's own best                   0.8943

  Verdict  tuning gained 0.02019 of balanced_accuracy over the defaults,
  measured on folds the search never saw.
  The search's own best score was 0.00519 better than the held-out result.
  That gap is what reporting an inner score as performance would have hidden.

Most tools print the third number and call it performance. michi prints it next to the honest one, because the gap is the lesson — and reporting an inner search score as performance is the second most common silent leak in tabular ML, after target encoding.

michi fit — train and save

Trains one model on every row and writes a joblib file. Reports no accuracy on purpose: a score measured on the rows a model trained on is the most confidently wrong number a tool can print.

michi predict — the file you submit

Needs no label column. Writes CSV, TSV, or parquet with an optional id column and class probabilities.

id,prediction,proba_0,proba_1
0,1,0.0112,0.9888
1,0,0.8517,0.1483

Deep learning works here

predict accepts anything eval does, including module:object:

michi predict model.joblib test.csv -o preds.csv   # sklearn
michi predict mynet:model  test.csv -o preds.csv   # PyTorch, TF, ONNX, yours

michi calls predict/predict_proba and never inspects what is behind them, so a torch module wrapped in a class works identically to a random forest — with no per-framework loader for michi to maintain and get wrong. Verified against a non-sklearn model through the protocol.

tune --save-params writes YAML that fit --params reads, so the two hand off without you joining them by hand. A test asserts the round trip.

Install: pip install komichi

Assets 2
Loading