Skip to content

v0.106.1: Trustworthy class-ordered-data warning

Choose a tag to compare

@mrothroc mrothroc released this 28 Aug 23:37
· 134 commits to main since this release

Classification label-diversity diagnostic

Mixlab warns when classification records look class-ordered within shards, a
condition that balanced global class_counts cannot reveal. The check only
fired when every sampled batch was single-label, so one batch straddling a
shard boundary cleared the flag and silenced the warning for the rest of the
run — and real class-ordered corpora cross a boundary every few batches, so the
diagnostic was quietest exactly when it mattered.

It now scores the mean distinct-label count across the sample and reports
mean/min/max, so occasional two- or three-label boundary batches no longer hide
near-degenerate ordering.

The threshold measures diversity above the one-label floor every batch
necessarily has, 1 + 0.25*(achievable - 1). A plain "25% of achievable" rule
demands both labels in every binary batch, which correctly shuffled data misses
by chance: measured over 400 seeded runs, that fired on 86% of 5%-minority
binary corpora and 74% of balanced binary at four rows per batch.

Low batch diversity has two causes the count alone cannot separate —
class-ordered records, or one class dominating the corpus — so the warning now
names both rather than asserting the first.

Other Changes

  • chore: make lint now pins the same Go toolchain as the pre-commit hook,
    so the two stop disagreeing; the go-vet-only fallback announces itself
    instead of reporting "all checks passed" when golangci-lint is absent.
  • docs: corrections and additions across the module CLAUDE.md files.

Install

brew tap mrothroc/mixlab https://github.com/mrothroc/mixlab
brew install mixlab

Or build from source:

CGO_ENABLED=1 go build -tags mlx -o mixlab ./cmd/mixlab/