Skip to content

v0.9.0 - Measured label search

Choose a tag to compare

@strot-ai strot-ai released this 13 Oct 16:48
· 14 commits to main since this release

The compiler measures instead of guessing.

The v0.8.1 semantic benchmark exposed a failure mode: synthesized routing labels beat bare class names on AG News (+14pp) but LOST on Emotion (-8.3pp), because the class names were already optimal and extra wording diluted them. The compiler was guessing at wording with no way to know when to leave well enough alone.

Measured label search

compile() now scores candidate labelings - the synthesized labels AND the bare registry keys - on the teacher-labeled sample, through the actual artifact, and bakes the winner:

  • Free when target_risk is set (shares the single teacher-labeling call); opt-in alone via measure_labels=True
  • A winner that breaks the golden gate is rejected (goldens stay a regression gate, not an optimizer)
  • Scores recorded in label_search metadata and the episode log; the flag is part of the cache key

Validated live (real model, 300 held-out test records each)

Dataset Naive Compiled (v0.8.1) Measured compile (v0.9.0)
Emotion 93.3% 85.0% 93.3% - fully recovered (winner: registry keys)
AG News 72.7% 86.7% 76.7% (winner: synthesized; weaker synthesis draw)

Honest limits, documented in benchmarks/: measurement bounds the downside (floor near the best available candidate) but does not eliminate synthesis variance, and a 25-point sample is a noisy judge. Multiple synthesis candidates and larger measurement samples are the known next steps.

Also in this release: semantic-text benchmark published with all results including the losses; developer guide updated.

93 tests pass.