Skip to content

v1.3.0

Latest

Choose a tag to compare

@BHUPESH003 BHUPESH003 released this 06 Sep 07:10

Reproducible retrieval benchmark

  • npm run eval:report runs matha's existing golden-set eval standalone against a naive substring-match baseline (matha's own pre-scoring behaviour), and writes BENCHMARKS.md.
  • Real, measured lift over the naive baseline on the in-house fixture: recall@5 0.846 → 0.957, precision@5 0.694 → 0.797.
  • Deterministic and reproducible — the fixture clock is fixed, so recall/precision are exactly reproducible on any machine.
  • Not a public academic benchmark claim: there's no published dataset for "retrieve the right decision for a given file + intent," so the report says so directly rather than compare against an unrelated task.

No breaking changes — .matha/ schema unchanged since v1.2.0.