Skip to content

Releases: undercutsh/firstpass

v0.2.0 — Undercut: measured routing, public benchmarks, live site

Choose a tag to compare

@justinwinter justinwinter released this 18 Aug 09:12

Rebranded to Undercut — repo moved to undercutsh/firstpass, install is npx skills add undercutsh/firstpass.

What's in this release

  • probe policy (shipped) — format-strict work starts cheap (model-agnostic), caps at standard. Replaces a vendor-specific rule that round-5 cross-vendor testing proved brittle.
  • Public-benchmark validation — GSM8K + HumanEval, official test cases, raw results published in testing/results/.
  • Landing page — getundercut.sh (Vercel + Cloudflare DNS), canonical to the full domain.
  • Flagging-reliability test — stock dispatchers route 100% of units correctly despite ~90% flag accuracy; no custom flagging model needed.
  • CI, CONTRIBUTING, issue/PR templates, trust backlog, verified models.md.

Measured headline (public benchmarks, official test cases)

  • Up to −71% cost on GSM8K (OpenAI), equal-or-better pass
  • Up to −95% cost on HumanEval (Gemini), identical pass
  • Tiered never worse on any synthetic-suite cell; better on 6 of 12

See testing/README.md for full methodology and caveats.