Repository navigation
v0.46.0
Stageflow release notes
0.46.0
Judge configuration, calibration against your labels, and console workflows to improve grader reliability.
Features
- Judge settings —
rubricandgoal_reachedgraders acceptjudge_settings(temperature,reasoning,votes) per grader or indefaults, merged key by key. Withvotesof 3 or 5, each judge runs that many times and the majority wins. Validation warns whenvotes> 1 at temperature 0 (eval.judge_votes_no_variation). See Evals. - Calibrate judges against your labels — Label judged results Pass or Fail, then measure agreement (kappa, TPR/TNR, confusion matrix) on a stable 70/30 split of tuning vs fair-test rows, with optional stability testing. Calibration replays stored outputs (no stage re-runs). New:
sf eval calibrate,calibrate_eval_graderand calibration-list MCP tools, andPOST /api/evals/:id/graders/:grader/calibrate. Migration 018 addseval_grader_labelsandeval_calibrations. Runs can be cancelled (EVAL_CALIBRATION_CANCELLED). - Tune from mistakes — Suggest worked examples from judge disagreements on tuning rows, or try alternate rubric wording and validate on fair-test rows before saving.
- Console: label queue and guided calibration — Full-screen label queue, guided “Is this judge reliable?” panel, Teach and Wording dialogs, “Needs my label” filter on Cases, and “Label more like this” in the case drawer.
Changed
- Guide playbook — One sentence beside every confirmation card; declines outside SQL/code; names console building blocks; states missing features plainly; notes when file instructions were ignored.
- Internal-agent eval snapshots include graders so the Cases grader filter works for Guide evals.
0.45.0
Console keyboard fixes and clearer GitHub PR trigger semantics.
Fixes
- Console shortcuts work — ⌘, opens Settings; T on Providers tests connections; Tab in the command palette cycles result type; ⌘↵ on run-detail gate sends the note with Accept; ⌘S in Workshop saves while the chat box is focused.
- Search fields keep caret keys — In the command palette and model search, Option/Command/Control/Shift with arrows and Home/End move the cursor; plain Up/Down still move the highlighted row.
Changed
- GitHub PR trigger actions — Normalized
actionisopened|synchronize|updated|merged|closed.updatedno longer covers pushes; usesynchronizewhen the PR head SHA changes.event.match.actionis required forgithub.*sources. See Triggers. - GitHub poller — Upgrade no longer false-fires
opened. Missingheadswith an existing cursor backfills quietly;openedrequires equal created/updated timestamps or a PR created after the cursor, not merely an unknown prior SHA.