Skip to content

v0.46.0

Choose a tag to compare

@github-actions github-actions released this 29 Oct 03:57
9bab26c

Stageflow release notes

0.46.0

Judge configuration, calibration against your labels, and console workflows to improve grader reliability.

Features

  • Judge settings — rubric and goal_reached graders accept judge_settings (temperature, reasoning, votes) per grader or in defaults, merged key by key. With votes of 3 or 5, each judge runs that many times and the majority wins. Validation warns when votes > 1 at temperature 0 (eval.judge_votes_no_variation). See Evals.
  • Calibrate judges against your labels — Label judged results Pass or Fail, then measure agreement (kappa, TPR/TNR, confusion matrix) on a stable 70/30 split of tuning vs fair-test rows, with optional stability testing. Calibration replays stored outputs (no stage re-runs). New: sf eval calibrate, calibrate_eval_grader and calibration-list MCP tools, and POST /api/evals/:id/graders/:grader/calibrate. Migration 018 adds eval_grader_labels and eval_calibrations. Runs can be cancelled (EVAL_CALIBRATION_CANCELLED).
  • Tune from mistakes — Suggest worked examples from judge disagreements on tuning rows, or try alternate rubric wording and validate on fair-test rows before saving.
  • Console: label queue and guided calibration — Full-screen label queue, guided “Is this judge reliable?” panel, Teach and Wording dialogs, “Needs my label” filter on Cases, and “Label more like this” in the case drawer.

Changed

  • Guide playbook — One sentence beside every confirmation card; declines outside SQL/code; names console building blocks; states missing features plainly; notes when file instructions were ignored.
  • Internal-agent eval snapshots include graders so the Cases grader filter works for Guide evals.

0.45.0

Console keyboard fixes and clearer GitHub PR trigger semantics.

Fixes

  • Console shortcuts work — ⌘, opens Settings; T on Providers tests connections; Tab in the command palette cycles result type; ⌘↵ on run-detail gate sends the note with Accept; ⌘S in Workshop saves while the chat box is focused.
  • Search fields keep caret keys — In the command palette and model search, Option/Command/Control/Shift with arrows and Home/End move the cursor; plain Up/Down still move the highlighted row.

Changed

  • GitHub PR trigger actions — Normalized action is opened | synchronize | updated | merged | closed. updated no longer covers pushes; use synchronize when the PR head SHA changes. event.match.action is required for github.* sources. See Triggers.
  • GitHub poller — Upgrade no longer false-fires opened. Missing heads with an existing cursor backfills quietly; opened requires equal created/updated timestamps or a PR created after the cursor, not merely an unknown prior SHA.