Skip to content

Releases: mupt-ai/self-bench

Production: Fuller Card Previews, Public Results API, and BYOK Costs

Choose a tag to compare

@bjamhong bjamhong released this 05 Oct 20:31
6efe20f

Deploys main at 6efe20f.

  • #304 Prompt hovered cards' readers to open the page, fill the preview, and open repository pages side by side
  • #303 Add Mintlify docs and public results API
  • #301 Count BYOK provider charges in gateway generation costs

Production: Confidence Intervals on Results Charts and Verified Task Images

Choose a tag to compare

@avyayv avyayv released this 05 Oct 03:39
3057a78

Deploys main at 3057a78.

  • #298 Show 95% confidence intervals on the results charts
  • #276 Start trials from verified task images
  • #296 Use gateway-reported charges for evaluation costs when available
  • #297 Move one-off operator tools out of live source
  • #295 Simplify Results and empty states

Production: Results Charts Without Error Bars and E2B Plan Errors

Choose a tag to compare

@avyayv avyayv released this 05 Oct 04:26
1d033ba

Deploys main at 1d033ba.

  • #302 Remove confidence-interval error bars from the results charts
  • #300 Say when an E2B key's plan is too small for 24-hour sandboxes

Production: Temporal Workers on GKE and Dari Routing for Evaluations

Choose a tag to compare

@avyayv avyayv released this 03 Oct 04:56
e61843e

Deploys main at e61843e.

  • #293 Run every Temporal worker on GKE, on nodes system pods can't preempt
  • #294 Allow Dari Routing for production evaluations
  • #290 Keep the overlapping-lists test from depending on how fast CI starts requests
  • #288 Set up gcloud once in prod deploys too

Production: Browsable, Downloadable Tasks and a Faster Results Page

Choose a tag to compare

@bjamhong bjamhong released this 02 Oct 16:55
ba70ba1

Deploys main at ba70ba1. It also carries #279, whose own release (prod-20261002-20b1479) was cancelled before deploying, so one deploy ships both.

  • #280 Publish a release's tasks on selfbench.dev, to browse and download: Publish Tasks in the release dialog; a Tasks list and viewer on repository pages; downloads without the repository snapshot; task data kept out of search and training crawls, cached a minute, and marked with a canary once its secret is set (Terraform adds the empty selfbench-task-canary secret)
  • #279 Keep run summaries in the database so the Results page stops reading every run's record

Production: Time-Limited Trials Scored and Backfillable

Choose a tag to compare

@avyayv avyayv released this 02 Oct 20:03
6e6fd60

Deploys main at 6e6fd60. - #282 Score trials whose agent hit its time limit instead of dropping them: Harbor's verifier result now counts, Pi timeout costs are recovered, timed-out trials are marked in the app, and backfill-agent-timeouts.js can score earlier timeouts

Production: Long-Prompt Costs Priced Where Rates Change

Choose a tag to compare

@avyayv avyayv released this 02 Oct 22:15
2a70a03

Deploys main at 2a70a03. - #284 Price long prompts only where rates change, and recover missing eval costs - #287 Find Claude Code's own cost by its result event - #285 Close the task viewer on a click around it, and make the snapshot rebuild one command - #286 Publish a release's tasks by default, and box Publish Tasks so it is plain to see

Production: Faster Results Page from Stored Run Summaries

Choose a tag to compare

@bjamhong bjamhong released this 02 Oct 16:23
20b1479

Deploys main at 20b1479.

  • #279 Keep run summaries in the database so the Results page stops reading every run's record (adds the evaluation_summaries table)
  • #277 Open the README with a selfbench.dev demo and results chart (README only)
  • #278 Explain how SelfBench works with an animated GIF (README only)

Production: Canary-First Task Instructions and Run Summaries Stored Under Load

Choose a tag to compare

@bjamhong bjamhong released this 02 Oct 21:02
019c361

Deploys main at 019c361.

  • #283 Open each published instruction with the canary, and show it without
  • #281 Store the summaries a list of runs rebuilds, however busy the server is

Production: Results Grouped by Configuration and Repository Page Headings

Choose a tag to compare

@bjamhong bjamhong released this 01 Oct 22:58
ee0918a

Deploys main at ee0918a.

  • #236 Group the Results page by configuration, run and task
  • #275 Make a repository page's name its h1