Skip to content

v0.5.6

Choose a tag to compare

View all tags
@github-actions github-actions released this 26 Oct 04:05
· 5 commits to main since this release
v0.5.6
This tag was signed with the committer’s verified signature.
metcalfc Chad Metcalf
SSH Key Fingerprint: iyajAqdNc3NUVyNh3o38Zq8ytQoMZnpmhQhm3z9O0Qs
Verified
Learn about vigilant mode.
89eef96
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Nothing to do before you upgrade. A grading method change for runs with a claude-code or codex answerer: they start a new trend from this release. Runs that use only API answerers are unaffected.

  • Codex's search track searches the live web. codex 0.163 drops --search when it comes before exec, so Codex search answers searched a cache rather than the live web (its requests said external_web_access: false). Codex is now called with web_search="live" after exec, as the search policy intends.
  • Codex's recall track has no web search. Recall is defined as having no tools, but Codex still offered its model a cached web search, and only the prompt asked it not to use it. Codex is now called with web_search="disabled" on recall, so recall measures what the model already believes, as the other answerers' recall does.
  • A run's method covers how CLI answerers are called. A run records the arguments it gives claude-code and codex on each track, and run.Method includes them. A change in how a CLI is called now starts a new trend line instead of mixing with older runs. For the same reason, a queued run asked with older arguments doesn't resume, and re-grading skips it ("asked differently").
  • Codex answers make only the calls they need. Plugins, ChatGPT apps and analytics are off for every Codex call. Before, each answer fetched the plugin catalog, listed apps and posted analytics events over the account's login.
  • Claude Code telemetry is off. Every claude call (answers, and grading and analysis through claude-code) gets CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC=1, which a value in Cheap Seats' own environment can't override. WebSearch and WebFetch are unaffected.

Trends. Codex runs start a new method from this release: the first two changes change what a Codex answer can do, so earlier Codex results aren't comparable with new ones, and re-grading can't repair them. claude-code runs also start a new method once, because their arguments are recorded for the first time; how they're asked doesn't change. Runs with only API answerers keep their method. The grader is unchanged.

No schema change: a run's snapshot gains cli_args, and runs stored before it keep their method.

Image: ghcr.io/cheapseatsdev/cheapseats:0.5.6 · Chart: oci://ghcr.io/cheapseatsdev/charts/cheapseats --version 0.5.6

Assets 3
Loading