chore(evals): standardize model judge config#24
Conversation
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ca33058465
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
|
||
| Use any account-supported model approved for the local calibration run. Examples below use | ||
| `gpt-5.5`; that is an example, not repo policy. The standard manual config judges an existing | ||
| candidate. If a future workflow needs provider-backed candidate generation, add a separate explicit |
There was a problem hiding this comment.
Omit the nonexistent generate run from reports
When following this new no-generation flow, the user is told to judge an existing candidate and not enable generation, but the report command below still passes --generate <run-id>-generate; compileReport then unconditionally reads that supplied generate run's report.md (evals/adapter.mjs:355-358), so the documented flow fails unless the user happened to create an unrelated generation bundle. Please either omit --generate from this manual pointwise-only report command or add a matching generation step/config.
Useful? React with 👍 / 👎.
Summary
Verification
Did not run eval:judge:coverage or provider/model calls.