chore(model-eval): refresh verifier pilot candidates to current catalog (add claude-opus-4-8) - #2818
Conversation
The candidate set drifted from the catalog: it listed claude-sonnet-4-6 (no longer catalogued) and omitted claude-opus-4-8 (the current high-capability Anthropic model that supersedes the opus-4-6 incumbent). Add claude-opus-4-8 + claude-sonnet-5 as candidates and drop the defunct sonnet-4-6 so maint-78 tests the models that actually supersede the incumbents. OpenAI already lists the gpt-5.6 candidates. Candidates only gate what gets benchmarked, not selection. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
Bugbot is not enabled for your account, so this pull request was not reviewed. Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs. |
|
Warning Review limit reachedYou’ve reached a temporary PR review limit under our Fair Usage Limits Policy. Next review available in: 24 minutes Your organization has reached its usage spending cap. Adjust your spending cap in the billing tab. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: ASSERTIVE Plan: Pro Run ID: 📒 Files selected for processing (1)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
Workflow source neededPR #2818 needs either a linked GitHub issue or one valid non-issue Workflow Source before PR metadata automation can manage it safely. Please do one of:
Once a valid source is present, this warning will not be reposted. |
Automated Status SummaryHead SHA: 5b60aa7
Coverage Overview
Updated automatically; will refresh on subsequent CI/Docker completions. Keepalive checklistScopeNo scope information available Tasks
Acceptance criteria
|
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 75e0cc0f78
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| {"provider":"anthropic","model_id":"claude-opus-4-8","role":"candidate"}, | ||
| {"provider":"anthropic","model_id":"claude-sonnet-5","role":"candidate"}, |
There was a problem hiding this comment.
When maint-78-model-evaluation-pilot.yml runs the default candidate file, each row is passed through tools.run_model_eval_pilot into pr_verifier.evaluate_pr, whose Anthropic client still sends temperature=0.1 on every request. Anthropic documents that Claude Opus 4.8 / Claude 4.7+ and Claude Sonnet 5 reject non-default sampling parameters with a 400 (docs), so these newly-added candidates will return errors, be marked schema_valid: false, and make unusable_candidates() abort the pilot during preflight instead of producing the intended benchmark. Update the client to omit sampling params for these models before adding them here.
Useful? React with 👍 / 👎.
1 participant
What
Refresh the verifier-pilot candidate set (
config/model_eval_candidates.json) to match the current catalog.It had drifted: it listed
claude-sonnet-4-6(no longer in the model catalog) and omittedclaude-opus-4-8— the current high-capability Anthropic model that supersedes theclaude-opus-4-6incumbent verifier. So amaint-78pilot could not test the model that actually supersedes the incumbent, and risked a preflight abort on the defunctsonnet-4-6.claude-opus-4-8(candidate),claude-sonnet-4-6→claude-sonnet-5(candidate);claude-opus-4-6stays incumbent.gpt-5.6-terra,gpt-5.6-solcandidates vsgpt-5.4incumbent).MODEL_SELECTION_POLICY.md.Why now
Surfaced while validating that the
verifier-balancedselections (still on the supersededopus-4-6/gpt-5.4incumbents) can actually be re-evaluated against current models. The candidate-file drift is a concrete instance of the broader "catalog-only evidence can never promote; the manual eval never runs" gap — tracked separately as a design issue.Verification
Valid JSON;
maint-78dispatched against this branch to run the paired 30-case narrowing pilot withclaude-opus-4-8+gpt-5.6-terra/solas candidates.🤖 Generated with Claude Code