fix(llm): omit temperature for Anthropic models that reject it (unblocks verifier eval of Opus 4.8 / Claude 5) by stranske · Pull Request #2822 · stranske/Workflows

stranske · 2026-07-24T21:38:45Z

What

The frozen verifier client passed temperature=0.1 unconditionally to Anthropic (_build_anthropic_client), but the newer "thinking" generation returns HTTP 400 invalid_request_error on a custom temperature. This broke the maint-78 verifier pilot: run 30112277194 (2026-07-24) failed preflight because claude-opus-4-8 and claude-sonnet-5 400'd on temperature — and preflight is all-or-nothing, so the whole pilot aborted (even though the OpenAI candidates passed).

Fix

Mirror the existing OpenAI reasoning-model handling (_is_reasoning_model → omit temperature) for Anthropic:

Add _anthropic_rejects_temperature(model) — matches claude-opus-4-8 + the Claude 5 family (claude-<name>-5…); minor-version -5 suffixes like claude-haiku-4-5 are not matched.
_build_anthropic_client omits temperature for those, and preserves temperature=0.1 for the incumbent claude-opus-4-6 (which still accepts it) — so the production verifier's behavior is unchanged.

Scope / honesty

Interim, evidence-based guard. The durable, no-maintenance approach (capability-aware / retry-on-400, plus per-candidate preflight tolerance) is tracked in [Design] Self-feeding verifier-model promotion: auto-trigger pilot + live-harvested corpus + tiered auto-promote/rollback #2819.
This does not fix the other pilot failure: github-models/codex-mini-latest returned 404 (endpoint/path issue) — separate from temperature. With all-or-nothing preflight, a re-run still needs that handled (or the github incumbent excluded).

Verification

ruff==0.15.20 ✅ · black==26.5.1 -l100 ✅ · mypy==2.1.0 ✅ (CI-pinned). pytest tests/tools/test_langchain_client.py → 63 passed, incl. 4 new: helper matches opus-4-8/sonnet-5/fable-5/opus-5; incumbent opus-4-6 + minor -5 still accept; builder omits temperature for new models, keeps it for opus-4-6.

🤖 Generated with Claude Code

Summary by CodeRabbit

Bug Fixes
- Improved compatibility with newer Claude models by not sending an unsupported temperature setting for models that reject it.
- Kept the existing temperature behavior for Claude models that still support it.
Tests
- Added assertions to verify correct temperature handling across supported Anthropic/Claude model versions.
Chores
- Updated worker runtime metadata (timestamp and PR number).

The verifier client passed temperature=0.1 unconditionally to Anthropic, but the newer "thinking" generation returns HTTP 400 invalid_request_error on a custom temperature. The maint-78 verifier pilot (run 30112277194, 2026-07-24) failed preflight because claude-opus-4-8 and claude-sonnet-5 400'd on temperature, which (preflight being all-or-nothing) aborted the whole run. Mirror the existing OpenAI reasoning-model handling for Anthropic: add _anthropic_rejects_temperature() and omit the param for Opus 4.8 + the Claude 5 family, while preserving temperature=0.1 for the incumbent claude-opus-4-6 (which still accepts it) so production verifier behavior is unchanged. Interim, evidence-based guard; durable capability-aware handling (retry-on-400) is tracked in #2819. ruff/black/mypy (CI-pinned) clean; tests/tools/test_langchain_client.py 63 pass incl. 4 new cases. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

cursor · 2026-07-24T21:38:49Z

Bugbot is not enabled for your account, so this pull request was not reviewed.

Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs.

coderabbitai · 2026-07-24T21:39:12Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: b6feab4b-cf25-49d9-a631-ac5955ba162b

📥 Commits

Reviewing files that changed from the base of the PR and between 5972dad and ee82966.

📒 Files selected for processing (1)

langsmith-fleet-worker-attempt.json

📝 Walkthrough

Walkthrough

Anthropic client construction now omits temperature for newer Claude models that reject it while retaining temperature=0.1 for supported models. Tests cover model classification and captured constructor arguments. Runtime metadata is also updated.

Changes

Anthropic temperature handling

Layer / File(s)	Summary
Conditional temperature policy `tools/langchain_client.py`	Adds model detection for Claude variants that reject explicit temperature and conditionally includes `temperature=0.1` when building the Anthropic client.
Temperature policy coverage `tests/tools/test_langchain_client.py`	Captures Anthropic client arguments and verifies rejecting models omit temperature while supported models retain it.

Runtime metadata

Layer / File(s)	Summary
Execution metadata update `langsmith-fleet-worker-attempt.json`	Updates the emission timestamp and pull request number while preserving the surrounding runtime metadata.

Estimated code review effort: 2 (Simple) | ~10 minutes

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the main change: omitting temperature for Anthropic models that reject it.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch fix-anthropic-temperature-newer-models

_{Comment @coderabbitai help to get the list of available commands.}

stranske-keepalive · 2026-07-24T21:39:35Z

Workflow source needed

PR #2822 needs either a linked GitHub issue or one valid non-issue Workflow Source before PR metadata automation can manage it safely.

Please do one of:

Add  or a normal Closes #123 / Related to #123 line.
Check one Workflow Source option in the PR body.
Add a hidden marker such as , , , , or .
Add a workflow source label such as workflow:source-direct-pr, workflow:source-local-request, workflow:source-review-followup, workflow:source-sync, or workflow:no-automation.

Once a valid source is present, this warning will not be reposted.

stranske-keepalive · 2026-07-24T21:40:03Z

Automated Status Summary

Head SHA: 6c2e9e3
Latest Runs: ⏳ pending — Gate
Required contexts: Gate / gate, Health 45 Agents Guard / guard
Required: core tests (3.12): ⏳ pending, core tests (3.13): ⏳ pending, docker smoke: ⏳ pending, gate: ⏳ pending

Workflow / Job	Result	Logs
(no jobs reported)	⏳ pending	—

Coverage Overview

Coverage history entries: 1

Coverage Trend

Metric	Value
Current	75.74%
Baseline	85.00%
Delta	-9.26%
Minimum	70.00%
Status	✅ Pass

Top Coverage Hotspots (lowest coverage)

File	Coverage	Missing
`scripts/issue_dedup_smoke.py`	0.0%	4
`scripts/runner_lib/__main__.py`	0.0%	3
`scripts/validate_template_sync.py`	0.0%	81
`scripts/langchain/topic_splitter.py`	19.1%	57
`tools/codex_log_analyzer.py`	19.6%	140
`scripts/repo_review_round2_runner.py`	25.1%	344
`scripts/prune_agent_stubs.py`	39.7%	26
`scripts/repo_review_round1_runner.py`	40.7%	133
`tools/ensure_workflow_timeout_variables.py`	42.1%	74
`scripts/sync_label_docs.py`	42.9%	64
`tools/discover_model_catalog.py`	44.8%	55
`scripts/repo_review_backlog_scan.py`	45.3%	116
`scripts/repo_review_body_writer.py`	46.5%	86
`tools/codex_session_analyzer.py`	47.9%	59
`scripts/create_verifier_labels.py`	48.3%	58

Low Coverage Files (<50.0%)

File	Coverage	Missing
`scripts/issue_dedup_smoke.py`	0.0%	4
`scripts/runner_lib/__main__.py`	0.0%	3
`scripts/validate_template_sync.py`	0.0%	81
`scripts/langchain/topic_splitter.py`	19.1%	57
`tools/codex_log_analyzer.py`	19.6%	140
`scripts/repo_review_round2_runner.py`	25.1%	344
`scripts/prune_agent_stubs.py`	39.7%	26
`scripts/repo_review_round1_runner.py`	40.7%	133
`tools/ensure_workflow_timeout_variables.py`	42.1%	74
`scripts/sync_label_docs.py`	42.9%	64
`tools/discover_model_catalog.py`	44.8%	55
`scripts/repo_review_backlog_scan.py`	45.3%	116
`scripts/repo_review_body_writer.py`	46.5%	86
`tools/codex_session_analyzer.py`	47.9%	59
`scripts/create_verifier_labels.py`	48.3%	58

Updated automatically; will refresh on subsequent CI/Docker completions.

Keepalive checklist

Scope

No scope information available

Tasks

No tasks defined

Acceptance criteria

No acceptance criteria defined

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 5972dad4f8

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-07-24T21:40:26Z

+    if not _anthropic_rejects_temperature(model):
+        kwargs["temperature"] = 0.1


Apply the temperature guard to the consumer template

When maint-68-sync-consumer-repos.yml distributes the manifest-declared tools/langchain_client.py, the sync compiler prefers templates/consumer-repo/tools/langchain_client.py; that copy remains unchanged and still passes temperature=0.1 unconditionally. Consequently, consumer verifier evaluations using claude-opus-4-8 or Claude 5 will continue receiving the same HTTP 400 that this change fixes locally. Mirror the guard and its tests into the consumer template.

AGENTS.md reference: AGENTS.md:L39-L41

Useful? React with 👍 / 👎.

coderabbitai

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@tests/tools/test_langchain_client.py`:
- Around line 815-820: Extend
test_anthropic_rejects_temperature_matches_newer_generation to assert that
_anthropic_rejects_temperature returns true for the Claude 5 haiku model, using
the same pattern as the existing opus, sonnet, and fable assertions.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

Push a commit to this branch (recommended)
Create a new PR with the fixes

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: a5b21003-a5cb-4d22-a9db-06498f421b71

📥 Commits

Reviewing files that changed from the base of the PR and between b1054df and 5972dad.

📒 Files selected for processing (2)

tests/tools/test_langchain_client.py
tools/langchain_client.py

coderabbitai · 2026-07-24T21:41:20Z

+def test_anthropic_rejects_temperature_matches_newer_generation():
+    # Confirmed-rejecting (maint-78 pilot 2026-07-24) + Claude 5 family.
+    assert langchain_client._anthropic_rejects_temperature("claude-opus-4-8")
+    assert langchain_client._anthropic_rejects_temperature("claude-sonnet-5")
+    assert langchain_client._anthropic_rejects_temperature("claude-fable-5")
+    assert langchain_client._anthropic_rejects_temperature("claude-opus-5")


📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Cover the haiku Claude 5 family.

Production code includes haiku, but this table does not assert claude-haiku-5; removing that family would pass the suite.

Proposed test

assert langchain_client._anthropic_rejects_temperature("claude-sonnet-5") + assert langchain_client._anthropic_rejects_temperature("claude-haiku-5") assert langchain_client._anthropic_rejects_temperature("claude-fable-5")

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change

def test_anthropic_rejects_temperature_matches_newer_generation():

# Confirmed-rejecting (maint-78 pilot 2026-07-24) + Claude 5 family.

assert langchain_client._anthropic_rejects_temperature("claude-opus-4-8")

assert langchain_client._anthropic_rejects_temperature("claude-sonnet-5")

assert langchain_client._anthropic_rejects_temperature("claude-fable-5")

assert langchain_client._anthropic_rejects_temperature("claude-opus-5")

def test_anthropic_rejects_temperature_matches_newer_generation():

# Confirmed-rejecting (maint-78 pilot 2026-07-24) + Claude 5 family.

assert langchain_client._anthropic_rejects_temperature("claude-opus-4-8")

assert langchain_client._anthropic_rejects_temperature("claude-sonnet-5")

assert langchain_client._anthropic_rejects_temperature("claude-haiku-5")

assert langchain_client._anthropic_rejects_temperature("claude-fable-5")

assert langchain_client._anthropic_rejects_temperature("claude-opus-5")

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/tools/test_langchain_client.py` around lines 815 - 820, Extend test_anthropic_rejects_temperature_matches_newer_generation to assert that _anthropic_rejects_temperature returns true for the Claude 5 haiku model, using the same pattern as the existing opus, sonnet, and fable assertions.

Source: Path instructions

…guard) # Conflicts: # tests/tools/test_langchain_client.py

stranske added bug Something isn't working priority: high labels Jul 24, 2026

stranske temporarily deployed to agent-standard July 24, 2026 21:38 — with GitHub Actions Inactive

stranske temporarily deployed to agent-standard July 24, 2026 21:39 — with GitHub Actions Inactive

chatgpt-codex-connector Bot reviewed Jul 24, 2026

View reviewed changes

stranske temporarily deployed to agent-standard July 24, 2026 21:40 — with GitHub Actions Inactive

coderabbitai Bot reviewed Jul 24, 2026

View reviewed changes

stranske added the autofix:escalated label Jul 24, 2026

stranske temporarily deployed to agent-standard July 24, 2026 21:48 — with GitHub Actions Inactive

stranske enabled auto-merge (squash) July 24, 2026 21:51

stranske mentioned this pull request Jul 24, 2026

[Reliability] Backplane reference-run staleness (168h) reddens main fleet-wide + blocks unrelated PRs #2823

Closed

3 tasks

chore(codex-autofix): apply updates (PR #2822)

ee82966

agents-workflows-bot Bot temporarily deployed to agent-standard July 24, 2026 21:53 Inactive

stranske removed the autofix:escalated label Jul 24, 2026

stranske mentioned this pull request Jul 24, 2026

feat(model-eval): per-provider preflight tolerance (one bad candidate no longer aborts the pilot) — part of #2819 #2824

Merged

stranske merged commit 5275c48 into main Jul 24, 2026
52 checks passed

stranske deleted the fix-anthropic-temperature-newer-models branch July 24, 2026 21:57

agents-workflows-bot Bot mentioned this pull request Jul 24, 2026

chore(main): release 1.19.0 #2721

Merged

stranske added a commit that referenced this pull request Jul 24, 2026

merge main (resolve langchain_client conflict with #2822 temperature …

602af05

…guard) # Conflicts: # tests/tools/test_langchain_client.py

stranske mentioned this pull request Jul 31, 2026

chore(model-registry): advance provisional verifier selections to current catalog #2852

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(llm): omit temperature for Anthropic models that reject it (unblocks verifier eval of Opus 4.8 / Claude 5) - #2822

fix(llm): omit temperature for Anthropic models that reject it (unblocks verifier eval of Opus 4.8 / Claude 5)#2822
stranske merged 2 commits into
mainfrom
fix-anthropic-temperature-newer-models

stranske commented Jul 24, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

cursor Bot commented Jul 24, 2026

Uh oh!

coderabbitai Bot commented Jul 24, 2026 •

edited

Loading

Walkthrough

Changes

Uh oh!

stranske-keepalive Bot commented Jul 24, 2026

Uh oh!

stranske-keepalive Bot commented Jul 24, 2026 •

edited by agents-workflows-bot Bot

Loading

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

chatgpt-codex-connector Bot Jul 24, 2026

Uh oh!

coderabbitai Bot left a comment

Uh oh!

coderabbitai Bot Jul 24, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

		if not _anthropic_rejects_temperature(model):
		kwargs["temperature"] = 0.1

Conversation

stranske commented Jul 24, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Fix

Scope / honesty

Verification

Summary by CodeRabbit

Uh oh!

cursor Bot commented Jul 24, 2026

Uh oh!

coderabbitai Bot commented Jul 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Uh oh!

stranske-keepalive Bot commented Jul 24, 2026

Workflow source needed

Uh oh!

stranske-keepalive Bot commented Jul 24, 2026 • edited by agents-workflows-bot Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Automated Status Summary

Coverage Overview

Coverage Trend

Top Coverage Hotspots (lowest coverage)

Low Coverage Files (<50.0%)

Keepalive checklist

Scope

Tasks

Acceptance criteria

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Jul 24, 2026

Choose a reason for hiding this comment

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

coderabbitai Bot Jul 24, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

stranske commented Jul 24, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Jul 24, 2026 •

edited

Loading

stranske-keepalive Bot commented Jul 24, 2026 •

edited by agents-workflows-bot Bot

Loading