Description
All LLM-based evaluators default to a deprecated model ID as their judge model.
File: src/strands_evals/evaluators/evaluator.py, line 24
DEFAULT_BEDROCK_MODEL_ID = "us.anthropic.claude-sonnet-4-20250514-v1:0"
Claude Sonnet 4 (claude-sonnet-4-20250514) is now in legacy mode on Amazon Bedrock. While it still functions today, it is deprecated and will eventually be removed. When that happens, all evaluators that rely on the default model will silently fail — Experiment._run_evaluator() catches exceptions and records them as score: 0 results (lines 405-413), so users would see all-zero scores with no obvious indication that the judge model is unavailable.
Impact
All 10 LLM-based evaluators use this default when model=None:
OutputEvaluatorTrajectoryEvaluatorHarmfulnessEvaluatorHelpfulnessEvaluatorCoherenceEvaluatorConcisenessEvaluatorFaithfulnessEvaluatorGoalSuccessRateEvaluatorInteractionsEvaluatorResponseRelevanceEvaluator
Current Behavior
The default works today since Sonnet 4 is still accessible in legacy mode. However:
- It will break without warning once the model is fully retired
- Users following documentation examples without passing
model= will be affected
- The failure mode is silent — errors are caught and recorded as
score: 0 with a reason like Evaluator error: ..., which is easy to miss in the default display
Suggested Fix
Update DEFAULT_BEDROCK_MODEL_ID to a current model. The Claude Sonnet 4.6 model ID on Bedrock is:
DEFAULT_BEDROCK_MODEL_ID = "us.anthropic.claude-sonnet-4-6-v1:0"
To prevent this from recurring, consider one of:
- Using a model alias/pointer that Bedrock resolves to the latest version (if available)
- Raising a clear warning or error when the default model returns a deprecation signal
- Documenting the default model prominently so users know to override it
Workaround
Explicitly pass a model to every evaluator:
from strands.models import BedrockModel
evaluator = OutputEvaluator(
rubric="...",
model=BedrockModel(model_id="us.anthropic.claude-sonnet-4-6-v1:0")
)Environment
strands-agents-evals version: 0.1.14strands-agents version: 1.35.0- Python: 3.12
- Region: us-east-1
Description
All LLM-based evaluators default to a deprecated model ID as their judge model.
File:
src/strands_evals/evaluators/evaluator.py, line 24Claude Sonnet 4 (
claude-sonnet-4-20250514) is now in legacy mode on Amazon Bedrock. While it still functions today, it is deprecated and will eventually be removed. When that happens, all evaluators that rely on the default model will silently fail —Experiment._run_evaluator()catches exceptions and records them asscore: 0results (lines 405-413), so users would see all-zero scores with no obvious indication that the judge model is unavailable.Impact
All 10 LLM-based evaluators use this default when
model=None:OutputEvaluatorTrajectoryEvaluatorHarmfulnessEvaluatorHelpfulnessEvaluatorCoherenceEvaluatorConcisenessEvaluatorFaithfulnessEvaluatorGoalSuccessRateEvaluatorInteractionsEvaluatorResponseRelevanceEvaluatorCurrent Behavior
The default works today since Sonnet 4 is still accessible in legacy mode. However:
model=will be affectedscore: 0with a reason likeEvaluator error: ..., which is easy to miss in the default displaySuggested Fix
Update
DEFAULT_BEDROCK_MODEL_IDto a current model. The Claude Sonnet 4.6 model ID on Bedrock is:To prevent this from recurring, consider one of:
Workaround
Explicitly pass a model to every evaluator:
Environment
strands-agents-evalsversion: 0.1.14strands-agentsversion: 1.35.0