Set explicit Claude Sonnet fallback when workflow model is unset by Copilot · Pull Request #48618 · github/gh-aw

Copilot · 2026-07-28T13:07:56Z

Scout failed because unset Claude model selection resolved to claude-opus-5, which was rejected by the AI credits gateway pricing policy. This change makes the unset-model path resolve to a Sonnet-tier default instead of relying on Claude CLI implicit selection.

Root cause addressed
- Claude unset-model fallback path emitted empty fallback expressions (... || ''), allowing runtime implicit model selection (Opus in this run).
Model fallback behavior changes
- Updated Claude model env fallback expressions to use explicit default:
  - GH_AW_MODEL_AGENT_CLAUDE
  - GH_AW_MODEL_DETECTION_CLAUDE
  - GH_AW_MODEL_EVALS_CLAUDE
- Fallback chain is now:
  - repo/org override → enterprise default override → claude-sonnet-5
Expression-model path alignment
- For expression-based model values, updated GH_AW_MODEL_FALLBACK for Claude to include the same explicit Sonnet fallback (instead of empty fallback).
Evals parity
- Updated evals Claude fallback expression builder to use the same explicit Sonnet default, keeping fallback semantics consistent across execution phases.
Test updates
- Adjusted model env var tests to assert Claude fallback includes claude-sonnet-5 rather than empty fallback.

// before
compilerenv.BuildModelOverrideExpressionEmptyFallback(
  constants.EnvVarModelAgentClaude,
  compilerenv.DefaultModelClaude,
)

// after
compilerenv.BuildModelOverrideExpression(
  constants.EnvVarModelAgentClaude,
  compilerenv.DefaultModelClaude,
  constants.CopilotBYOKDefaultModel, // "claude-sonnet-5"
)

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

github-actions · 2026-07-28T13:26:14Z

✅ PR Code Quality Reviewer completed the code quality review.

github-actions · 2026-07-28T13:26:29Z

🧠 Matt Pocock Skills Reviewer has completed the skills-based review. ✅

github-actions · 2026-07-28T13:26:40Z

✅ Design Decision Gate 🏗️ completed the design decision gate check.

No ADR enforcement needed: PR does not have the 'implementation' label and has ≤100 new lines of code in business logic directories (11 additions detected).

github-actions · 2026-07-28T13:26:59Z

✅ Test Quality Sentinel completed test quality analysis.

github-actions · 2026-07-28T13:29:18Z

Great work! 🚀 This fix handles a critical issue with Claude model fallback behavior. When workflow model is unset, the old code emitted empty fallback expressions, allowing Claude CLI to implicitly select Opus (which was then rejected by the pricing gateway). Your changes now ensure an explicit Sonnet fallback across all three phases (agent, detection, evals), keeping the behavior consistent.

What looks good:

✅ Clear bug fix with thorough explanation in the PR body
✅ Changes are tightly focused on the fallback expression updates (claude_engine.go, evals_steps.go)
✅ Test expectations updated to match the new explicit Sonnet default
✅ Consistent handling across all three model env var contexts
✅ Created by the Copilot agent—following the right contribution workflow

This is ready for review!

Generated by ✅ Contribution Check · sonnet50 · 65.5 AIC · ⌖ 14.5 AIC · ⊞ 8.3K · ◷

Copilot

Pull request overview

Adds an explicit Claude Sonnet fallback when workflows omit a model.

Changes:

Defaults Claude agent, detection, and evals execution to claude-sonnet-5.
Aligns expression-model fallback behavior.
Updates model environment tests.

Show a summary per file

File	Description
`pkg/workflow/model_env_vars_test.go`	Updates Claude fallback expectations.
`pkg/workflow/evals_steps.go`	Adds the explicit Claude evals fallback.
`pkg/workflow/claude_engine.go`	Applies explicit fallbacks across Claude phases.

Review details

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comments suppressed due to low confidence (2)

pkg/workflow/claude_engine.go:518

Unset Claude agent runs will now execute claude-sonnet-5, but the generated run-info step still resolves Claude through getDefaultAgentModel to AgentDefaultModel ("agent"; see compiler_yaml_lookups.go:110). This makes the recorded/audited model disagree with the model actually passed to Claude. Please align the run-info fallback and its test with the new Claude default.

			env[constants.EnvVarModelAgentClaude] = compilerenv.BuildModelOverrideExpression(constants.EnvVarModelAgentClaude, compilerenv.DefaultModelClaude, constants.CopilotBYOKDefaultModel)

pkg/workflow/claude_engine.go:531

This makes Claude's fallback depend on CopilotBYOKDefaultModel, which is documented as Copilot BYOK-only in pkg/constants/engine_constants.go:333-338. A future Copilot default change would silently change Claude's expression fallback too. Introduce a Claude-specific (or deliberately shared, neutrally named) default constant and use it across the Claude fallback paths.

		env[constants.EnvVarModelFallback] = compilerenv.BuildModelOverrideExpression(claudeModelVar, compilerenv.DefaultModelClaude, constants.CopilotBYOKDefaultModel)

Files reviewed: 3/3 changed files
Comments generated: 2
Review effort level: Medium

 		return compilerenv.BuildModelOverrideExpression(constants.EnvVarModelEvalsCopilot, compilerenv.DefaultModelCopilot, constants.CopilotBYOKDefaultModel)
 	case string(constants.ClaudeEngine):
-		return compilerenv.BuildModelOverrideExpressionEmptyFallback(constants.EnvVarModelEvalsClaude, compilerenv.DefaultModelClaude)
+		return compilerenv.BuildModelOverrideExpression(constants.EnvVarModelEvalsClaude, compilerenv.DefaultModelClaude, constants.CopilotBYOKDefaultModel)


 	if workflowData.Model == "" {
 		if phase == runPhaseEvals {
-			env[constants.EnvVarModelEvalsClaude] = compilerenv.BuildModelOverrideExpressionEmptyFallback(constants.EnvVarModelEvalsClaude, compilerenv.DefaultModelClaude)
+			env[constants.EnvVarModelEvalsClaude] = compilerenv.BuildModelOverrideExpression(constants.EnvVarModelEvalsClaude, compilerenv.DefaultModelClaude, constants.CopilotBYOKDefaultModel)


github-actions

Changes look correct and consistent. The explicit claude-sonnet-5 fallback is applied uniformly across all three Claude execution phases (agent, detection, evals) and for the expression-model path. Tests are updated to match. No issues found.

🧵 Reviewed using Impeccable skills by Impeccable Skills Reviewer · sonnet46 · 20 AIC · ⌖ 5.18 AIC · ⊞ 5K

github-actions

Skills-Based Review 🧠

Applied /codebase-design — one naming observation, otherwise the fix is clean and well-targeted.

📋 Key Themes & Highlights

Key Themes

Constant reuse across engines: CopilotBYOKDefaultModel is now the explicit fallback for both the Copilot and Claude engines, but its name signals Copilot-only intent. This is a minor but real navigation hazard.

Positive Highlights

✅ Root cause directly addressed — empty fallback expressions replaced throughout all three Claude execution phases (agent, detection, evals)
✅ Consistent change: all four call sites updated symmetrically
✅ Tests updated to assert the new non-empty fallback value
✅ PR description clearly documents the fallback chain

🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer · sonnet46 · 22.9 AIC · ⌖ 4.7 AIC · ⊞ 6.7K
_{Comment /matt to run again}

github-actions · 2026-07-28T13:32:45Z

🧪 Test Quality Sentinel Report

✅ Test Quality Score: 85/100 — Excellent

Analyzed 4 test case modification(s) across 3 existing table-driven test functions: 4 design, 0 implementation, 0 violation(s).

📊 Metrics (4 modified table-driven rows)

Metric	Value
Analyzed	4 rows (Go: 4, JS: 0)
✅ Design	4 (100%)
⚠️ Implementation	0 (0%)
Edge/error coverage	2 (50%)
Duplicate clusters	0
Inflation	No (test:prod ratio ~0.67)
🚨 Violations	0

Test	File	Classification	Issues
`TestModelEnvVarInjectionForAgentJob` — Claude row	`model_env_vars_test.go:25`	`behavioral_contract`, `design_test`	None
`TestModelEnvVarInjectionForDetectionJob` — Claude row	`model_env_vars_test.go:109`	`behavioral_contract`, `design_test`	None
`TestExpressionModelUsesEnvVar` — Claude rows x2	`model_env_vars_test.go:337,346`	`behavioral_contract`, `design_test`	None

Verdict

✅ Passed. 0% implementation tests (threshold: 30%). Changes correctly update expected values from empty string to constants.CopilotBYOKDefaultModel to reflect the new explicit Sonnet fallback, preserving all behavioral assertions.

🧪 Test quality analysis by Test Quality Sentinel · sonnet46 · 30.2 AIC · ⌖ 8.56 AIC · ⊞ 8.1K · ◷
_{Comment /review to run again}

github-actions

✅ Test Quality Sentinel: 85/100. 0% implementation tests (threshold: 30%).

github-actions

Verdict: non-blocking naming concern only

The fallback logic change is correct, targeted, and well-tested (build + targeted unit tests pass). One maintainability nit flagged inline about reusing a Copilot-named constant for the Claude fallback default.

Themes

Root-cause fix (empty fallback → explicit claude-sonnet-5 default) is sound and consistent across agent/detection/evals/expression paths.
Test updates correctly reflect the new fallback expressions.
Only concern: constants.CopilotBYOKDefaultModel is now reused for a semantically distinct purpose (Claude implicit-model bug fix vs. Copilot BYOK requirement), which could cause confusion or accidental coupling in future changes.

🔎 Code quality review by PR Code Quality Reviewer · sonnet50 · 74.2 AIC · ⌖ 4.38 AIC · ⊞ 7.4K
_{Comment /review to run again}

github-actions · 2026-07-28T13:36:11Z

+			env[constants.EnvVarModelDetectionClaude] = compilerenv.BuildModelOverrideExpression(constants.EnvVarModelDetectionClaude, compilerenv.DefaultModelClaude, constants.CopilotBYOKDefaultModel)
 		} else {
-			env[constants.EnvVarModelAgentClaude] = compilerenv.BuildModelOverrideExpressionEmptyFallback(constants.EnvVarModelAgentClaude, compilerenv.DefaultModelClaude)
+			env[constants.EnvVarModelAgentClaude] = compilerenv.BuildModelOverrideExpression(constants.EnvVarModelAgentClaude, compilerenv.DefaultModelClaude, constants.CopilotBYOKDefaultModel)


Fallback default reuses a Copilot-named constant for the Claude/Detection/Evals path; naming is misleading for future maintainers.

💡 Details

The change correctly fixes the root-cause bug (Claude implicit model resolution defaulting to Opus). However, constants.CopilotBYOKDefaultModel is reused as the generic Claude fallback in three call sites here plus evals_steps.go, despite its name and doc comment ("explicit fallback model for Copilot BYOK mode") describing a Copilot-specific concept. A reader modifying Copilot BYOK behavior later could unknowingly change Claude's fallback too, or vice versa, since they are now coupled via one constant that happens to hold the same string value today.

Suggested fix: introduce a neutral constant, e.g. constants.SonnetDefaultModel = "claude-sonnet-5", and have both CopilotBYOKDefaultModel and the Claude fallback usages reference it (or just use it directly for Claude), so the two use cases remain independently adjustable.

Rationale: coupling two independently-evolving fallback behaviors (Copilot BYOK vs. Claude implicit-model bug fix) through a single misnamed constant is a latent maintenance hazard, even though today's value happens to match.

pelikhan · 2026-07-28T14:16:07Z

@copilot run pr-finisher skill

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot · 2026-07-28T14:35:39Z

@copilot run pr-finisher skill

Completed a pr-finisher pass and pushed commit ab7730d with the requested fixes: added Claude evals-phase fallback coverage, aligned evals expression fallback coverage for Claude, introduced a neutral Sonnet default constant, and updated the affected wasm golden.

github-actions · 2026-07-29T10:47:57Z

🎉 This pull request is included in a new release.

Release: v0.83.5

Set Claude fallback model to Sonnet default

738d59d

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot AI assigned Copilot and pelikhan Jul 28, 2026

Copilot AI changed the title ~~Fix Claude default fallback model to Sonnet for unset workflow models~~ Set explicit Claude Sonnet fallback when workflow model is unset Jul 28, 2026

Copilot finished work on behalf of pelikhan July 28, 2026 13:08

Copilot created this pull request from a session on behalf of pelikhan July 28, 2026 13:08 View session

Copilot AI requested a review from pelikhan July 28, 2026 13:08

pelikhan marked this pull request as ready for review July 28, 2026 13:24

Copilot AI review requested due to automatic review settings July 28, 2026 13:24

Copilot started reviewing on behalf of pelikhan July 28, 2026 13:25 View session

github-actions Bot mentioned this pull request Jul 28, 2026

[Contribution Check Report] Contribution Check — 2026-07-28 #48472

Closed

Copilot AI reviewed Jul 28, 2026

View reviewed changes

github-actions Bot approved these changes Jul 28, 2026

View reviewed changes

github-actions Bot reviewed Jul 28, 2026

View reviewed changes

github-actions Bot approved these changes Jul 28, 2026

View reviewed changes

github-actions Bot reviewed Jul 28, 2026

View reviewed changes

github-actions Bot mentioned this pull request Jul 28, 2026

[pr-sous-chef] pr-sous-chef run report #48464

Closed

Merge branch 'main' into copilot/investigate-opus-5-default

10f0903

Copilot started work on behalf of pelikhan July 28, 2026 14:16 View session

Copilot AI and others added 2 commits July 28, 2026 14:21

chore: start pr-finisher plan

247d442

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

test: cover claude evals fallback and decouple sonnet default naming

ab7730d

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot finished work on behalf of pelikhan July 28, 2026 14:37

pelikhan merged commit 42eeb4c into main Jul 28, 2026

pelikhan deleted the copilot/investigate-opus-5-default branch July 28, 2026 14:41

Uh oh!

Conversation

Copilot AI commented Jul 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jul 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jul 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jul 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jul 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jul 28, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Review details

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Skills-Based Review 🧠

Key Themes

Positive Highlights

Uh oh!

github-actions Bot commented Jul 28, 2026

🧪 Test Quality Sentinel Report

Verdict

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Verdict: non-blocking naming concern only

Uh oh!

github-actions Bot Jul 28, 2026

Choose a reason for hiding this comment

Uh oh!

pelikhan commented Jul 28, 2026

Uh oh!

Copilot AI commented Jul 28, 2026

Uh oh!

github-actions Bot commented Jul 29, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Copilot AI commented Jul 28, 2026 •

edited

Loading

github-actions Bot commented Jul 28, 2026 •

edited

Loading

github-actions Bot commented Jul 28, 2026 •

edited

Loading

github-actions Bot commented Jul 28, 2026 •

edited

Loading

github-actions Bot commented Jul 28, 2026 •

edited

Loading