fix(evals): propagate ModelMappings into evals WorkflowData to unblock AWF model alias resolution#47956
Merged
Merged
Conversation
…lves model aliases The evals job was failing with 'model small is unsupported or unrecognized by this AWF version' because buildEvalsEngineSteps did not copy ModelMappings from the parent WorkflowData into evalsData. Without ModelMappings, BuildAWFConfigJSON omits the apiProxy.models section, so the AWF cannot resolve alias names like 'small' to concrete model IDs. All evals runs produced UNKNOWN answers (empty evals.log) due to the AWF exiting with code 1 on the unresolvable model alias. Fixes the pattern to mirror threat_detection_inline_engine.go which already propagates ModelMappings correctly (line 131). Adds a regression test TestBuildEvalsEngineStepsModelMappingsPropagated with two sub-cases (with and without ModelMappings) to guard against re-introduction of this bug. Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Copilot
AI
changed the title
[WIP] Investigate broken evals job for daily evals report
fix(evals): propagate ModelMappings into evals WorkflowData to unblock AWF model alias resolution
Jul 25, 2026
pelikhan
marked this pull request as ready for review
July 25, 2026 08:37
Contributor
There was a problem hiding this comment.
Pull request overview
Fixes #47955 by propagating model aliases into eval-job AWF configuration, allowing aliases such as small to resolve correctly.
Changes:
- Propagates
ModelMappingsinto eval workflow data. - Adds regression tests for present and absent mappings.
- Regenerates affected workflow lock files with
apiProxy.models.
Show a summary per file
| File | Description |
|---|---|
pkg/workflow/evals_steps.go |
Propagates model mappings to eval jobs. |
pkg/workflow/evals_steps_test.go |
Tests AWF model-map emission. |
.github/workflows/unbloat-docs.lock.yml |
Regenerates eval AWF configuration. |
.github/workflows/tidy.lock.yml |
Regenerates eval AWF configuration. |
.github/workflows/spec-enforcer.lock.yml |
Regenerates eval AWF configuration. |
.github/workflows/security-review.lock.yml |
Regenerates eval AWF configuration. |
.github/workflows/refiner.lock.yml |
Regenerates eval AWF configuration. |
.github/workflows/plan.lock.yml |
Regenerates eval AWF configuration. |
.github/workflows/necromancer.lock.yml |
Regenerates eval AWF configuration. |
.github/workflows/issue-monster.lock.yml |
Regenerates eval AWF configuration. |
.github/workflows/issue-arborist.lock.yml |
Regenerates eval AWF configuration. |
.github/workflows/duplicate-code-detector.lock.yml |
Regenerates eval AWF configuration. |
.github/workflows/daily-regulatory.lock.yml |
Regenerates eval AWF configuration. |
.github/workflows/daily-observability-report.lock.yml |
Regenerates eval AWF configuration. |
.github/workflows/daily-news.lock.yml |
Regenerates eval AWF configuration. |
.github/workflows/daily-hippo-learn.lock.yml |
Regenerates eval AWF configuration. |
.github/workflows/daily-function-namer.lock.yml |
Regenerates eval AWF configuration. |
.github/workflows/daily-fact.lock.yml |
Regenerates eval AWF configuration. |
.github/workflows/daily-evals-report.lock.yml |
Regenerates eval AWF configuration. |
.github/workflows/daily-doc-updater.lock.yml |
Regenerates eval AWF configuration. |
.github/workflows/daily-compiler-quality.lock.yml |
Regenerates eval AWF configuration. |
.github/workflows/daily-cli-tools-tester.lock.yml |
Regenerates eval AWF configuration. |
.github/workflows/daily-cache-strategy-analyzer.lock.yml |
Regenerates eval AWF configuration. |
.github/workflows/craft.lock.yml |
Regenerates eval AWF configuration. |
.github/workflows/copilot-centralization-optimizer.lock.yml |
Regenerates eval AWF configuration. |
.github/workflows/copilot-centralization-drilldown.lock.yml |
Regenerates eval AWF configuration. |
.github/workflows/commit-changes-analyzer.lock.yml |
Regenerates eval AWF configuration. |
.github/workflows/code-simplifier.lock.yml |
Regenerates eval AWF configuration. |
.github/workflows/ci-coach.lock.yml |
Regenerates eval AWF configuration. |
.github/workflows/changeset.lock.yml |
Regenerates eval AWF configuration. |
.github/workflows/brave.lock.yml |
Regenerates eval AWF configuration. |
.github/workflows/auto-triage-issues.lock.yml |
Regenerates eval AWF configuration. |
.github/workflows/agentic-token-optimizer.lock.yml |
Regenerates eval AWF configuration. |
.github/workflows/agent-persona-explorer.lock.yml |
Regenerates eval AWF configuration. |
.github/workflows/agent-performance-analyzer.lock.yml |
Regenerates eval AWF configuration. |
.github/workflows/ab-testing-advisor.lock.yml |
Regenerates eval AWF configuration. |
Review details
Tip
Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- Files reviewed: 109/109 changed files
- Comments generated: 0
- Review effort level: Medium
This was referenced Jul 25, 2026
Contributor
|
🎉 This pull request is included in a new release. Release: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Evals jobs have been producing 100%
UNKNOWNanswers across all 82 evals-enabled workflows because the AWF exits with code 1 on an unresolvable model alias — silently, since the execution step usescontinue-on-error: true.Root cause
buildEvalsEngineStepsconstructedevalsDatawithout copyingModelMappingsfrom the parentWorkflowData.BuildAWFConfigJSONonly emits theapiProxy.modelssection whenModelMappingsis non-nil, so the AWF config lacked the alias table needed to resolve"small"→ concrete model IDs:The threat-detection job already handles this correctly (
threatDetectionData.ModelMappings = data.ModelMappings); the evals job was missing the equivalent assignment.Changes
pkg/workflow/evals_steps.go— addModelMappings: data.ModelMappingstoevalsDatastruct inbuildEvalsEngineSteps, mirroring the detection job patternpkg/workflow/evals_steps_test.go— addTestBuildEvalsEngineStepsModelMappingsPropagatedwith two sub-cases: confirms\"models\"is present in the shell-escaped AWF config JSON whenModelMappingsis set, and absent when nil