Guide Codex workflows to compatible Copilot models - #58872
Conversation
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
|
🧠 Matt Pocock Skills Reviewer has completed the skills-based review. ✅ Warning Firewall blocked 1 domainThe following domain was blocked by the firewall during workflow execution:
To allow these domains, add them to the network:
allowed:
- defaults
- "github.com"See Network Configuration for more information.
|
|
✅ Design Decision Gate 🏗️ completed the design decision gate check. See the comment below for the result and any generated ADR draft. Warning Firewall blocked 2 domainsThe following domains were blocked by the firewall during workflow execution:
To allow these domains, add them to the network:
allowed:
- defaults
- "github.com"
- "proxy.golang.org"See Network Configuration for more information.
|
|
✅ Test Quality Sentinel completed test quality analysis. Warning Threat Detection Engine Failure — The analysis engine could not complete. This is a tooling failure, not a security finding. What happenedThe threat detection engine failed to produce results. Review the workflow run logs for details.
|
|
✅ PR Code Quality Reviewer completed the code quality review.
|
|
✅ Ponytail Reviewer completed successfully! Lean already. Ship. Warning Firewall blocked 1 domainThe following domain was blocked by the firewall during workflow execution:
To allow these domains, add them to the network:
allowed:
- defaults
- "ab.chatgpt.com"See Network Configuration for more information.
|
Comment MemoryNote This comment is managed by comment memory.It stores persistent context for this thread in the code block at the top of this comment.
|
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
There was a problem hiding this comment.
Request changes
The compiler-side warning is a good idea, but the new compatibility check is still too loose to rely on for model safety.
Blocking themes
- The warning currently treats any model name containing
codexas compatible, which is a brittle substring heuristic rather than an explicit compatibility check. - The bulk workflow migration is fine on its own; the blocking concern is that the compiler guidance can silently miss incompatible future model names.
🔎 Code quality review by PR Code Quality Reviewer · copilot · gpt54 · 49.5 AIC · ⌖ 7.39 AIC · ⊞ 21.8K
Comment /review to run again
| baseModel := strings.SplitN(model, "?", 2)[0] | ||
| usesGitHubInference := strings.HasPrefix(baseModel, "copilot/") || | ||
| NewCodexEngine().ResolveLLMProvider(data) == LLMProviderGitHub | ||
| if !usesGitHubInference || strings.Contains(baseModel, "codex") { |
There was a problem hiding this comment.
This warning gate is using strings.Contains(baseModel, "codex") as the compatibility test, which means any future or malformed GitHub model name that happens to include codex will bypass the warning even if the runtime still lacks the capabilities Codex needs.
💡 Why this should be tightened
Right now the guard is effectively a naming heuristic, not a compatibility check. That is fragile in exactly the place this change is supposed to protect: model selection. If GitHub adds a general-purpose model whose marketing name includes codex, or if an alias expands to such a name, the compiler will suppress the warning and the workflow will still fail later at runtime.
A safer pattern here is to check an explicit allowlist/predicate for known Codex-capable GitHub models instead of relying on a substring:
func isCompatibleCodexGitHubModel(model string) bool {
baseModel := strings.SplitN(strings.ToLower(strings.TrimSpace(model)), "?", 2)[0]
return baseModel == "copilot/gpt-5.3-codex"
}If you need this to be forward-compatible, wire that predicate to the model inventory / alias resolution layer rather than embedding a free-form string match in the warning path.
🏗️ ADR requiredI did not find an existing ADR in the PR body, and the current branch ADR set does not cover this change. Evidence used
Action taken
Next action for the author
Warning Firewall blocked 2 domainsThe following domains were blocked by the firewall during workflow execution:
To allow these domains, add them to the network:
allowed:
- defaults
- "github.com"
- "proxy.golang.org"See Network Configuration for more information.
|
There was a problem hiding this comment.
Skills-Based Review 🧠
Applied /tdd and /codebase-design to the core change (pkg/workflow/configured_model_validation.go + tests). The rest of the diff is a mechanical model rename (copilot/mai-code-1-flash-picker to copilot/gpt-5.3-codex) across 46 workflow .md/.lock.yml pairs plus docs.
📋 Review notes
The new warnCodexCopilotModelCompatibility function is small, single-purpose, and well covered: the accompanying test table exercises general-purpose Copilot models, copilot/auto, an explicit GitHub provider override, a Codex-capable model, runtime expression models, non-Copilot providers, and the copilot engine (no warning) — good edge-case coverage per /tdd.
The function fits the existing Compiler.warn* pattern (warnUnknownConfiguredModels sits right next to it and follows the same stderr/IncrementWarningCount shape), so it is consistent with the codebase's existing vocabulary per /codebase-design. Ran go test ./pkg/workflow/ -run TestWarnCodex and -run TestWarnUnknownConfiguredModels locally — both pass.
No blocking issues found. The bulk lock-file/frontmatter changes are auto-generated and match the intended model swap.
Warning
Firewall blocked 1 domain
The following domain was blocked by the firewall during workflow execution:
github.com
To allow these domains, add them to the network.allowed list in your workflow frontmatter:
network:
allowed:
- defaults
- "github.com"See Network Configuration for more information.
🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer · copilot · sonnet50 · 46.5 AIC · ⌖ 14.6 AIC · ⊞ 10.3K
Comment /matt to run again
There was a problem hiding this comment.
🟢 Approval recommended
The compiler warning logic is targeted and covered by tests, and the workflow/doc updates consistently migrate to a Codex-capable model.
Pull request overview
Adds compiler-time guidance to help Codex workflows avoid incompatible general-purpose copilot/* models when routing through GitHub-hosted inference, and migrates affected workflows/docs to a Codex-capable model to prevent runtime tool/capability failures.
Changes:
- Add a Codex + GitHub inference compatibility warning when a non-Codex Copilot model is configured (skipping runtime expressions and Codex-capable model names).
- Update many Codex workflows to use
copilot/gpt-5.3-codexand regenerate corresponding lock files. - Update Codex documentation/glossary and ship the behavior change via a changeset.
File summaries
| File | Description |
|---|---|
| pkg/workflow/configured_model_validation.go | Adds compiler warning for Codex + GitHub inference when a non-Codex Copilot model is selected. |
| pkg/workflow/configured_model_validation_test.go | Adds unit coverage for the new Codex/Copilot compatibility warning behavior. |
| pkg/workflow/compiler_orchestrator_workflow.go | Wires the new warning into workflow validation. |
| docs/src/content/docs/reference/glossary.md | Documents that Codex via GitHub-hosted inference requires Codex-capable models. |
| docs/src/content/docs/engines/codex.md | Updates Codex engine docs to use a Codex-capable copilot/* model example. |
| .github/workflows/workflow-normalizer.md | Migrates workflow model to copilot/gpt-5.3-codex. |
| .github/workflows/workflow-normalizer.lock.yml | Regenerates lock file to reflect the model migration. |
| .github/workflows/windows.md | Migrates workflow model to copilot/gpt-5.3-codex. |
| .github/workflows/windows.lock.yml | Regenerates lock file to reflect the model migration. |
| .github/workflows/windows-grower.md | Migrates workflow model to copilot/gpt-5.3-codex. |
| .github/workflows/windows-grower.lock.yml | Regenerates lock file to reflect the model migration. |
| .github/workflows/weekly-safe-outputs-spec-review.md | Migrates workflow model to copilot/gpt-5.3-codex. |
| .github/workflows/weekly-safe-outputs-spec-review.lock.yml | Regenerates lock file to reflect the model migration. |
| .github/workflows/weekly-network-domains-audit.md | Migrates workflow model to copilot/gpt-5.3-codex. |
| .github/workflows/weekly-network-domains-audit.lock.yml | Regenerates lock file to reflect the model migration. |
| .github/workflows/weekly-issue-summary.md | Migrates workflow model to copilot/gpt-5.3-codex. |
| .github/workflows/weekly-issue-summary.lock.yml | Regenerates lock file to reflect the model migration. |
| .github/workflows/video-analyzer.md | Migrates workflow model to copilot/gpt-5.3-codex. |
| .github/workflows/video-analyzer.lock.yml | Regenerates lock file to reflect the model migration. |
| .github/workflows/update-astro.md | Migrates workflow model to copilot/gpt-5.3-codex. |
| .github/workflows/update-astro.lock.yml | Regenerates lock file to reflect the model migration. |
| .github/workflows/terminal-stylist.md | Migrates workflow model to copilot/gpt-5.3-codex. |
| .github/workflows/terminal-stylist.lock.yml | Regenerates lock file to reflect the model migration. |
| .github/workflows/super-linter.md | Migrates workflow model to copilot/gpt-5.3-codex. |
| .github/workflows/super-linter.lock.yml | Regenerates lock file to reflect the model migration. |
| .github/workflows/sub-issue-closer.md | Migrates workflow model to copilot/gpt-5.3-codex. |
| .github/workflows/stale-repo-identifier.md | Migrates workflow model to copilot/gpt-5.3-codex. |
| .github/workflows/smoke-test-tools.md | Migrates workflow model to copilot/gpt-5.3-codex. |
| .github/workflows/smoke-test-tools.lock.yml | Regenerates lock file to reflect the model migration. |
| .github/workflows/smoke-temporary-id.md | Migrates workflow model to copilot/gpt-5.3-codex. |
| .github/workflows/smoke-service-ports.md | Migrates workflow model to copilot/gpt-5.3-codex. |
| .github/workflows/smoke-service-ports.lock.yml | Regenerates lock file to reflect the model migration. |
| .github/workflows/smoke-otel-backends.md | Migrates workflow model to copilot/gpt-5.3-codex. |
| .github/workflows/smoke-otel-backends.lock.yml | Regenerates lock file to reflect the model migration. |
| .github/workflows/smoke-drive.md | Migrates workflow model to copilot/gpt-5.3-codex. |
| .github/workflows/smoke-drive.lock.yml | Regenerates lock file to reflect the model migration. |
| .github/workflows/smoke-ci.md | Migrates workflow model to copilot/gpt-5.3-codex. |
| .github/workflows/smoke-ci.lock.yml | Regenerates lock file to reflect the model migration. |
| .github/workflows/skillet.md | Migrates workflow model to copilot/gpt-5.3-codex. |
| .github/workflows/skillet.lock.yml | Regenerates lock file to reflect the model migration. |
| .github/workflows/sighthound-security-scan.md | Migrates workflow model to copilot/gpt-5.3-codex. |
| .github/workflows/sighthound-security-scan.lock.yml | Regenerates lock file to reflect the model migration. |
| .github/workflows/security-review.md | Migrates workflow model to copilot/gpt-5.3-codex. |
| .github/workflows/research.md | Migrates workflow model to copilot/gpt-5.3-codex. |
| .github/workflows/repo-tree-map.md | Migrates workflow model to copilot/gpt-5.3-codex. |
| .github/workflows/repo-tree-map.lock.yml | Regenerates lock file to reflect the model migration. |
| .github/workflows/refactoring-cadence.md | Migrates workflow model to copilot/gpt-5.3-codex. |
| .github/workflows/refactoring-cadence.lock.yml | Regenerates lock file to reflect the model migration. |
| .github/workflows/python-data-charts.md | Migrates workflow model to copilot/gpt-5.3-codex. |
| .github/workflows/python-data-charts.lock.yml | Regenerates lock file to reflect the model migration. |
| .github/workflows/purelock.md | Migrates workflow model to copilot/gpt-5.3-codex. |
| .github/workflows/ponytail-reviewer.md | Migrates workflow model to copilot/gpt-5.3-codex. |
| .github/workflows/ponytail-reviewer.lock.yml | Regenerates lock file to reflect the model migration. |
| .github/workflows/pdf-summary.md | Migrates workflow model to copilot/gpt-5.3-codex. |
| .github/workflows/pdf-summary.lock.yml | Regenerates lock file to reflect the model migration. |
| .github/workflows/outcome-collector.md | Migrates workflow model to copilot/gpt-5.3-codex. |
| .github/workflows/outcome-collector.lock.yml | Regenerates lock file to reflect the model migration. |
| .github/workflows/metrics-collector.md | Migrates workflow model to copilot/gpt-5.3-codex. |
| .github/workflows/metrics-collector.lock.yml | Regenerates lock file to reflect the model migration. |
| .github/workflows/issue-triage-agent.md | Migrates workflow model to copilot/gpt-5.3-codex. |
| .github/workflows/issue-monster.md | Migrates workflow model to copilot/gpt-5.3-codex. |
| .github/workflows/github-remote-mcp-auth-test.md | Migrates workflow model to copilot/gpt-5.3-codex. |
| .github/workflows/github-remote-mcp-auth-test.lock.yml | Regenerates lock file to reflect the model migration. |
| .github/workflows/front-page-copy-guard.md | Migrates workflow model to copilot/gpt-5.3-codex. |
| .github/workflows/front-page-copy-guard.lock.yml | Regenerates lock file to reflect the model migration. |
| .github/workflows/feature-grower.md | Migrates workflow model to copilot/gpt-5.3-codex. |
| .github/workflows/feature-grower.lock.yml | Regenerates lock file to reflect the model migration. |
| .github/workflows/duplicate-code-detector.md | Updates workflow example from copilot/auto to a Codex-capable Copilot model. |
| .github/workflows/daily-windows-terminal-integration-builder.md | Migrates workflow model to copilot/gpt-5.3-codex. |
| .github/workflows/daily-windows-defender-scan.md | Migrates workflow model to copilot/gpt-5.3-codex. |
| .github/workflows/daily-windows-defender-scan.lock.yml | Regenerates lock file to reflect the model migration. |
| .github/workflows/daily-storify.md | Migrates workflow model to copilot/gpt-5.3-codex. |
| .github/workflows/daily-max-ai-credits-test.md | Migrates workflow model to copilot/gpt-5.3-codex. |
| .github/workflows/daily-max-ai-credits-test.lock.yml | Regenerates lock file to reflect the model migration. |
| .github/workflows/daily-evals-report.md | Migrates workflow model to copilot/gpt-5.3-codex. |
| .github/workflows/daily-credit-limit-test.md | Migrates workflow model to copilot/gpt-5.3-codex. |
| .github/workflows/daily-credit-limit-test.lock.yml | Regenerates lock file to reflect the model migration. |
| .github/workflows/daily-architecture-diagram.md | Migrates workflow model to copilot/gpt-5.3-codex. |
| .github/workflows/copilot-centralization-optimizer.md | Migrates workflow model to copilot/gpt-5.3-codex. |
| .github/workflows/copilot-centralization-optimizer.lock.yml | Regenerates lock file to reflect the model migration. |
| .github/workflows/copilot-centralization-drilldown.md | Migrates workflow model to copilot/gpt-5.3-codex. |
| .github/workflows/copilot-centralization-drilldown.lock.yml | Regenerates lock file to reflect the model migration. |
| .github/workflows/api-consumption-report.md | Migrates workflow model to copilot/gpt-5.3-codex. |
| .changeset/codex-copilot-model.md | Records the new warning/recommendation behavior as a minor change. |
Review details
- Files reviewed: 99/99 changed files
- Comments generated: 1
- Review effort level: Lite
💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.
| reader, writer, err := os.Pipe() | ||
| require.NoError(t, err) | ||
| os.Stderr = writer |
There was a problem hiding this comment.
Reviewed the Codex/Copilot model compatibility change end-to-end (compiler warning logic, orchestrator wiring, docs, and workflow migrations).
Findings:
warnCodexCopilotModelCompatibilitycorrectly scopes to thecodexengine, skips runtime expressions (${{ ... }}) and blank models, and only warns when GitHub inference is used with a non-Codex model. Logic matches the accompanying unit tests (TestWarnCodexCopilotModelCompatibility), including thecopilot/auto, explicitLLMProvider: github, andcopilot/gpt-5.3-codex?effort=high(query-string) cases.warnUnknownConfiguredModelscleanly delegates to the optionalconfiguredModelValidatorhook without behavior change when unset — verified byTestWarnUnknownConfiguredModelsWithoutInventory.- Wiring in
validateWorkflowBuildContext(compiler_orchestrator_workflow.go) calls both warnings after model alias validation and before engine settings validation — no ordering issues. - The 46 workflow migrations to
copilot/gpt-5.3-codexand regenerated lock files are mechanical and consistent with the new compiler guidance. - Documentation updates in
docs/src/content/docs/engines/codex.mdand the glossary correctly reflect the new model-capability requirement. - Ran
go test ./pkg/workflow/... -run TestWarnCodexCopilotModelCompatibility|TestWarnUnknownConfiguredModels— all pass.
No blocking issues found. This is a backend/compiler change with no UI surface, so Impeccable UI-focused modes (critique/audit/harden/etc.) don't apply; reviewed for correctness and test coverage instead.
Warning
Firewall blocked 1 domain
The following domain was blocked by the firewall during workflow execution:
github.com
To allow these domains, add them to the network.allowed list in your workflow frontmatter:
network:
allowed:
- defaults
- "github.com"See Network Configuration for more information.
🧵 Reviewed using Impeccable skills by Impeccable Skills Reviewer · copilot · sonnet50 · 66.9 AIC · ⌖ 13.3 AIC · ⊞ 8.3K
|
Category: bug | Risk: high | Score: 74 | Action: fast_track | Batch: workflow-reliability. Warning Firewall blocked 2 domainsThe following domains were blocked by the firewall during workflow execution:
[!TIP] tools:
github:
mode: gh-proxySee GitHub Tools for more information on To allow these domains, add them to the network:
allowed:
- defaults
- "api.github.com"
- "github.com"See Network Configuration for more information.
|
|
Great work, SummaryThis change addresses a critical runtime issue: Codex workflows require Codex-capable models (e.g., ✅ Compiler validation — Added guardrails in ✅ Workflow migration — Systematically updated 46 affected workflows to use the Codex-capable model, with regenerated lock files. ✅ Documentation & ADR — Replaced ✅ Test coverage — Added 90 lines of unit tests in Quality Signals
This looks ready for review by the core team. The mechanical bulk of lock-file updates is expected and the core logic addition is well-tested. Warning Firewall blocked 2 domainsThe following domains were blocked by the firewall during workflow execution:
[!TIP] tools:
github:
mode: gh-proxySee GitHub Tools for more information on To allow these domains, add them to the network:
allowed:
- defaults
- "api.github.com"
- "github.com"See Network Configuration for more information.
|
Codex requires Codex-capable models for features such as custom tools. General-purpose
copilot/*models can fail at runtime with unsupported-model or tool errors.Compiler guidance
Workflow migration
copilot/gpt-5.3-codex.Documentation
copilot/autoexamples with a Codex-compatible selection.