diff --git a/actions/setup/js/models.json b/actions/setup/js/models.json index bcb7d7fc321..a669a98ea9e 100644 --- a/actions/setup/js/models.json +++ b/actions/setup/js/models.json @@ -262,7 +262,25 @@ "cost": { "input": "2.5e-07", "output": "2e-06", - "cache_read": "2.5000000000000002e-08" + "cache_read": "2.5e-08" + }, + "provider_type": "openai", + "wire_api": "responses" + }, + "mini": { + "cost": { + "input": "2.5e-07", + "output": "2e-06", + "cache_read": "2.5e-08" + }, + "provider_type": "openai", + "wire_api": "responses" + }, + "small": { + "cost": { + "input": "2.5e-07", + "output": "2e-06", + "cache_read": "2.5e-08" }, "provider_type": "openai", "wire_api": "responses" diff --git a/docs/troubleshooting/copilot-model-alias-failure-investigation.md b/docs/troubleshooting/copilot-model-alias-failure-investigation.md new file mode 100644 index 00000000000..72abbf827a6 --- /dev/null +++ b/docs/troubleshooting/copilot-model-alias-failure-investigation.md @@ -0,0 +1,83 @@ +# Investigation: Copilot failure with model alias `small` + +## Scope + +- Workflow run: https://github.com/github/gh-aw/actions/runs/28986878657 +- Failing job: https://github.com/github/gh-aw/actions/runs/28986878657/job/86018033327 +- Failing step: `Execute GitHub Copilot CLI` (step 24) +- Workflow file: `/home/runner/work/gh-aw/gh-aw/.github/workflows/smoke-copilot-small.md` + +## Observed failure + +The run fails before the Copilot harness starts. + +``` +[ERROR] Error: model 'small' is unsupported or unrecognized by this AWF version. Did you mean 'gpt-4'? +``` + +The error is emitted by `awf` and the step exits with code `1`. + +## What code emits this error + +- In this repository, the Copilot execution step is generated by: + - `pkg/workflow/copilot_engine_execution.go` (sets `COPILOT_MODEL`) + - `pkg/workflow/awf_helpers.go` (`BuildAWFCommand`, emits `awf --config ...`) +- The failing job log shows that exact generated command runs: + - `awf --config "${RUNNER_TEMP}/gh-aw/awf-config.json" ...` + - with `COPILOT_MODEL: small` +- The literal message text + - `Error: model 'small' is unsupported or unrecognized by this AWF version. Did you mean 'gpt-4'?` + is **not** present in `github/gh-aw` source (only in this investigation doc). +- Conclusion: this string is generated by the external `awf` binary/runtime (from `gh-aw-firewall`), and `gh-aw` is surfacing it via the generated AWF invocation. + +## Root cause in `github/gh-aw-firewall` + +The failure is caused by validation ordering and data flow inside AWF: + +1. `src/commands/validators/config-assembly.ts` reads `COPILOT_MODEL` and calls: + - `validateCopilotModel(copilotModel)` + - on failure, it logs `validation.message` and exits (`process.exit(1)`). +2. `src/copilot-model.ts` generates the line-15 failure text (`Error: model 'small' is unsupported or unrecognized by this AWF version...`) and validates against a hardcoded `SUPPORTED_COPILOT_MODELS` set plus a limited retired-alias map. +3. That validator does **not** read runtime alias config (`config.modelAliases` / `AWF_MODEL_ALIASES`), so aliases such as `small` are not considered valid at this preflight step. +4. AWF does carry alias config forward (`modelAliases` is assembled and exported as `AWF_MODEL_ALIASES`), but that is for api-proxy model resolution later in the request path, after the failing preflight check. + +So `small` fails before alias resolution can run, while concrete models in the allowlist (for example `gpt-5-mini`) can pass this validator. + +## Evidence gathered + +1. The workflow explicitly sets: + +```yaml +engine: + id: copilot + model: small +``` + +2. The runtime environment confirms: + +```text +COPILOT_MODEL: small +``` + +3. The generated AWF config in the failing step includes builtin model aliases, including: + +```json +"small": ["mini"] +``` + +## Current hypothesis + +`small` is accepted by gh-aw as a valid alias and is emitted into AWF config, but AWF v0.27.27 rejects `COPILOT_MODEL=small` in preflight model validation (`validateCopilotModel`) before alias resolution (`AWF_MODEL_ALIASES`) is applied in the api-proxy path. + +## Why this matters + +The smoke workflow intended to validate Copilot behavior with a compact alias currently validates an integration mismatch instead, causing a deterministic failure. + +## Suggested follow-up validation + +1. Re-run the smoke workflow with `engine.model: mini`. +2. Re-run with `engine.model: gpt-5-mini`. +3. Compare whether failure only affects alias names vs. concrete provider model identifiers. +4. If alias handling is not supported in this AWF path, either: + - update AWF to a version that resolves aliases in this path, or + - keep smoke workflow on a concrete model and add a dedicated alias-compatibility test once AWF support is confirmed. diff --git a/pkg/cli/data/models.json b/pkg/cli/data/models.json index 132fbb78cb3..e0f55de370a 100644 --- a/pkg/cli/data/models.json +++ b/pkg/cli/data/models.json @@ -381,7 +381,25 @@ "cost": { "input": "2.5e-07", "output": "2e-06", - "cache_read": "2.5000000000000002e-08" + "cache_read": "2.5e-08" + }, + "provider_type": "openai", + "wire_api": "responses" + }, + "mini": { + "cost": { + "input": "2.5e-07", + "output": "2e-06", + "cache_read": "2.5e-08" + }, + "provider_type": "openai", + "wire_api": "responses" + }, + "small": { + "cost": { + "input": "2.5e-07", + "output": "2e-06", + "cache_read": "2.5e-08" }, "provider_type": "openai", "wire_api": "responses"