From 96f05870bcc5e6dc304931eb830f80f97adbd8de Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Thu, 9 Jul 2026 01:51:04 +0000 Subject: [PATCH 1/9] docs: add investigation report for copilot model alias failure Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com> --- ...pilot-model-alias-failure-investigation.md | 57 +++++++++++++++++++ 1 file changed, 57 insertions(+) create mode 100644 docs/troubleshooting/copilot-model-alias-failure-investigation.md diff --git a/docs/troubleshooting/copilot-model-alias-failure-investigation.md b/docs/troubleshooting/copilot-model-alias-failure-investigation.md new file mode 100644 index 00000000000..56eeb2a0f75 --- /dev/null +++ b/docs/troubleshooting/copilot-model-alias-failure-investigation.md @@ -0,0 +1,57 @@ +# Investigation: Copilot failure with model alias `small` + +## Scope + +- Workflow run: https://github.com/github/gh-aw/actions/runs/28986878657 +- Failing job: https://github.com/github/gh-aw/actions/runs/28986878657/job/86018033327 +- Failing step: `Execute GitHub Copilot CLI` (step 24) +- Workflow file: `/home/runner/work/gh-aw/gh-aw/.github/workflows/smoke-copilot-small.md` + +## Observed failure + +The run fails before the Copilot harness starts. + +``` +[ERROR] Error: model 'small' is unsupported or unrecognized by this AWF version. Did you mean 'gpt-4'? +``` + +The error is emitted by `awf` and the step exits with code `1`. + +## Evidence gathered + +1. The workflow explicitly sets: + +```yaml +engine: + id: copilot + model: small +``` + +2. The runtime environment confirms: + +```text +COPILOT_MODEL: small +``` + +3. The generated AWF config in the failing step includes builtin model aliases, including: + +```json +"small": ["mini"] +``` + +## Current hypothesis + +`small` is accepted by gh-aw as a valid alias and is emitted into AWF config, but AWF v0.27.27 still rejects the incoming model string (`small`) before alias resolution is applied for this execution path. + +## Why this matters + +The smoke workflow intended to validate Copilot behavior with a compact alias currently validates an integration mismatch instead, causing a deterministic failure. + +## Suggested follow-up validation + +1. Re-run the smoke workflow with `engine.model: mini`. +2. Re-run with `engine.model: gpt-5-mini`. +3. Compare whether failure only affects alias names vs. concrete provider model identifiers. +4. If alias handling is not supported in this AWF path, either: + - update AWF to a version that resolves aliases in this path, or + - keep smoke workflow on a concrete model and add a dedicated alias-compatibility test once AWF support is confirmed. From 0b3f244a518e7953fd13799aac36295b06821773 Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Thu, 9 Jul 2026 02:17:49 +0000 Subject: [PATCH 2/9] docs: trace small model error to awf runtime emitter Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com> --- .../copilot-model-alias-failure-investigation.md | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/docs/troubleshooting/copilot-model-alias-failure-investigation.md b/docs/troubleshooting/copilot-model-alias-failure-investigation.md index 56eeb2a0f75..a4649d8188b 100644 --- a/docs/troubleshooting/copilot-model-alias-failure-investigation.md +++ b/docs/troubleshooting/copilot-model-alias-failure-investigation.md @@ -17,6 +17,19 @@ The run fails before the Copilot harness starts. The error is emitted by `awf` and the step exits with code `1`. +## What code emits this error + +- In this repository, the Copilot execution step is generated by: + - `/home/runner/work/gh-aw/gh-aw/pkg/workflow/copilot_engine_execution.go` (sets `COPILOT_MODEL`) + - `/home/runner/work/gh-aw/gh-aw/pkg/workflow/awf_helpers.go` (`BuildAWFCommand`, emits `awf --config ...`) +- The failing job log shows that exact generated command runs: + - `awf --config "${RUNNER_TEMP}/gh-aw/awf-config.json" ...` + - with `COPILOT_MODEL: small` +- The literal message text + - `Error: model 'small' is unsupported or unrecognized by this AWF version. Did you mean 'gpt-4'?` + is **not** present in `github/gh-aw` source (only in this investigation doc). +- Conclusion: this string is generated by the external `awf` binary/runtime (from `gh-aw-firewall`), and `gh-aw` is surfacing it via the generated AWF invocation. + ## Evidence gathered 1. The workflow explicitly sets: From b827333b68ec08531703687322d8a4a5f031c52f Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Thu, 9 Jul 2026 02:20:09 +0000 Subject: [PATCH 3/9] docs: use repo-relative paths in error-source notes Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com> --- .../copilot-model-alias-failure-investigation.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/troubleshooting/copilot-model-alias-failure-investigation.md b/docs/troubleshooting/copilot-model-alias-failure-investigation.md index a4649d8188b..1a15fb4114e 100644 --- a/docs/troubleshooting/copilot-model-alias-failure-investigation.md +++ b/docs/troubleshooting/copilot-model-alias-failure-investigation.md @@ -20,8 +20,8 @@ The error is emitted by `awf` and the step exits with code `1`. ## What code emits this error - In this repository, the Copilot execution step is generated by: - - `/home/runner/work/gh-aw/gh-aw/pkg/workflow/copilot_engine_execution.go` (sets `COPILOT_MODEL`) - - `/home/runner/work/gh-aw/gh-aw/pkg/workflow/awf_helpers.go` (`BuildAWFCommand`, emits `awf --config ...`) + - `pkg/workflow/copilot_engine_execution.go` (sets `COPILOT_MODEL`) + - `pkg/workflow/awf_helpers.go` (`BuildAWFCommand`, emits `awf --config ...`) - The failing job log shows that exact generated command runs: - `awf --config "${RUNNER_TEMP}/gh-aw/awf-config.json" ...` - with `COPILOT_MODEL: small` From b972fd73d8cf41220a98b0575f1cc15eceaee9ea Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Thu, 9 Jul 2026 02:34:14 +0000 Subject: [PATCH 4/9] docs: add gh-aw-firewall root-cause chain for small model alias failure Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com> --- .../copilot-model-alias-failure-investigation.md | 15 ++++++++++++++- 1 file changed, 14 insertions(+), 1 deletion(-) diff --git a/docs/troubleshooting/copilot-model-alias-failure-investigation.md b/docs/troubleshooting/copilot-model-alias-failure-investigation.md index 1a15fb4114e..aa5e3c2bae4 100644 --- a/docs/troubleshooting/copilot-model-alias-failure-investigation.md +++ b/docs/troubleshooting/copilot-model-alias-failure-investigation.md @@ -30,6 +30,19 @@ The error is emitted by `awf` and the step exits with code `1`. is **not** present in `github/gh-aw` source (only in this investigation doc). - Conclusion: this string is generated by the external `awf` binary/runtime (from `gh-aw-firewall`), and `gh-aw` is surfacing it via the generated AWF invocation. +## Root cause in `github/gh-aw-firewall` + +The failure is caused by validation ordering and data flow inside AWF: + +1. `src/commands/validators/config-assembly.ts` reads `COPILOT_MODEL` and calls: + - `validateCopilotModel(copilotModel)` + - on failure, it logs `validation.message` and exits (`process.exit(1)`). +2. `src/copilot-model.ts` generates the exact error string and validates against a hardcoded `SUPPORTED_COPILOT_MODELS` set plus a small retired-alias map. +3. That validator does **not** read runtime alias config (`config.modelAliases` / `AWF_MODEL_ALIASES`), so aliases such as `small` are not considered valid at this preflight step. +4. AWF does carry alias config forward (`modelAliases` is assembled and exported as `AWF_MODEL_ALIASES`), but that is for api-proxy model resolution later in the request path, after the failing preflight check. + +So `small` fails before alias resolution can run, while concrete models in the allowlist (for example `gpt-5-mini`) can pass this validator. + ## Evidence gathered 1. The workflow explicitly sets: @@ -54,7 +67,7 @@ COPILOT_MODEL: small ## Current hypothesis -`small` is accepted by gh-aw as a valid alias and is emitted into AWF config, but AWF v0.27.27 still rejects the incoming model string (`small`) before alias resolution is applied for this execution path. +`small` is accepted by gh-aw as a valid alias and is emitted into AWF config, but AWF v0.27.27 rejects `COPILOT_MODEL=small` in preflight model validation (`validateCopilotModel`) before alias resolution (`AWF_MODEL_ALIASES`) is applied in the api-proxy path. ## Why this matters From 379e34710b938223eb2a9d3a54b5322b49e4e418 Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Thu, 9 Jul 2026 02:36:44 +0000 Subject: [PATCH 5/9] docs: clarify retired-alias wording in root-cause note Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com> --- .../copilot-model-alias-failure-investigation.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/troubleshooting/copilot-model-alias-failure-investigation.md b/docs/troubleshooting/copilot-model-alias-failure-investigation.md index aa5e3c2bae4..100e7dfce83 100644 --- a/docs/troubleshooting/copilot-model-alias-failure-investigation.md +++ b/docs/troubleshooting/copilot-model-alias-failure-investigation.md @@ -37,7 +37,7 @@ The failure is caused by validation ordering and data flow inside AWF: 1. `src/commands/validators/config-assembly.ts` reads `COPILOT_MODEL` and calls: - `validateCopilotModel(copilotModel)` - on failure, it logs `validation.message` and exits (`process.exit(1)`). -2. `src/copilot-model.ts` generates the exact error string and validates against a hardcoded `SUPPORTED_COPILOT_MODELS` set plus a small retired-alias map. +2. `src/copilot-model.ts` generates the exact error string and validates against a hardcoded `SUPPORTED_COPILOT_MODELS` set plus a limited retired-alias map. 3. That validator does **not** read runtime alias config (`config.modelAliases` / `AWF_MODEL_ALIASES`), so aliases such as `small` are not considered valid at this preflight step. 4. AWF does carry alias config forward (`modelAliases` is assembled and exported as `AWF_MODEL_ALIASES`), but that is for api-proxy model resolution later in the request path, after the failing preflight check. From b3d769aba1aecfc2620ed229d1029204bcb5be89 Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Thu, 9 Jul 2026 02:39:07 +0000 Subject: [PATCH 6/9] docs: clarify which firewall error string is emitted Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com> --- .../copilot-model-alias-failure-investigation.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/troubleshooting/copilot-model-alias-failure-investigation.md b/docs/troubleshooting/copilot-model-alias-failure-investigation.md index 100e7dfce83..72abbf827a6 100644 --- a/docs/troubleshooting/copilot-model-alias-failure-investigation.md +++ b/docs/troubleshooting/copilot-model-alias-failure-investigation.md @@ -37,7 +37,7 @@ The failure is caused by validation ordering and data flow inside AWF: 1. `src/commands/validators/config-assembly.ts` reads `COPILOT_MODEL` and calls: - `validateCopilotModel(copilotModel)` - on failure, it logs `validation.message` and exits (`process.exit(1)`). -2. `src/copilot-model.ts` generates the exact error string and validates against a hardcoded `SUPPORTED_COPILOT_MODELS` set plus a limited retired-alias map. +2. `src/copilot-model.ts` generates the line-15 failure text (`Error: model 'small' is unsupported or unrecognized by this AWF version...`) and validates against a hardcoded `SUPPORTED_COPILOT_MODELS` set plus a limited retired-alias map. 3. That validator does **not** read runtime alias config (`config.modelAliases` / `AWF_MODEL_ALIASES`), so aliases such as `small` are not considered valid at this preflight step. 4. AWF does carry alias config forward (`modelAliases` is assembled and exported as `AWF_MODEL_ALIASES`), but that is for api-proxy model resolution later in the request path, after the failing preflight check. From b9f0b2c3531f124729f86b687ea9266ab3502ee3 Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Thu, 9 Jul 2026 02:57:47 +0000 Subject: [PATCH 7/9] workaround: expand copilot small/mini models in catalogs Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com> --- actions/setup/js/models.json | 18 ++++++++++++++++++ pkg/cli/data/models.json | 18 ++++++++++++++++++ 2 files changed, 36 insertions(+) diff --git a/actions/setup/js/models.json b/actions/setup/js/models.json index bcb7d7fc321..97129a936f8 100644 --- a/actions/setup/js/models.json +++ b/actions/setup/js/models.json @@ -267,6 +267,24 @@ "provider_type": "openai", "wire_api": "responses" }, + "mini": { + "cost": { + "input": "2.5e-07", + "output": "2e-06", + "cache_read": "2.5000000000000002e-08" + }, + "provider_type": "openai", + "wire_api": "responses" + }, + "small": { + "cost": { + "input": "2.5e-07", + "output": "2e-06", + "cache_read": "2.5000000000000002e-08" + }, + "provider_type": "openai", + "wire_api": "responses" + }, "gpt-5.2": { "cost": { "input": "1.75e-06", diff --git a/pkg/cli/data/models.json b/pkg/cli/data/models.json index 132fbb78cb3..18b7fc943c1 100644 --- a/pkg/cli/data/models.json +++ b/pkg/cli/data/models.json @@ -386,6 +386,24 @@ "provider_type": "openai", "wire_api": "responses" }, + "mini": { + "cost": { + "input": "2.5e-07", + "output": "2e-06", + "cache_read": "2.5000000000000002e-08" + }, + "provider_type": "openai", + "wire_api": "responses" + }, + "small": { + "cost": { + "input": "2.5e-07", + "output": "2e-06", + "cache_read": "2.5000000000000002e-08" + }, + "provider_type": "openai", + "wire_api": "responses" + }, "gpt-5.2": { "cost": { "input": "1.75e-06", From e79674f99a91fa751404b8df4bcfcad12f66e23d Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Thu, 9 Jul 2026 03:00:24 +0000 Subject: [PATCH 8/9] chore: normalize mini/small cache_read formatting Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com> --- actions/setup/js/models.json | 4 ++-- pkg/cli/data/models.json | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/actions/setup/js/models.json b/actions/setup/js/models.json index 97129a936f8..0f5ab694590 100644 --- a/actions/setup/js/models.json +++ b/actions/setup/js/models.json @@ -271,7 +271,7 @@ "cost": { "input": "2.5e-07", "output": "2e-06", - "cache_read": "2.5000000000000002e-08" + "cache_read": "2.5e-08" }, "provider_type": "openai", "wire_api": "responses" @@ -280,7 +280,7 @@ "cost": { "input": "2.5e-07", "output": "2e-06", - "cache_read": "2.5000000000000002e-08" + "cache_read": "2.5e-08" }, "provider_type": "openai", "wire_api": "responses" diff --git a/pkg/cli/data/models.json b/pkg/cli/data/models.json index 18b7fc943c1..999db89f0a4 100644 --- a/pkg/cli/data/models.json +++ b/pkg/cli/data/models.json @@ -390,7 +390,7 @@ "cost": { "input": "2.5e-07", "output": "2e-06", - "cache_read": "2.5000000000000002e-08" + "cache_read": "2.5e-08" }, "provider_type": "openai", "wire_api": "responses" @@ -399,7 +399,7 @@ "cost": { "input": "2.5e-07", "output": "2e-06", - "cache_read": "2.5000000000000002e-08" + "cache_read": "2.5e-08" }, "provider_type": "openai", "wire_api": "responses" From 0b3a04e4647f219928ae199bd8c86684fc23f6a2 Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Thu, 9 Jul 2026 03:03:06 +0000 Subject: [PATCH 9/9] chore: align gpt-5-mini cache_read precision Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com> --- actions/setup/js/models.json | 2 +- pkg/cli/data/models.json | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/actions/setup/js/models.json b/actions/setup/js/models.json index 0f5ab694590..a669a98ea9e 100644 --- a/actions/setup/js/models.json +++ b/actions/setup/js/models.json @@ -262,7 +262,7 @@ "cost": { "input": "2.5e-07", "output": "2e-06", - "cache_read": "2.5000000000000002e-08" + "cache_read": "2.5e-08" }, "provider_type": "openai", "wire_api": "responses" diff --git a/pkg/cli/data/models.json b/pkg/cli/data/models.json index 999db89f0a4..e0f55de370a 100644 --- a/pkg/cli/data/models.json +++ b/pkg/cli/data/models.json @@ -381,7 +381,7 @@ "cost": { "input": "2.5e-07", "output": "2e-06", - "cache_read": "2.5000000000000002e-08" + "cache_read": "2.5e-08" }, "provider_type": "openai", "wire_api": "responses"