Add CI guard to prevent SUPPORTED_COPILOT_MODELS drift from pricing catalog by Copilot · Pull Request #6039 · github/gh-aw-firewall

Copilot · 2026-07-09T02:40:33Z

When a new model is added to containers/api-proxy/ai-credits-pricing.js but not to SUPPORTED_COPILOT_MODELS in src/copilot-model.ts, AWF rejects valid COPILOT_MODEL values with a misleading "retired or unsupported" error before containers start. This recurred with mai-code-1-flash (#5831).

Changes

Single source of truth comment — SUPPORTED_COPILOT_MODELS now documents exactly which related files need updating when adding or retiring a model:

/**
 * Single source of truth for all Copilot CLI completion models...
 * When adding a new model here, also update:
 *   - containers/api-proxy/ai-credits-pricing.js  (AI-credits billing)
 *   - docs/model-api-mapping.json                  (endpoint routing)
 * When retiring a model, move it to RETIRED_COPILOT_MODEL_ALIASES and mirror
 * in containers/api-proxy/guards/retired-model-guard.js.
 */

CI guard (src/copilot-model-catalog-sync.test.ts) — fails if any key in ai-credits-pricing.js is absent from SUPPORTED_COPILOT_MODELS and not in an explicit INTENTIONALLY_EXCLUDED_FROM_SUPPORTED set. The exclusion set requires an inline comment explaining why the model isn't a Copilot CLI completion model (e.g. embedding models, BYOK-only variants). Error message tells contributors exactly what to do.
testHelpers export on copilot-model.ts (following the repo's @internal / ts-prune-ignore-next pattern) to expose the internal Set to the sync test without widening the public API.
Pointer comment in ai-credits-pricing.js directing contributors to SUPPORTED_COPILOT_MODELS and the CI guard.

Implements the three-part fix from issue #6024: 1. Single source of truth comment — SUPPORTED_COPILOT_MODELS in src/copilot-model.ts now documents exactly which related files must be updated when adding or retiring a model. 2. CI guard — new src/copilot-model-catalog-sync.test.ts fails if any model added to ai-credits-pricing.js is missing from SUPPORTED_COPILOT_MODELS and not explicitly excluded. This catches the class of bug fixed in #5831 (mai-code-1-flash) before it ships. 3. Pointer comment in ai-credits-pricing.js — directs contributors to add new Copilot CLI models to SUPPORTED_COPILOT_MODELS as well. testHelpers export added to copilot-model.ts (@internal / ts-prune-ignore-next) to give the sync test access to the internal Set.

Copilot

Pull request overview

Adds a CI-level guard to prevent drift between the api-proxy pricing catalog and the host-side Copilot model allowlist, so valid COPILOT_MODEL values aren’t rejected pre-container with misleading errors.

Changes:

Documented SUPPORTED_COPILOT_MODELS as the “single source of truth” and added guidance on which related catalogs/guards must be updated when adding/retiring models.
Added src/copilot-model-catalog-sync.test.ts to fail CI when containers/api-proxy/ai-credits-pricing.js contains a model missing from SUPPORTED_COPILOT_MODELS (unless explicitly excluded with justification).
Added a sync-pointer comment in ai-credits-pricing.js and exported testHelpers from copilot-model.ts for focused tests.

Show a summary per file

File	Description
src/copilot-model.ts	Adds a “single source of truth” doc comment and exposes `testHelpers` for the sync test.
src/copilot-model-catalog-sync.test.ts	New CI guard test to enforce parity between pricing catalog and supported Copilot CLI models.
containers/api-proxy/ai-credits-pricing.js	Adds an in-file note pointing contributors to the allowlist and CI guard to avoid drift.

Review details

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Files reviewed: 3/3 changed files
Comments generated: 2
Review effort level: Low

+/** Mirror of the separator normalisation in copilot-model.ts. */
+function normalizeSeparators(s: string): string {
+  return s.replace(/[._]/g, '-').toLowerCase();
+}


lpcox · 2026-07-09T03:04:55Z

@copilot address review feedback

- Export testHelpers.supportedCopilotModels as a ReadonlySet defensive copy to prevent accidental mutation of the production singleton in tests - Update normalizeSeparators() comment to accurately describe full canonicalization (lowercase + separator replacement), not just separator normalization

Copilot · 2026-07-09T03:07:52Z

@copilot address review feedback

Done in the latest commit. Both review items addressed:

normalizeSeparators() comment — updated to "Mirrors the full canonicalization in copilot-model.ts: lowercase then replace separators." to accurately describe what the function does.
testHelpers.supportedCopilotModels mutation risk — now exported as new Set(SUPPORTED_COPILOT_MODELS) as ReadonlySet<string>, a defensive copy that prevents tests from mutating the production singleton.

github-actions · 2026-07-09T03:15:05Z

✅ Smoke Copilot BYOK completed. Copilot BYOK mode operational. 🔓

github-actions · 2026-07-09T03:15:06Z

🔑 Smoke Copilot PAT PAT auth validated. All systems operational. ✅

github-actions · 2026-07-09T03:15:06Z

❌ Security Guard failed. Please review the logs for details.

github-actions · 2026-07-09T03:15:06Z

🔌 Smoke Services — All services reachable! ✅

github-actions · 2026-07-09T03:15:07Z

Chroot tests passed! Smoke Chroot - All security and functionality tests succeeded.

github-actions · 2026-07-09T03:15:07Z

✅ Contribution Check completed successfully!

PR #6039 follows the applicable CONTRIBUTING.md guidelines: it includes a clear description with related issue reference, adds a relevant test for the new CI guard, keeps files in appropriate locations, and does not require additional documentation beyond inline guidance for this internal test invariant.

github-actions · 2026-07-09T03:15:07Z

✅ Build Test Suite completed successfully!

github-actions · 2026-07-09T03:15:08Z

❌ Smoke Copilot BYOK AOAI (api-key) reports failed. AOAI BYOK (api-key) mode investigation needed...

github-actions · 2026-07-09T03:15:08Z

✨ The prophecy is fulfilled... Smoke Codex has completed its mystical journey. The stars align. 🌟

github-actions · 2026-07-09T03:15:10Z

✅ Smoke Gemini completed. All facets verified. 💎

Smoke test complete. Results reported to PR #6039. Overall status: FAIL.

github-actions · 2026-07-09T03:15:10Z

📰 VERDICT: Smoke Copilot has concluded. All systems operational. This is a developing story. 🎤

github-actions · 2026-07-09T03:15:14Z

✅ Smoke Copilot BYOK AOAI (Entra) completed. Copilot AOAI BYOK (Entra) mode operational. 🔓

github-actions · 2026-07-09T03:15:23Z

✅ Smoke Claude passed

github-actions · 2026-07-09T03:15:46Z

📡 Smoke OTel Tracing completed. All tracing scenarios validated. ✅

github-actions · 2026-07-09T03:16:29Z

⚠️ Coverage Regression Detected

This PR decreases test coverage. Please add tests to maintain coverage levels.

Overall Coverage

Metric	Base	PR	Delta
Lines	98.98%	98.98%	➡️ +0.00%
Statements	98.94%	98.94%	➡️ +0.00%
Functions	99.44%	99.44%	➡️ +0.00%
Branches	95.64%	95.60%	📉 -0.04%

Coverage comparison generated by scripts/ci/compare-coverage.ts

github-actions · 2026-07-09T03:19:53Z

Smoke Test: Copilot BYOK (Direct) ✅ PASS

Tests:

✅ MCP connectivity
✅ GitHub.com reachable (HTTP 200)
✅ File write/read verified
✅ BYOK inference active (api-proxy → api.githubcopilot.com)

Mode: Direct BYOK via COPILOT_PROVIDER_API_KEY (api-proxy sidecar injection)

All systems operational.

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

awmgmcpg

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "awmgmcpg"

See Network Configuration for more information.

🔑 BYOK report filed by Smoke Copilot BYOK
_{Add label ready-for-aw to run again}

github-actions · 2026-07-09T03:20:19Z

Smoke Test: Claude Engine Validation

Check	Result
API Status	✅ PASS
GH Check	✅ PASS
File Status	✅ PASS

Overall Result: PASS ✅

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

awmgmcpg

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "awmgmcpg"

See Network Configuration for more information.

Generated by Smoke Claude for #6039 · 35 AIC · ⊞ 3.3K · ◷
_{Add label ready-for-aw to run again}

github-actions · 2026-07-09T03:20:39Z

Smoke Test: Copilot PAT Auth ✅ PASS

Test	Result
GitHub MCP connectivity	✅
GitHub.com HTTP	✅
File write/read	✅

Auth mode: PAT (COPILOT_GITHUB_TOKEN)
Author: @lpcox

All tests passed.

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

awmgmcpg

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "awmgmcpg"

See Network Configuration for more information.

🔑 PAT report filed by Smoke Copilot PAT
_{Add label ready-for-aw to run again}

github-actions · 2026-07-09T03:20:48Z

Smoke Test Results

Redis PING: ❌ Network unreachable
PostgreSQL pg_isready: ❌ No response
PostgreSQL SELECT 1: ❌ Network unreachable

Overall: FAIL — host.docker.internal (172.17.0.1) is unreachable from this environment. Service containers are not accessible.

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

awmgmcpg

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "awmgmcpg"

See Network Configuration for more information.

🔌 Service connectivity validated by Smoke Services
_{Add label ready-for-aw to run again}

github-actions · 2026-07-09T03:21:37Z

Chroot Version Comparison Results

Runtime	Host Version	Chroot Version	Match?
Python	Python 3.12.13	Python 3.12.3	❌ NO
Node.js	v24.18.0	v22.23.1	❌ NO
Go	go1.22.12	go1.22.12	✅ YES

Overall: ❌ Not all tests passed — Python and Node.js versions differ between host and chroot environments.

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

awmgmcpg

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "awmgmcpg"

See Network Configuration for more information.

Tested by Smoke Chroot
_{Add label ready-for-aw to run again}

github-actions · 2026-07-09T03:21:58Z

Smoke Test Results

Test	Status
GitHub MCP connectivity	MCP tools unavailable in sandbox
GitHub.com HTTP	HTTP 200 PASS
File write/read	Pre-step template vars not expanded

Overall: PASS (connectivity verified)

@lpcox — smoke test complete.

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

awmgmcpg

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "awmgmcpg"

See Network Configuration for more information.

📰 BREAKING: Report filed by Smoke Copilot
_{Add label ready-for-aw to run again}

github-actions · 2026-07-09T03:22:05Z

Smoke test summary

fix: use tag-only image ref in rootless permission repair ✅
chore: upgrade gh-aw extension to v0.82.5 pre-release ✅
Add CI guard to prevent SUPPORTED_COPILOT_MODELS drift from pricing catalog ✅
Playwright title check ❌
Build ✅
Overall: FAIL

Warning

Firewall blocked 2 domains

The following domains were blocked by the firewall during workflow execution:

awmgmcpg
registry.npmjs.org

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "awmgmcpg"
    - "registry.npmjs.org"

See Network Configuration for more information.

🔮 The oracle has spoken through Smoke Codex
_{Add label ready-for-aw to run again}

github-actions · 2026-07-09T03:22:53Z

🏗️ Build Test Suite Results

Ecosystem	Project	Build/Install	Tests	Status
Bun	elysia	✅	1/1 passed	✅ PASS
Bun	hono	✅	1/1 passed	✅ PASS
C++	fmt	✅	N/A	✅ PASS
C++	json	✅	N/A	✅ PASS
Deno	oak	N/A	1/1 passed	✅ PASS
Deno	std	N/A	1/1 passed	✅ PASS
.NET	hello-world	✅	N/A	✅ PASS
.NET	json-parse	✅	N/A	✅ PASS
Go	color	✅	1/1 passed	✅ PASS
Go	env	✅	1/1 passed	✅ PASS
Go	uuid	✅	1/1 passed	✅ PASS
Java	gson	✅	1/1 passed	✅ PASS
Java	caffeine	✅	1/1 passed	✅ PASS
Node.js	clsx	✅	All passed	✅ PASS
Node.js	execa	✅	All passed	✅ PASS
Node.js	p-limit	✅	All passed	✅ PASS
Rust	fd	✅	1/1 passed	✅ PASS
Rust	zoxide	✅	1/1 passed	✅ PASS

Overall: 8/8 ecosystems passed — ✅ PASS

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

awmgmcpg

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "awmgmcpg"

See Network Configuration for more information.

Generated by Build Test Suite for #6039 · 34.1 AIC · ⊞ 6.9K · ◷
_{Add label ready-for-aw to run again}

github-actions · 2026-07-09T03:23:22Z

Smoke Test: Gemini Engine Validation

GitHub MCP Testing: ❌ (Tools not found/accessible)
GitHub.com Connectivity: ❌ (HTTP 000 - DNS resolution failed)
File Writing Testing: ✅
Bash Tool Testing: ✅

Overall status: FAIL

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

localhost

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "localhost"

See Network Configuration for more information.

💎 Faceted by Smoke Gemini
_{Add label ready-for-aw to run again}

github-actions · 2026-07-09T03:24:54Z

Smoke Test: API Proxy OpenTelemetry Tracing

Scenario	Result	Details
1. Module Loading	✅ Pass	`otel.js` loads cleanly; exports: `startRequestSpan`, `setTokenAttributes`, `setBudgetAttributes`, `endSpan`, `endSpanError`, `shutdown`, `isEnabled`
2. Test Suite	✅ Pass	59 tests passed, 0 failed across `otel.test.js` + `otel-fanout.test.js` (2 suites)
3. Env Var Forwarding	✅ Pass	`api-proxy-env-config.ts` forwards `GH_AW_OTLP_ENDPOINTS`, `OTEL_EXPORTER_OTLP_ENDPOINT`, `OTEL_EXPORTER_OTLP_HEADERS`, `GITHUB_AW_OTEL_TRACE_ID`, `GITHUB_AW_OTEL_PARENT_SPAN_ID` to api-proxy container
4. Token Tracker Integration	✅ Pass	`onUsage` callback present in `token-tracker-http.js` (line 343) as OTEL hook point; `onSpanEnd` also wired at request completion
5. OTEL Diagnostics	✅ Pass	No OTEL endpoint configured → graceful fallback to file export at `/var/log/api-proxy/otel.jsonl`; no errors

All 5 scenarios pass. OTEL tracing integration is functional.

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

awmgmcpg

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "awmgmcpg"

See Network Configuration for more information.

📡 OTel tracing validated by Smoke OTel Tracing
_{Add label ready-for-aw to run again}

github-actions · 2026-07-09T03:28:19Z

Running in direct BYOK mode (AWF_AUTH_TYPE=github-oidc + AWF_AUTH_AZURE_* + COPILOT_PROVIDER_BASE_URL) via api-proxy → Azure OpenAI (Foundry, o4-mini-aw) authenticated via Microsoft Entra
@lpcox
PR Titles only:
${{ steps.smoke-data.outputs.SMOKE_PR_DATA }}
✅ GitHub MCP connectivity
✅ GitHub.com connectivity
✅ File write/read test
✅ BYOK inference test
Overall: PASS

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

awmgmcpg

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "awmgmcpg"

See Network Configuration for more information.

🪪 BYOK (AOAI Entra) report filed by Smoke Copilot BYOK AOAI (Entra)
_{Add label ready-for-aw to run again}

Initial plan

876f288

Copilot AI assigned Copilot and lpcox Jul 9, 2026

Copilot started work on behalf of lpcox July 9, 2026 02:41 View session

Copilot AI linked an issue Jul 9, 2026 that may be closed by this pull request

Prevent model allowlist drift with automated sync between validation and catalog #6024

Closed

Copilot AI changed the title ~~[WIP] Add automated sync for model validation and catalog~~ Add CI guard to prevent SUPPORTED_COPILOT_MODELS drift from pricing catalog Jul 9, 2026

Copilot AI requested a review from lpcox July 9, 2026 02:52

Copilot finished work on behalf of lpcox July 9, 2026 02:52

lpcox marked this pull request as ready for review July 9, 2026 02:56

Copilot AI review requested due to automatic review settings July 9, 2026 02:56

Copilot started reviewing on behalf of lpcox July 9, 2026 02:57 View session

Copilot AI reviewed Jul 9, 2026

View reviewed changes

Comment thread src/copilot-model-catalog-sync.test.ts Outdated

Comment on lines +20 to +23

/** Mirror of the separator normalisation in copilot-model.ts. */

function normalizeSeparators(s: string): string {

return s.replace(/[._]/g, '-').toLowerCase();

}

Comment thread src/copilot-model.ts

Copilot started work on behalf of lpcox July 9, 2026 03:05 View session

Copilot finished work on behalf of lpcox July 9, 2026 03:08

lpcox added the ready-for-aw label Jul 9, 2026

lpcox temporarily deployed to aoai-model July 9, 2026 03:15 — with GitHub Actions Inactive

github-actions Bot added the smoke-copilot-byok label Jul 9, 2026

github-actions Bot added the smoke-claude label Jul 9, 2026

github-actions Bot mentioned this pull request Jul 9, 2026

[aw] No-Op Runs #5883

Open

github-actions Bot added the smoke-copilot-pat label Jul 9, 2026

github-actions Bot added the smoke-copilot label Jul 9, 2026

github-actions Bot added the build-test label Jul 9, 2026

lpcox temporarily deployed to aoai-model July 9, 2026 03:27 — with GitHub Actions Inactive

github-actions Bot added the smoke-copilot-byok-aoai-entra label Jul 9, 2026

lpcox temporarily deployed to aoai-model July 9, 2026 03:28 — with GitHub Actions Inactive

lpcox merged commit 9aa89f2 into main Jul 9, 2026
84 of 88 checks passed

lpcox deleted the copilot/prevent-model-allowlist-drift branch July 9, 2026 03:39

Uh oh!

Conversation

Copilot AI commented Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Review details

Uh oh!

Uh oh!

lpcox commented Jul 9, 2026

Uh oh!

Copilot AI commented Jul 9, 2026

Uh oh!

github-actions Bot commented Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jul 9, 2026

⚠️ Coverage Regression Detected

Overall Coverage

Uh oh!

github-actions Bot commented Jul 9, 2026

Smoke Test: Copilot BYOK (Direct) ✅ PASS

Uh oh!

github-actions Bot commented Jul 9, 2026

Smoke Test: Claude Engine Validation

Uh oh!

github-actions Bot commented Jul 9, 2026

Smoke Test: Copilot PAT Auth ✅ PASS

Uh oh!

github-actions Bot commented Jul 9, 2026

Smoke Test Results

Uh oh!

github-actions Bot commented Jul 9, 2026

Chroot Version Comparison Results

Uh oh!

github-actions Bot commented Jul 9, 2026

Smoke Test Results

Uh oh!

Copilot AI commented Jul 9, 2026 •

edited

Loading

github-actions Bot commented Jul 9, 2026 •

edited

Loading

github-actions Bot commented Jul 9, 2026 •

edited

Loading

github-actions Bot commented Jul 9, 2026 •

edited

Loading

github-actions Bot commented Jul 9, 2026 •

edited

Loading

github-actions Bot commented Jul 9, 2026 •

edited

Loading

github-actions Bot commented Jul 9, 2026 •

edited

Loading

github-actions Bot commented Jul 9, 2026 •

edited

Loading

github-actions Bot commented Jul 9, 2026 •

edited

Loading

github-actions Bot commented Jul 9, 2026 •

edited

Loading

github-actions Bot commented Jul 9, 2026 •

edited

Loading

github-actions Bot commented Jul 9, 2026 •

edited

Loading

github-actions Bot commented Jul 9, 2026 •

edited

Loading

github-actions Bot commented Jul 9, 2026 •

edited

Loading

github-actions Bot commented Jul 9, 2026 •

edited

Loading