Add agent compatibility matrix and validation

[salmanmkc]

Add .github/aw/compat.json — the matrix that pins agentic CLI versions to gh-aw release ranges. The gh-aw-actions/setup action will eventually consume this matrix to install a known-good agent version per gh-aw release; until that lands, the matrix is documentation.

This file lives in gh-aw because gh-aw is the source of truth — the existing sync-actions workflow on github/gh-aw-actions mirrors files from gh-aw as part of the release process. A companion PR on gh-aw-actions extends sync-actions to mirror the new compat files alongside the existing setup-action sources, so the runtime fetch URL (raw.githubusercontent.com/github/gh-aw-actions/main/.github/aw/compat.json) keeps working.

Files

File	Purpose
.github/aw/compat.json	The matrix data
.github/aw/compat.schema.json	JSON Schema (draft-07) covering shape and types
.github/workflows/cgo.yml	New Validate compat.json structure and version formats step in the validate-yaml job

The shape of the changes mirrors the existing .github/aw/releases.json + .github/aw/releases.schema.json + inline actions/github-script validator already in cgo.yml. No new dependencies, no new files outside .github/aw/.

Initial v1 contents

Split-lifecycle shape with one catch-all and one bounded row:

Row	gh-aw range	min-agent	max-agent	open
catch-all	0.72.0 .. *	1.0.21	1.0.48	true
bounded	0.0.1 .. 0.71.99	0.0.0	1.0.21	(closed-by-construction)

How the catch-all range was determined

A 12-version sweep ran the canary's working scenarios against each of the following copilot CLI versions on gh-aw v0.72.1:

1.0.21, 1.0.22, 1.0.23, 1.0.25, 1.0.27, 1.0.30, 1.0.33, 1.0.36, 1.0.40, 1.0.43, 1.0.46, 1.0.48

Six canary scenarios were used as the bump-bar (the four safe-outputs.create-pull-request scenarios are currently broken for unrelated reasons and were excluded):

• code-search — github MCP read path (search files / content)
• create-issue — safe-outputs.create-issue write path
• detection-not-skipped — explicit regression test for the v1.0.22 silent-success fingerprint
• list-prs — github MCP read path (list PRs and issues)
• strict-mode — strict-mode end to end
• tool-connectivity — MCP tool surface non-empty

All twelve versions passed all six scenarios on the agent / detection / safe_outputs jobs. (Some runs showed cancelled on the conclusion job due to a global gh-aw-conclusion-<scenario> concurrency group stomping across the sweep branches; the cancellation does not reflect a CLI failure.)

Bounded row

max-agent: 1.0.21 for gh-aw 0.0.1 .. 0.71.99 is the last copilot CLI version known to work with pre-0.72 gh-aw releases before the v1.0.22 detection-skip incident (#25550). That specific failure mode no longer reproduces against current gh-aw because several MCP servers (safeoutputs, mcpscripts) moved out of the gateway and into agent-container CLI tools, narrowing the surface that the v1.0.22 bug could affect.

Validation

The new step in cgo.yml runs on every PR that touches .github/aw/compat.json or .github/aw/compat.schema.json. Schema covers shape & types; the inline JS covers:

• min-agent <= max-agent per row
• min-gh-aw <= max-gh-aw per row (or max-gh-aw == "*")
• gh-aw ranges per agent are non-overlapping
• open field permitted only on the catch-all (max-gh-aw == "*") row; bounded rows are closed-by-construction
• All required fields present, no unknown fields, valid semver formats

Validator was exercised locally against the happy path and against six failure cases (min>max, open on bounded row, overlapping rows, unknown row property, unknown agent, missing required field) — each case was caught with a clear error message.

What this does not do

• It is not yet consumed by the gh-aw-actions/setup action — until that ships, the matrix is documentation. Production gh-aw still installs whatever default it ships.
• There is no automated bump workflow yet — max-agent moves through human PRs as part of the weekly release.
• There is no automated rollback — incident response is still manual.

These are tracked as follow-ups.

[Copilot]

Pull request overview

Adds a versioned compatibility matrix that pins agentic CLI versions to gh-aw release ranges, plus a JSON Schema and a CI validation step. The matrix is documentation-only until the gh-aw-actions/setup action consumes it.

Changes:

• New .github/aw/compat.json with v1 catch-all + bounded rows for the copilot agent.
• New .github/aw/compat.schema.json (draft-07) covering shape, types, and semver formats.
• New inline actions/github-script validation step in cgo.yml enforcing semantic rules (range ordering, non-overlap, open only on catch-all, no unknown fields) on every PR touching the matrix files.

Show a summary per file

File	Description
.github/aw/compat.json	Initial v1 matrix data (catch-all + bounded copilot rows).
.github/aw/compat.schema.json	JSON Schema describing the matrix shape and semver formats.
.github/workflows/cgo.yml	Adds compat.json/compat.schema.json to path filters and adds a new validation step mirroring the existing releases.json validator.

Copilot's findings

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comments suppressed due to low confidence (1)

.github/aw/compat.schema.json:61

• The JSON schema permits open on every row (including bounded rows), but the inline validator at lines 808–810 rejects open on any row whose max-gh-aw is not "*". This means a compat.json that passes the schema can still fail the inline validator. Either tighten the schema (e.g., split into two row variants via oneOf/if-then so that open is only permitted when max-gh-aw == "*"), or relax the validator, so the two stay in sync. Otherwise downstream consumers that only run the schema (e.g., the setup action, IDE tooling) will accept files that CI later rejects.

    "row": {
      "type": "object",
      "required": ["min-gh-aw", "max-gh-aw", "min-agent", "max-agent"],
      "additionalProperties": false,
      "properties": {
        "min-gh-aw": { "$ref": "#/definitions/semver" },
        "max-gh-aw": { "$ref": "#/definitions/ghAwBound" },
        "min-agent": { "$ref": "#/definitions/semver" },
        "max-agent": { "$ref": "#/definitions/semver" },
        "open": {
          "type": "boolean",
          "description": "When false, the setup action stops bumping max-agent for this row (rollback / quarantine state)."
        }
      }
    }

• Files reviewed: 3/3 changed files
• Comments generated: 4

[salmanmkc]

CI status (2026-05-15 14:08Z):

The new Validate compat.json structure and version formats step in cgo.yml's validate-yaml job ran and passed ✅.

The CGO workflow itself shows red because of an unrelated Lint Gate failure on actions/setup/js/.../create_pull_request.test.cjs and git_helpers.test.cjs (Prettier format violations). This same failure is present on the most recent main commits (25922105674, 25921693911, 25921405340, 25921168031, 25920800626) and reproduces on the base branch — this PR doesn't touch either of those files. Leaving the pre-existing lint-gate failure alone here per scope. Once that's fixed on main, this PR's CGO run should go green on the next push or rebase.

[github-actions]

Skills-Based Review 🧠

Applied /grill-with-docs — this PR introduces new domain concepts (agent-compat-v1, open, cache-ttl-days) with new terminology, a JSON schema, and inline validation logic that all need to agree.

Key Themes

1. open field semantics are ambiguous — the field is described as "when false, stop bumping", but the implicit default when the field is omitted is never documented in the schema or the validator. A "default": true in the schema would make this self-describing.

2. $id points to the consumer, not the source — compat.schema.json uses gh-aw-actions as the schema's canonical $id while living in gh-aw. Schema tooling that resolves $id will get a 404 until the companion PR merges.

3. Two sources of truth for allowed agent names — allowedAgentKeys in the validator duplicates the agent-compat-v1.properties list in the schema. A linking comment at minimum, or reading the schema's property names at runtime, would prevent them drifting when a new engine is added.

4. cmp() is not null-safe — if parseSemver ever returns null (guarded in practice, but not defensively), the function throws a TypeError instead of producing a clean validation error.

5. Log message implies ajv schema validation ran when it didn't — the success message on line 866 says the schema is "the schema contract", but only hand-rolled checks execute. The existing releases.json step has the same pattern; small wording fix makes it accurate.

6. Inclusive overlap bounds are undocumented — the <= comparison means two rows sharing a boundary version (e.g., both claim 0.72.0) are flagged as overlapping. This is correct and intentional, but a one-line comment saves future range authors a head-scratch.

Positive Highlights

• ✅ Thorough validator test coverage — the PR description lists six failure cases exercised locally (min>max, open on bounded row, overlapping rows, unknown property, unknown agent, missing required field). That's a solid test surface for an inline script.
• ✅ Clean split-lifecycle shape — the catch-all / bounded-row design is well-explained in the PR and maps directly to the data. The 0.71.99 boundary is unambiguous.
• ✅ Mirrors the existing releases.json pattern — reusing the same CI job, the same github-script action pin, and the same file layout keeps the validate-yaml job coherent and easy to navigate.
• ✅ additionalProperties: false throughout the schema — tight schema prevents silent property typos, which is the right default for a pinned-version matrix.

Verdict

No blocking issues — the data, schema, and validator are functionally correct. The comments above are mostly about documentation precision and defensive coding. Happy to see this land as-is with the inline comments addressed in a follow-up if preferred.

🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer · ● 8.3M

[github-actions]

✅ smoke-ci: safeoutputs CLI comment + comment-memory run (25924408935)

Generated by 🧪 Smoke CI for issue #32396 · ◷

[github-actions]

Please resolve the open review threads and post a brief blocker summary if anything remains.

Generated by 👨‍🍳 PR Sous Chef · ◷

[github-actions]

@copilot review all comments — please address the unresolved review feedback on this PR.

Generated by 👨‍🍳 PR Sous Chef · ◷