Feat/rulesets check

[AlexAxthelm]

Centralize ruleset-drift checks in a composite action

Moves the ruleset-drift check that was duplicated across stitch,
stitch-etl-poc, and tpr (.github/workflows/admin-check_rulesets.yml) into a
single, centrally-maintained composite action, so the logic lives in one place
and consumer repos keep only a tiny per-ruleset overlay instead of hand-authoring
full ruleset JSON.

How it works

Three steps (actions/admin/rulesets-check/):

1. fetch (fetch_remote.sh) — list the repo's live rulesets via gh api
   (paginated) and dump each raw.
2. resolve (resolve_local.py) — for each .github/rulesets/*.overlay.json,
   deep-merge the overlay onto its named bundled template into a canonical ruleset.
3. diff (diff_rulesets.py) — compare resolved-local against live.

Allow-list model: the resolved template + overlay is the allow-list — only
keys you actually define are checked. A tracked key whose value differs (or is
absent) on the live ruleset fails; keys GitHub returns that you don't define
(id, source, timestamps, brand-new schema fields) are ignored. No strip list
and no ignore list to maintain.

Merge semantics: objects recurse; rules are addressed by .type (overlay writes
them as a {type: patch} map, one rule per type enforced); all other arrays are
replaced wholesale. Arrays are compared order-insensitively (every ruleset
array is semantically a set).

What's included

• Composite action actions/admin/rulesets-check/ with scripts, four bundled
  gitflow-* templates + a blank base, and an overlay.schema.json for editor
  tooling. Consumers call it directly from their own workflow (same pattern as
  forbidden-patterns).
• Nightly schema-coverage job .github/workflows/admin-rulesets-schema-check.yml
  (+ schema_coverage.py) — compares GitHub's published repository-ruleset
  schema against what the templates cover, scoped to the rule types we manage,
  and fails the scheduled run on drift (a new parameter on a tracked rule type
  we neither track nor listed in schema/acknowledged-untracked.json, or a tracked
  key the schema no longer lists). Adopting an entirely new GitHub rule type is a
  product choice, not drift, so it's not flagged. This is where upstream schema
  drift surfaces — centrally, once — instead of on every consumer PR.
• Diagnostics — each drift failure reports the dotted path + local/remote
  values (deterministically serialized) in the log, an annotation, and the job
  summary; failing rulesets' full JSON is dumped in a collapsed group
  (RUNNER_DEBUG=1 dumps all).

Consumer usage

jobs:
  rulesets:
    runs-on: ubuntu-latest
    permissions:
      contents: read
    steps:
      - uses: actions/checkout@v7
      - uses: RMI/actions/actions/admin/rulesets-check@main

Each ruleset is one *.overlay.json naming its template plus only the repo's
deltas (e.g. its own required-status-check contexts).

Robustness

Guards for the failure modes that matter in CI: a failed gh api list aborts
(instead of looking like "no rulesets"); ruleset name collisions fail loudly
instead of silently overwriting; overlay template values are rejected if they
look like a path (no traversal); array comparison is order-insensitive so a
hand-authored overlay doesn't false-fail on a reorder; and the nightly schema
parser composes allOf and fails clearly if it can't understand the schema shape,
rather than reporting false drift.

Testing

Validated end-to-end against stitch (PR RMI/stitch#250): all four overlays
resolve, and the diff correctly detects real drift on gitflow-main. Copilot
review comments addressed. Scripts have local unit/integration coverage for the
merge, diff, collision, path-traversal, and schema-coverage paths.

🤖 Generated with Claude Code

[Copilot]

Pull request overview

Adds a new rulesets-check composite action and supporting tooling to compare a repo’s live GitHub rulesets against repo-managed templates/overlays, plus a nightly workflow to detect when GitHub’s published ruleset schema introduces new (previously untracked) fields.

Changes:

• Introduces a composite action (actions/admin/rulesets-check) with scripts to fetch remote rulesets, resolve local overlays against bundled templates, and diff tracked keys.
• Adds bundled gitflow-oriented ruleset templates plus overlay schema/acknowledgement metadata.
• Adds a nightly workflow to compare GitHub’s OpenAPI ruleset schema against template coverage and open/update a tracking issue on gaps.

Reviewed changes

Copilot reviewed 15 out of 15 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
actions/admin/rulesets-check/templates/gitflow-production.json	Adds a production branch ruleset template (2 approvals + status-check rule placeholders).
actions/admin/rulesets-check/templates/gitflow-next-pr.json	Adds a PR-focused ruleset template for the next branch.
actions/admin/rulesets-check/templates/gitflow-next-lifecycle.json	Adds lifecycle protections (creation/deletion) for next, hotfix/**, demo/**.
actions/admin/rulesets-check/templates/gitflow-main.json	Adds a main/default-branch ruleset template including PR + status checks + copilot review rule.
actions/admin/rulesets-check/templates/blank.json	Adds a minimal base template for fully custom overlays.
actions/admin/rulesets-check/scripts/schema_coverage.py	Nightly schema coverage reporter comparing templates vs GitHub OpenAPI schema.
actions/admin/rulesets-check/scripts/resolve_local.py	Resolves per-repo overlays onto bundled templates via deep-merge + canonicalization.
actions/admin/rulesets-check/scripts/fetch_remote.sh	Fetches live rulesets via gh api and writes per-ruleset JSON files.
actions/admin/rulesets-check/scripts/diff_rulesets.py	Local-driven allow-list diff between resolved local rulesets and remote rulesets.
actions/admin/rulesets-check/schema/overlay.schema.json	Provides editor/reference JSON schema for overlay files.
actions/admin/rulesets-check/schema/acknowledged-untracked.json	Lists schema paths intentionally not tracked (used by nightly coverage).
actions/admin/rulesets-check/README.md	Documents usage, templates, overlay format, merge semantics, and nightly coverage.
actions/admin/rulesets-check/action.yml	Defines the composite action inputs/steps wiring fetch → resolve → diff.
.github/workflows/admin/rulesets-schema-check.yml	Nightly workflow to detect schema drift and open/update a tracking issue.
.github/workflows/admin/check-rulesets.yml	Reusable workflow wrapper around the composite action for consumers.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Pull request overview

Copilot reviewed 15 out of 15 changed files in this pull request and generated 1 comment.

Suppressed comments (1)

actions/admin/rulesets-check/README.md:14

• README claims the action emits a "job-summary table", but the implementation writes headings, code blocks, and bullet lists to $GITHUB_STEP_SUMMARY (no table formatting). Either update the README wording or change the summary rendering to a table to match.

  Genuinely new schema properties are caught centrally by the [nightly coverage
  check](#nightly-schema-coverage-check), not by nagging every consumer PR.
- Emits GitHub annotations and a job-summary table.

[Copilot]

Pull request overview

Copilot reviewed 15 out of 15 changed files in this pull request and generated 2 comments.

Suppressed comments (4)

Previously missed (4) — in code that hasn't changed since the last review.

actions/admin/rulesets-check/README.md:20

• The action requires gh and jq (see scripts/fetch_remote.sh), but the README doesn’t mention these runner prerequisites. This can confuse users running on self-hosted runners or minimal containers.

## Usage

### Direct (composite action)

actions/admin/rulesets-check/scripts/resolve_local.py:7

• This module documentation references PLAN.md §5, but there is no PLAN.md anywhere in the repository, so this is a broken doc reference for maintainers. Consider updating the reference to a stable in-repo doc (e.g. the README “Merge semantics” section) or adding the missing document.

For each ``<name>.overlay.json`` in the rulesets dir, load its template
(bundled with the action), deep-merge the overlay on top per the rules in
PLAN.md §5, and write the resolved, canonical ruleset (rules as a list) to the
output dir. Downstream, diff_rulesets.py compares these against the live remote

actions/admin/rulesets-check/scripts/diff_rulesets.py:6

• This module documentation references PLAN.md §6.3, but there is no PLAN.md anywhere in the repository, so this is a broken doc reference for maintainers. Consider updating the reference to a stable in-repo doc (e.g. the action README) or adding the missing document.

Replaces the old ``diff --recursive`` step (PLAN.md §6.3). Uses an **allow-list**
model: the resolved template+overlay *is* the allow-list — only keys we actually
define are checked. This makes the per-PR check quiet and precise, and moves the

actions/admin/rulesets-check/schema/overlay.schema.json:6

• The overlay schema description references PLAN.md §5, but PLAN.md doesn’t exist anywhere in this repo, so this is a broken documentation reference for users relying on schema metadata in editors.

  "$id": "https://github.com/RMI/actions/actions/admin/rulesets-check/schema/overlay.schema.json",
  "title": "rulesets-check overlay",
  "description": "A per-repo ruleset overlay: names a bundled template and layers sparse overrides on top. Use the 'blank' template to author a fully custom ruleset entirely from the overlay. Merge semantics are documented in the action README and PLAN.md §5. NOTE: resolve_local.py enforces the key constraints inline rather than validating against this file; it ships as reference/editor tooling.",
  "type": "object",

[Copilot]

Pull request overview

Copilot reviewed 15 out of 15 changed files in this pull request and generated 3 comments.

Suppressed comments (2)

actions/admin/rulesets-check/README.md:165

• This line says the job summary shows drift detail “in a table”, but the current implementation writes fenced code blocks via write_summary() in diff_rulesets.py. Updating the wording will keep docs aligned with actual output.

- the **step log** — a full block per failure (`local = …` / `remote = …`);
- a **check annotation** — a one-line, value-bearing summary (e.g.
  `rules.pull_request.parameters.require_last_push_approval differs — local=true remote=false`);
- the **job summary** — the same detail in a table.

actions/admin/rulesets-check/scripts/diff_rulesets.py:163

• failure_full() renders local/remote values with json.dumps(...) without sort_keys=True, which can make log/summary output unstable for dict-shaped values. Using sort_keys=True improves readability and makes repeated runs easier to compare.

    if f["kind"] == "mismatch":
        return (
            f"Ruleset '{f['name']}': value mismatch at '{f['path']}'\n"
            f"    local  = {json.dumps(f['local'])}\n"
            f"    remote = {json.dumps(f['remote'])}"
        )
    if f["kind"] == "missing":
        return (
            f"Ruleset '{f['name']}': tracked key '{f['path']}' is missing from the "
            f"live ruleset — the remote does not enforce a setting we require\n"
            f"    local  = {json.dumps(f['local'])}\n"
            f"    remote = (absent)"
        )

[Copilot]

Pull request overview

Copilot reviewed 14 out of 14 changed files in this pull request and generated no new comments.

Suppressed comments (2)

Previously missed (2) — in code that hasn't changed since the last review.

actions/admin/rulesets-check/schema/overlay.schema.json:5

• overlay.schema.json description references PLAN.md §5, but there is no PLAN.md in this repository. This is a broken documentation pointer for consumers using the schema for editor tooling; consider pointing to the README section that documents merge semantics instead (or remove the reference).

  "description": "A per-repo ruleset overlay: names a bundled template and layers sparse overrides on top. Use the 'blank' template to author a fully custom ruleset entirely from the overlay. Merge semantics are documented in the action README and PLAN.md §5. NOTE: resolve_local.py enforces the key constraints inline rather than validating against this file; it ships as reference/editor tooling.",

actions/admin/rulesets-check/scripts/schema_coverage.py:40

• schema_coverage.py's load_json() will raise an uncaught exception (traceback) on missing/invalid JSON, unlike the other scripts in this action which emit a GitHub-style ::error::... message and exit cleanly. Since this runs in a scheduled workflow and consumes a downloaded OpenAPI file, failing with a clear error message would make operational triage much easier.

def load_json(p: Path):
    return json.loads(Path(p).read_text())

[jdhoffa]

lgtm!