ci: add Delimit governance check for schema changes

[infracore]

Per the discussion in #21, here's a workflow that runs Delimit on every PR touching the manifest schema. It performs two checks and posts a single PR comment summarizing both.

What it does

1. Generator drift — runs pnpm run schema:export in a sandbox and diffs the regenerated schemas/v1/agent.schema.json against the committed file. Catches the case where the Zod source has been updated but the committed JSON Schema artifact is stale, or vice versa. The committed file is restored before the workflow exits, so the working tree is unmodified.

2. Schema classification — diffs the committed JSON Schema against the base branch and classifies each change as breaking or non-breaking using JSON Schema semantics: property add/remove, required add/remove, type widen/narrow, enum value add/remove, const change, additionalProperties flip, pattern, minLength/maxLength, minimum/maximum, and items type. $ref to #/definitions is resolved.

Advisory mode by default — never fails CI. No API keys, no external services. The action is published at delimit-ai/delimit-action.

Sample output (from a real run on infracore/agentspec)

I ran this workflow on a fork before opening this PR to verify it works end-to-end. Here is the actual rendered PR comment from that run:

### Delimit governance — JSON Schema

**Generator drift:** 2 change(s), 1 breaking

#### Generator drift

Artifact `schemas/v1/agent.schema.json` is stale vs `pnpm run schema:export` output (2 drift change(s), 0.728s to regen).

Re-run the generator and commit the result, or revert the source change.

- 🔴 const_changed at `/properties/apiVersion` — const value changed: 'agentspec.io/v1' → 'agentspec.io/v1alpha1'
- 🟢 enum_value_added at `/properties/spec/properties/compliance/properties/packs/items` — enum value added: 'metadata-quality'

> Advisory mode — CI will not fail. Set `fail_on_breaking: true` or `mode: enforce` to block on breaking changes.

The first change is the v1 → v1alpha1 rename I simulated by editing the Zod source on a test branch.

The second change is real and was already on main when I ran the test. Someone added metadata-quality to the compliance packs enum in the Zod source but the committed schemas/v1/agent.schema.json was not regenerated. The workflow caught it on the first run. Worth a separate cleanup commit.

Notes

• SHA pin. This pins to a specific commit of delimit-action because it relies on JSON Schema support that lands in the upcoming v1.9.0 release. Once v1.9.0 ships I will follow up to switch the pin to delimit-ai/delimit-action@v1.
• Pre-1.0 breaking changes are expected on agentspec; the action stays advisory by default and won't fail your CI. Set fail_on_breaking: true later if you want to enforce.
• Runtime is dominated by pnpm run schema:export (~0.7s in the test run, plus pnpm install). The Delimit diff itself is sub-100ms. Total workflow runtime under 30 seconds on the test fork.
• Permissions are scoped to contents: read and pull-requests: write for posting the comment.

Closes the spirit of the discussion in #21. Happy to iterate on the trigger paths, broader artifact set, or stricter mode if you want a different shape.

[infracore]

Quick follow-up: v1.9.0 of delimit-action just shipped with the JSON Schema support, so I pushed a commit to this branch switching the pin from the SHA to delimit-ai/delimit-action@v1. No other changes.

[skokaina]

thanks @infracore - looked in the meantime to the github repo documentation https://github.com/delimit-ai/delimit-action and couldn't see anything about generator_command and generator_artifact fields, what are they about ?

[infracore]

Thanks for flagging that. Both inputs shipped in v1.9.0 but haven't landed in the README yet. I'll get that fixed this week and link back here.

Here's what they do.

generator_command is an optional shell command that regenerates a committed artifact. For agentspec the relevant one is pnpm run schema:export, which rebuilds schemas/v1/agent.schema.json from the Zod source under packages/sdk/src/schema/.

generator_artifact is the path to the file that command produces, for example schemas/v1/agent.schema.json.

When both are set, the action runs generator_command inside the CI sandbox, diffs the regenerated output against the committed generator_artifact, and flags drift. It catches the case where someone edits the Zod source but forgets to re-run the exporter, so the committed JSON Schema silently lags behind the source of truth.

Both checks in this workflow (schema classification vs base branch, and generator drift) are independent. My preference is to keep both since the workflow already has them wired up, but if you'd rather keep scope minimal for the initial merge I can drop the generator_* pair and send drift detection as a follow-up PR. The schema classification check on its own still delivers the breaking-change guard from #21.

[infracore]

Quick follow-up: the README is updated. Both inputs are now in the Inputs table, and there's a new Generator drift detection section with a minimal example and notes on supported generators (Zod → zodToJsonSchema, protobuf, TypeBox).

Shipped in delimit-ai/delimit-action#4. Let me know if anything still looks unclear.

[skokaina]

Thank you for the update, i have activated the PR build, it failed on the delim governance check

[infracore]

Good news first: the check actually caught a real drift. When it ran pnpm run schema:export it found that schemas/v1/agent.schema.json is stale compared to what the generator produces. The specific change is an added metadata-quality enum value at /properties/spec/properties/compliance/properties/packs/items. That's classified as 🟢 non-breaking (enum additions don't break consumers), and it's advisory, so the intent was "heads up, please regen" rather than "block the PR".

The actual FAILURE is that the step that posts the PR comment got 403 Resource not accessible by integration trying to hit /issues/39/comments. That's a permissions issue on the workflow token. You need to add this to .github/workflows/delimit.yml under the delimit job:

permissions:
  contents: read
  pull-requests: write
  issues: write

Without that, GITHUB_TOKEN defaults to read-only for comments and the bot can't post. On our side, we also have a bug: the action shouldn't fail the whole check when the comment step throws, as long as the analysis verdict is advisory. I'm fixing that this week.

Two things you can do right now to resolve the check:

1. Run pnpm run schema:export and commit the updated schemas/v1/agent.schema.json — that clears the drift detection
2. Add the permissions block above — that lets Delimit post its comment on future runs

Let me know if you want me to open a PR against your workflow file with the permissions fix, happy to do it.