Skip to content

ci: build changed registry definitions on the pull request - #134

Merged
moshest merged 3 commits into
mainfrom
claude/validate-registry-definitions
Sep 6, 2026
Merged

ci: build changed registry definitions on the pull request#134
moshest merged 3 commits into
mainfrom
claude/validate-registry-definitions

Conversation

@moshest

@moshest moshest commented Sep 6, 2026

Copy link
Copy Markdown
Member

Why

ci.yml lints, builds and tests the TypeScript packages. Nothing reads registry/*.yaml.

So a definition naming a branch or docs_path that doesn't exist passes review and CI unchallenged, and first fails at 06:00 UTC in the nightly registry-update — after the PR that introduced it is merged and out of mind. Same shape as the sanity breakage that hid for four nights: the check that would have caught it ran nowhere near the change.

#133 made it concrete. Two new definitions arrived with zero check runs, so verifying them meant cloning the upstream repo and building both by hand.

What it does

On a PR touching registry/**/*.yaml, builds each definition the PR adds or modifies via the existing test-registry test <name> CLI. Only changed definitions, so cost tracks the diff rather than the 113-definition registry. No new tooling — it runs the command a maintainer would run locally.

Verification

This PR changes no registry files, so it cannot exercise its own workflow. Rather than merge on local component tests alone, I pushed a deliberate probe (commit 75ea47d) adding a definition pointing at a nonexistent branch, then removed it (856434f). Both commits are kept rather than squashed so the evidence stays in the history.

The probe run failed exactly as required, and for the right reason rather than incidentally:

Changed definition files:
  registry/ci-probe-delete-me/ci-probe-delete-me.yaml

Error: Git clone failed: fatal: Remote branch
  this-branch-does-not-exist-xyz not found in upstream origin
##[error]definition 'ci-probe-delete-me' failed to build
Failed to build: ci-probe-delete-me
##[error]Process completed with exit code 1

That confirms the whole chain end to end: the paths: filter triggered, diff detection found the file, the name resolved, the build failed on the bad ref specifically, the error annotation fired, and the job exited non-zero.

Checked before that, locally:

  • Nonexistent docs_path → exits 1, rather than "succeeding" with an empty package. The failure mode worth ruling out, since a silent empty build would sail through a naive check
  • Name extraction handles a name differing from its filename (registry/hex/phoenix.yamlphoenix), quoted names, and a missing name: field
  • Diff detection returns exactly the two files from Add gdscript and godot package definitions #133 and excludes deletions
  • Both Add gdscript and godot package definitions #133 definitions build: gdscript 129 sections / 60,921 tokens, godot 14,488 sections / 8,136,608 tokens
  • Lint, Build, Test green on the cleaned head

permissions: {} at top level with contents: read on the job; no secrets, so it works on fork PRs.

Note on cost

timeout-minutes: 45. A definition can pull a large documentation repo — godot builds ~8M tokens from 3.5k files — so the ceiling is deliberately generous but bounded. If large-repo definitions become common, the thing worth deciding is a size budget, not the timeout; there's no documented ceiling in the registry today.

🤖 Generated with Claude Code

https://claude.ai/code/session_01NBQQpA86yYzwJUiVz8ph2R

ci.yml lints, builds and tests the TypeScript packages; nothing there reads
registry/*.yaml. A definition naming a branch or docs_path that does not exist
passes review and CI unchallenged, then fails at 06:00 UTC in the nightly
registry-update — after the PR that introduced it is merged and out of mind.
That is the same shape as the sanity breakage that hid for four nights: the
check that would have caught it ran nowhere near the change.

#133 made it concrete. Two new definitions arrived with zero check runs, so
verifying them meant building both by hand.

This builds the definitions a PR adds or modifies, on the PR. Only changed
ones, so cost tracks the diff rather than the 113-definition registry.

Verified before committing, since a validator that passes broken input is
worse than none:

- A nonexistent git ref exits 1.
- A nonexistent docs_path exits 1 rather than producing an empty package,
  which was the failure mode worth ruling out.
- Name extraction handles a name that differs from its filename
  (registry/hex/phoenix.yaml is "phoenix"), quoted names, and a missing
  name: field.
- The diff detection returns exactly the two definitions from #133 and
  excludes deletions.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NBQQpA86yYzwJUiVz8ph2R
@changeset-bot

changeset-bot Bot commented Sep 6, 2026

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: 856434f

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

Adds a definition naming a branch that does not exist. The new workflow must
trigger on it and go red. Reverted in the next commit — this exists only to
prove the wiring, which #134 cannot test on itself since it changes no
registry files.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NBQQpA86yYzwJUiVz8ph2R
The probe did its job. On the previous commit the workflow triggered, resolved
the definition name, and failed with:

  Git clone failed: fatal: Remote branch this-branch-does-not-exist-xyz
  not found in upstream origin
  ##[error]definition 'ci-probe-delete-me' failed to build
  Process completed with exit code 1

That is the failure surfacing for the right reason rather than incidentally,
which is the only version of this check worth having. The two commits are kept
rather than squashed away so the evidence stays in the branch history.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NBQQpA86yYzwJUiVz8ph2R
@moshest
moshest marked this pull request as ready for review September 6, 2026 12:15
@moshest
moshest merged commit afdd8e6 into main Sep 6, 2026
3 checks passed
@moshest
moshest deleted the claude/validate-registry-definitions branch September 6, 2026 12:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants