Skip to content

fix(ci): guard eval schema generation#1725

Merged
christso merged 1 commit into
mainfrom
av-3uni-schema-guard
Jul 7, 2026
Merged

fix(ci): guard eval schema generation#1725
christso merged 1 commit into
mainfrom
av-3uni-schema-guard

Conversation

@christso

@christso christso commented Jul 7, 2026

Copy link
Copy Markdown
Collaborator

Summary

Eval schema drift is now a focused CI failure instead of a late surprise in the broad test job. bun run validate:eval-schema regenerates skills-data/agentv-eval-writer/references/eval.schema.json from the Zod source, fails when that run changes the artifact, and tells workers to commit the generated diff instead of editing JSON by hand.

The root package also has a direct regeneration alias, and the repo verification guidance plus schema source comments now point eval-schema authors at the root guard command.

Validation

  • bun install --frozen-lockfile
  • bun run validate:eval-schema
  • bun test packages/core/test/evaluation/validation/eval-schema-sync.test.ts
  • bun x biome check scripts/check-eval-schema.ts package.json .github/workflows/ci.yml .agents/verification.md packages/core/src/evaluation/validation/eval-file.schema.ts packages/core/scripts/generate-eval-schema.ts
  • bun run generate:eval-schema
  • git diff --check
  • Manual review fallback found no actionable issues. The full ce-code-review subagent flow was not run because this tool surface only permits subagents when the user explicitly asks for delegation.

Post-Deploy Monitoring & Validation

No additional operational monitoring required: this change only affects CI and local developer commands.

Healthy signal: PR CI reaches the Check generated eval schema sync step and passes before the broader test suite. Failure signal: that step emits the schema drift error and diff. Mitigation: run bun run validate:eval-schema, review the generated schema diff, and commit it with the Zod schema change. Validation window and owner: the next PR that touches packages/core/src/evaluation/validation/eval-file.schema.ts, owned by that PR's author and reviewer.


Compound Engineering
GPT-5

@cloudflare-workers-and-pages

Copy link
Copy Markdown

Deploying agentv with  Cloudflare Pages  Cloudflare Pages

Latest commit: 9167915
Status: ✅  Deploy successful!
Preview URL: https://c2f4bbe0.agentv.pages.dev
Branch Preview URL: https://av-3uni-schema-guard.agentv.pages.dev

View logs

@christso christso marked this pull request as ready for review July 7, 2026 12:29
@christso christso merged commit e6f126c into main Jul 7, 2026
11 checks passed
@christso christso deleted the av-3uni-schema-guard branch July 7, 2026 12:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant