feat(verify): extract dogfood engine into public @objectstack/verify + CLI

[os-zhuang]

Why

ObjectStack is a platform where third parties have an AI author arbitrary metadata. The real risk isn't "a platform change broke the CRM example" — it's "the AI used a valid primitive the examples don't exercise, and it silently breaks at runtime" (the #2018 timezone-bucketing class: green on build / 900+ unit tests / spec-liveness / CodeQL, only visible when the real engine + services + HTTP run together).

The internal @objectstack/dogfood package already contained an app-agnostic objectstack verify engine in embryo (its derive/verify comments literally say "point it at any app … a third-party app like hotcrm"). This PR productizes that engine so third-party and template authors can run the same runtime proofs against their own apps.

What changed

New public library @objectstack/verify (packages/verify)

• bootStack (was bootDogfoodStack), deriveCrudCases, runCrudVerification, runRlsProofs + report types — moved & generalized out of packages/dogfood/src.
• tsup ESM+CJS+DTS, publishConfig.access: public, third-party-facing README.

New objectstack verify CLI command (packages/cli/src/commands/verify.ts)

• oclif command reusing the existing loadConfig; flags --app/--rls/--multi-tenant/--json.
• Exit-code contract (CI-usable): create-failed / read-failed / fidelity-gaps / rls-hole → exit 1; inconclusive verdicts (needs-fixture / skipped / member-visible) warn and exit 0.

@objectstack/dogfood kept, slimmed (single source of truth)

• now consumes @objectstack/verify; deps 14 → 6.
• drops 3 now-redundant auto-verify tests, replaced by an objectstack verify --rls CI step over the example apps (Dogfood Regression Gate job).
• keeps the hand-written golden regressions the generic verifier can't auto-derive: fix(analytics): make organization timezone drive date-dimension bucketing (#1982) #2018 tz bucketing, fix(security)[P0]: enforce RLS on by-id writes — close member-edits-others'-records hole (#1985) #1994 cross-owner RLS (incl. the automated red proof), fix: array field types persist + Field.time accepts time-of-day #2004 field fidelity.

Release plumbing

• @objectstack/verify added to the changeset fixed group (@objectstack/dogfood stays private/excluded); check-changeset-fixed.mjs passes (73 packages in sync). Changeset added for verify + cli.

Validated end-to-end on a real third-party app (hotcrm)

Ran the built CLI against ../hotcrm (a standalone marketplace CRM on published @objectstack@^9.9.1, with real sharing / territory / role-hierarchy rules — not in this repo):

• Booted hotcrm's full 14-object stack through the real HTTP surface and verified data fidelity.
• Demonstrated the exit-1 red path on a real app — crm_product.sku reads back uppercased (an intentional normalization hook), correctly flagged as a fidelity-gaps mismatch → exit 1.
• fix(security)[P0]: enforce RLS on by-id writes — close member-edits-others'-records hole (#1985) #1994 authorization invariant holds on hotcrm: --rls --multi-tenant → 8 consistent / 0 holes (a fresh member can neither read 404 nor mutate 403 cross-org records) across its real sharing rules.

The exercise also surfaced two @objectstack/verify tool-quality issues (not hotcrm bugs), both addressed here:

1. Fixed — the CLI ran fidelity + RLS proofs on the same in-process stack, so the RLS phase's admin-creates collided with the fidelity phase's rows on unique-constrained fields (a 409 that silently skipped those objects). The RLS phase now boots a separate pristine stack (hotcrm --rls --multi-tenant: 5-consistent/3-skipped → 8-consistent/0-skipped-by-409).
2. Documented — fields normalized on write (uppercase/trim hooks, canonicalizing formulas) read back as fidelity-gaps even when working as designed; the report shows the wrote → read diff. A declared-transform allowlist is a planned enhancement.

Validated across the template corpus (9 apps)

Ran the CLI across all 9 apps in ../templates (the AI-authored starter corpus: expense, contracts, content, compliance, todo, project, procurement, hr, helpdesk), on an older @objectstack than this branch — a cross-version run:

• Data fidelity: 0 gaps / 0 failures across the corpus (~18 objects auto-verified).
• RLS --multi-tenant: 0 holes — every auto-exercised object is rls-consistent; the fix(security)[P0]: enforce RLS on by-id writes — close member-edits-others'-records hole (#1985) #1994 invariant holds across all 9 AI-authored apps.
• The verifier handled the whole corpus cleanly — no crashes, no false 409s after the stack-isolation fix.

Coverage caveat: the auto-derive skips objects whose required fields it can't synthesize (lookups / master-detail), so relational-heavy templates have ~half their objects skipped — "0 holes" is over the auto-exercised subset. Auto-synthesizing related records in topological order to reach those objects is a planned enhancement.

Verification

• @objectstack/verify builds (ESM+CJS+DTS).
• dogfood: 5 files / 57 tests green after the extraction and after rebasing onto the latest main (which carries spec/RLS schema changes) — zero behavior change.
• objectstack verify against the framework examples: crm 4 verified / 2 skipped / exit 0, crm --rls 0 holes, showcase exit 0; clean process exit (fixed a hang: the in-process stack leaves handles open after stop(), so the command forces this.exit()).
• objectstack verify against hotcrm (real third-party): boots, verifies, proves exit-1 and the fix(security)[P0]: enforce RLS on by-id writes — close member-edits-others'-records hole (#1985) #1994 invariant (above).
• canonical lint, check-changeset-fixed.mjs, and ci.yml YAML all green/valid.

Fix found along the way

The harness used a port-less loopback inject origin (http://localhost) which fails better-auth's default dev trusted-origins (http://localhost:*) → objectstack verify got a 403 on dev-admin sign-in in a bare node CLI (the dogfood tests passed only because turbo's strict-env run masked it). Switched to a ported loopback base so sign-in is deterministic across the node CLI and the test runner.

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
spec	Ready	Preview, Comment	Jun 19, 2026 12:12am

[github-actions]

📓 Docs Drift Check

This PR changes 3 package(s): @objectstack/cli, @objectstack/dogfood, @objectstack/verify.

15 hand-written doc(s) reference the affected code and may need an implementation-accuracy re-verification:

• content/docs/concepts/cloud-artifact-api.mdx (via packages/cli)
• content/docs/concepts/implementation-status.mdx (via @objectstack/cli)
• content/docs/concepts/packages.mdx (via @objectstack/cli)
• content/docs/getting-started/cli.mdx (via @objectstack/cli)
• content/docs/getting-started/quick-start.mdx (via @objectstack/cli)
• content/docs/guides/authentication.mdx (via @objectstack/cli)
• content/docs/guides/client-sdk.mdx (via @objectstack/cli)
• content/docs/guides/hook-bodies.mdx (via packages/cli)
• content/docs/guides/packages.mdx (via @objectstack/cli)
• content/docs/guides/project-scoping.mdx (via @objectstack/cli)
• content/docs/guides/runtime-services/data-service.mdx (via packages/cli)
• content/docs/guides/runtime-services/index.mdx (via packages/cli)
• content/docs/guides/skills.mdx (via packages/cli)
• content/docs/protocol/objectos/plugin-spec.mdx (via @objectstack/cli)
• content/docs/protocol/objectos/realtime-protocol.mdx (via @objectstack/cli)

Advisory only. To re-verify, run the docs-accuracy-audit workflow scoped to these files:
node scripts/docs-audit/affected-docs.mjs origin/main → pass the list as args.docs.