TML-2747: drop the storage.namespaces wrapper (flat storage.<ns> IR)#649
TML-2747: drop the storage.namespaces wrapper (flat storage.<ns> IR)#649wmadden-electric wants to merge 61 commits into
Conversation
… stack S1 closed but shipped storage under a `namespaces` wrapper and never wired the domain plane, so the emitted IR diverges from ADR 221. Re-plan the umbrella's remaining work from scratch as one sequential thread (S2 IR-canonicalization -> S3 Postgres public-by-default -> S4 runtime qualification -> S5 explicit-namespace DSL, deferrable) rather than two parallel projects: the only real concurrency (GAP 2) is not worth the cross-project coordination, and GAP 1 must precede S4 regardless. Signed-off-by: Will Madden <madden@prisma.io>
Intent-driven slice spec for TML-2747: bring the storage plane to ADR 221's canonical `storage.<ns>` shape (no `namespaces` segment; `storageHash` a reserved sibling key). File discovery left to the implementer. Signed-off-by: Will Madden <madden@prisma.io>
D1 surgical substrate change (wrapper drop + in-memory typing decision); D2 mechanical artifact regeneration + full gate sweep. Kept apart so design judgment and mechanical fan-out do not share a dispatch. Signed-off-by: Will Madden <madden@prisma.io>
…substrate Namespace ids become direct keys under `storage` alongside the reserved `storageHash`/`types` keys (ADR 221 canonical shape). Adds `storage-plane-keys` with the reserved-key vocabulary plus `storageNamespaceEntries`/`storageNamespaceValues`/`getStorageNamespace` walk helpers, and reworks `elementCoordinates` to enumerate namespace-id keys directly while skipping reserved keys. `StorageBase` loses its `namespaces` member; `Storage` keeps only the typed `storageHash`. Updates the foundation testing factories and the framework unit tests to the flat shape. Signed-off-by: Will Madden <madden@prisma.io>
SqlStorage sets namespace ids as own-enumerable keys via `Object.defineProperty`, keeping `storageHash`/`types` as reserved siblings; `buildSqlStorageInput` spreads a namespace map into the flat input. Validators and canonicalization hooks walk the flat shape via the framework `storageNamespaceEntries`/`getStorageNamespace` helpers, and the arktype storage schema validates every non-reserved key as a namespace entry. Signed-off-by: Will Madden <madden@prisma.io>
The SQL emitter generates `storage.<ns>` namespace entries as direct siblings of `storageHash` (no `namespaces` wrapper) and walks the flat shape for table/enum resolution. Authoring builders and their unit tests move to the flat shape. Signed-off-by: Will Madden <madden@prisma.io>
…aces Query-builder selection and relational-core codec lookup walk namespace entries directly under `storage` rather than through `storage.namespaces`. Storage-path update only — no change to the per-namespace type structure. Signed-off-by: Will Madden <madden@prisma.io>
The sql-builder runtime and `Db`/`UnboundTables`/`TableProxyContract` types index namespace entries directly under `storage` instead of `storage.namespaces`. Storage-path update only — the `TableNamesAcrossNamespaces`/`TableInAnyNamespace` indexing keeps the same per-namespace structure, just rooted at the flat storage object. The deferred `UnboundTables` rewrite (TML-2745) is NOT pulled in. Signed-off-by: Will Madden <madden@prisma.io>
The SQL runtime context and codec validation walk namespace entries directly under `storage` via the framework helpers instead of `storage.namespaces`. Unit-test fixtures updated to the flat shape. Signed-off-by: Will Madden <madden@prisma.io>
The SQL family contract-serializer hydration rebuilds `SqlStorage` from flat namespace keys (via `buildSqlStorageInput`), and the schema-verify, migration field-event planner, and contract-to-schema-IR walks iterate namespace entries directly under `storage`. Family unit tests and helpers updated to the flat shape. Signed-off-by: Will Madden <madden@prisma.io>
MongoStorage sets namespace ids as own-enumerable keys via `Object.defineProperty` with `storageHash` reserved; `buildMongoStorageInput` spreads a namespace map into the flat input. The contract schema validates every non-reserved storage key as a namespace envelope, and validate-storage / canonicalization hooks walk the flat shape. Foundation tests and the hand-authored orm-contract fixture updated to the flat shape. Signed-off-by: Will Madden <madden@prisma.io>
The Mongo authoring interpreters/builders, emitter (flat `storage.<ns>` namespace type generation), family contract-serializer hydration, schema verifier, schema-diff, and contract-to-schema walks read namespace entries directly under `storage`. Authoring + family unit tests updated to the flat shape. Signed-off-by: Will Madden <madden@prisma.io>
The migration aggregate verifier, project-schema-to-space, and integrity check read namespace entries directly under `storage` via the framework helpers instead of `storage.namespaces`. Aggregate tests updated to the flat shape. Signed-off-by: Will Madden <madden@prisma.io>
Postgres contract serializer, schema, migration planners (enum-planning, issue-planner, planner-strategies, verify-postgres-namespaces), the postgres SQL renderer, and the sqlite planner read namespace entries directly under `storage` via the framework helpers. Target serializer and snapshot tests updated to the flat shape. Signed-off-by: Will Madden <madden@prisma.io>
The Mongo target contract serializer and contract construction wrap the flat namespace shape (namespace ids keyed directly under `storage`), and target serializer/schema-verifier tests follow. Signed-off-by: Will Madden <madden@prisma.io>
The sql-orm-client model accessor, collection contract, where-binding, and query-plan helpers resolve tables through namespace entries directly under `storage` via the framework helpers. Extension PSL-interpretation and descriptor tests updated to the flat shape. Signed-off-by: Will Madden <madden@prisma.io>
`storageNamespaceEntries`/`storageNamespaceValues`/`getStorageNamespace` take `object` and are generic over the namespace concretion (callers select `SqlNamespace` etc.), so class instances and plain records flow in without per-call-site casts. Drops the now-redundant `as Record<string, unknown>` arg casts across the SQL substrate, routes build-contract through `buildSqlStorageInput`, and fixes the typecheck fallout in SQL authoring/runtime/family and their tests. Signed-off-by: Will Madden <madden@prisma.io>
Removes the now-unnecessary `as Record<string, unknown>` arg casts at the generic walk-helper call sites across the Mongo family/target, the SQL extensions, sqlite, and the postgres adapter, and fixes the related typecheck fallout. Signed-off-by: Will Madden <madden@prisma.io>
…ntCoordinates Finishes the postgres target wrapper drop the mechanical pass left incomplete: imports the generic walk helpers in the migration planners, schema, and serializer; fixes a botched `ctx.toContract.storage` lookup; routes test fixtures through `buildSqlStorageInput`; and selects `SqlNamespace` at `.tables`/`.enum` access sites. Also widens `elementCoordinates` to accept `object` so class instances pass without a cast. Signed-off-by: Will Madden <madden@prisma.io>
Imports the generic walk helper in the sqlite migration planner and routes the planner/serializer test fixtures through `buildSqlStorageInput` with `SqlNamespace`-typed lookups, finishing the wrapper drop the mechanical pass left incomplete for this target. Signed-off-by: Will Madden <madden@prisma.io>
…d tests Trims the unused walk-helper imports in the SQL renderer, selects `SqlNamespace` at the `.tables` lookup, and routes every migration test fixture through `buildSqlStorageInput`, finishing the wrapper drop for the postgres adapter. Signed-off-by: Will Madden <madden@prisma.io>
Routes every sqlite migration test fixture through `buildSqlStorageInput` (namespace ids keyed directly under `storage`), finishing the wrapper drop for the sqlite adapter that the mechanical pass left incomplete. Signed-off-by: Will Madden <madden@prisma.io>
The flat `TableProxyContract` constraint only requires the reserved `storageHash` key (intersecting `Record<string, namespaceEntry>` would force `storageHash` itself to be a namespace, which the reserved-key storage shape cannot satisfy and which broke `Db<Contract<SqlStorage>>` in the adapter runtimes). The table-name walk re-establishes the `StorageTable` upper bound the prior `namespaces`-keyed constraint gave via `infer Tables extends Record<string, StorageTable>`, and excludes the reserved sibling keys. Drops the now-unused walk-helper imports in the runtime and routes the field-proxy fixture through `buildSqlStorageInput`. This is the reserved-key typing decision, not the deferred multi-namespace DSL surface (TML-2745). Signed-off-by: Will Madden <madden@prisma.io>
Imports the generic walk helper across the model accessor, collection contract, where-binding, and query-plan helpers; selects `SqlNamespace` at `.tables` lookups; flattens the `NamespaceTableDef` source type to walk namespace-id keys directly (excluding the reserved `storageHash`/`types` siblings); and gives the test helper a mutable namespace view for in-place fixture assembly. Signed-off-by: Will Madden <madden@prisma.io>
…tor/postgres tests Adds the generic walk-helper imports, selects `SqlNamespace` at `.tables`/`.enum` lookups, repairs a mechanically-mangled `getStorageNamespace` call in the cipherstash descriptor test, and routes the cipherstash codec fixture through `buildSqlStorageInput`. Signed-off-by: Will Madden <madden@prisma.io>
Routes the pgvector planner test helpers and storage-types fixtures
through `buildSqlStorageInput` with the namespaces-keyed input type, and
drops the stale `namespaces: {}` wrapper from the cli contract-enrichment
test IR. pgvector typechecks fully green.
Signed-off-by: Will Madden <madden@prisma.io>
Drops the stale `namespaces` wrapper from the db-verify test contract so namespace ids key directly under `storage`. CLI typechecks fully green. Signed-off-by: Will Madden <madden@prisma.io>
…rage schema The storage arktype schema used `+: reject`, which rejected the namespace-id keys that now sit directly under `storage` (ADR 221 flat shape) before the narrow could validate them — so genuine flat contracts failed structural validation with "storage.<ns> must be removed". Switches the base to `+: ignore` and lets the narrow validate every non-reserved key as a namespace entry. Flattens the validators test fixtures to the wrapper-less shape. Signed-off-by: Will Madden <madden@prisma.io>
…izer tests Unwraps the `namespaces` wrapper from the mongo family test fixtures and routes the fake-storage builder through `buildMongoStorageInput`, and restores the optional `storageHash` on the mongo storage schema. Signed-off-by: Will Madden <madden@prisma.io>
…ath patterns Drops the `namespaces` wrapper from the test fixtures, helpers, sort/ preserve-empty path patterns, and the structural assertion so they match the flat ADR 221 storage shape. Signed-off-by: Will Madden <madden@prisma.io>
…lat storage Signed-off-by: Will Madden <madden@prisma.io>
D1 flattened the IR SqlStorage type and the runtime emit but left the contract-builder output type BuiltStorage nesting namespace ids under a `namespaces` wrapper, so every defineContract result was unassignable to the flat Contract<SqlStorage>. Hoist namespace ids (and the named- namespace mapped type) to direct keys under storage, mirroring the flat IR shape. Type-only; no runtime behaviour change. Signed-off-by: Will Madden <madden@prisma.io>
…s to flat shape Signed-off-by: Will Madden <madden@prisma.io>
…value-objects fixtures to flat shape Signed-off-by: Will Madden <madden@prisma.io>
…at shape Signed-off-by: Will Madden <madden@prisma.io>
Signed-off-by: Will Madden <madden@prisma.io>
Signed-off-by: Will Madden <madden@prisma.io>
Signed-off-by: Will Madden <madden@prisma.io>
…at storage Regenerate src contracts (audit, feature-flags, app) and per-space migration refs (ops.json, migration.json, head.json) to the flat storage shape via the documented emit + tsx incantation. The migration end-contract bookends are not auto-regenerated (TML-2698) and remain on the prior shape; they are explicitly excluded from the snapshot-read-shapes scan. Signed-off-by: Will Madden <madden@prisma.io>
…ns to flat storage Signed-off-by: Will Madden <madden@prisma.io>
Signed-off-by: Will Madden <madden@prisma.io>
…ath predicates to flat shape Signed-off-by: Will Madden <madden@prisma.io>
extractContractNamespaceIds still reached through the dropped storage.namespaces wrapper, so flat-storage contracts yielded zero declared namespaces and introspect fell back to the single public schema. Cross-namespace contracts then verified as missing every non-public table. Read namespace ids via storageNamespaceEntries so the multi-namespace introspection walk covers every declared schema. Signed-off-by: Will Madden <madden@prisma.io>
Signed-off-by: Will Madden <madden@prisma.io>
…o flat storage Re-express the contract-ts edge case for the flat shape: storage with only storageHash now deserializes to an empty unbound namespace and is accepted rather than rejected. Repair the dropped trailing comma the codemod left in sql-storage fixtures and switch interpreter type assertions to flat namespace-key access. Signed-off-by: Will Madden <madden@prisma.io>
…orm-client storage reads to flat shape Signed-off-by: Will Madden <madden@prisma.io>
…ter test reads to flat storage Signed-off-by: Will Madden <madden@prisma.io>
…erals and assertions The flat-storage codemod had not reached test/integration, so these contract literals and toMatchObject assertions still wrapped namespace entries under storage.namespaces. Promote namespace ids to direct storage keys (ADR 221 flat shape). Signed-off-by: Will Madden <madden@prisma.io>
…spaces-wrapper-drop Signed-off-by: Will Madden <madden@prisma.io> # Conflicts: # examples/mongo-blog-leaderboard/src/contract.d.ts # examples/mongo-blog-leaderboard/src/contract.json # examples/mongo-demo/src/contract.d.ts # examples/mongo-demo/src/contract.json # examples/retail-store/src/contract.d.ts # examples/retail-store/src/contract.json # packages/2-mongo-family/2-authoring/contract-psl/test/interpreter.test.ts # test/integration/test/authoring/side-by-side/mongo/contract.json
|
Important Review skippedToo many files! This PR contains 292 files, which is 142 over the limit of 150. To get a review, narrow the scope: ⚙️ Run configurationConfiguration used: Path: .coderabbit.yml Review profile: CHILL Plan: Pro Run ID: ⛔ Files ignored due to path filters (8)
📒 Files selected for processing (292)
You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
@prisma-next/extension-author-tools
@prisma-next/mongo-runtime
@prisma-next/family-mongo
@prisma-next/sql-runtime
@prisma-next/family-sql
@prisma-next/extension-arktype-json
@prisma-next/extension-cipherstash
@prisma-next/middleware-cache
@prisma-next/mongo
@prisma-next/extension-paradedb
@prisma-next/extension-pgvector
@prisma-next/extension-postgis
@prisma-next/postgres
@prisma-next/sql-orm-client
@prisma-next/sqlite
@prisma-next/target-mongo
@prisma-next/adapter-mongo
@prisma-next/driver-mongo
@prisma-next/contract
@prisma-next/utils
@prisma-next/config
@prisma-next/errors
@prisma-next/framework-components
@prisma-next/operations
@prisma-next/ts-render
@prisma-next/contract-authoring
@prisma-next/ids
@prisma-next/psl-parser
@prisma-next/psl-printer
@prisma-next/cli
@prisma-next/cli-telemetry
@prisma-next/emitter
@prisma-next/migration-tools
prisma-next
@prisma-next/vite-plugin-contract-emit
@prisma-next/mongo-codec
@prisma-next/mongo-contract
@prisma-next/mongo-value
@prisma-next/mongo-contract-psl
@prisma-next/mongo-contract-ts
@prisma-next/mongo-emitter
@prisma-next/mongo-schema-ir
@prisma-next/mongo-query-ast
@prisma-next/mongo-orm
@prisma-next/mongo-query-builder
@prisma-next/mongo-lowering
@prisma-next/mongo-wire
@prisma-next/sql-contract
@prisma-next/sql-errors
@prisma-next/sql-operations
@prisma-next/sql-schema-ir
@prisma-next/sql-contract-psl
@prisma-next/sql-contract-ts
@prisma-next/sql-contract-emitter
@prisma-next/sql-lane-query-builder
@prisma-next/sql-relational-core
@prisma-next/sql-builder
@prisma-next/target-postgres
@prisma-next/target-sqlite
@prisma-next/adapter-postgres
@prisma-next/adapter-sqlite
@prisma-next/driver-postgres
@prisma-next/driver-sqlite
commit: |
size-limit report 📦
|
…test The flat-storage migration left getStorageNamespace / storageNamespaceEntries / storageNamespaceValues imported but unused (the only textual hit was inside an error-message string), tripping biome noUnusedImports under --error-on-warnings in CI. Signed-off-by: Will Madden <madden@prisma.io>
…test The preserve-empty predicate now matches the flat storage.<ns>.collections path (no namespaces wrapper); the test still passed the old wrapped path, so the empty-collections-slot case asserted false where the flat hook returns true. Signed-off-by: Will Madden <madden@prisma.io>
| /** | ||
| * Own-enumerable keys under a storage plane object that are not namespace | ||
| * entries. Walkers (`elementCoordinates`, validators, serializers) skip these. | ||
| */ | ||
| export const STORAGE_PLANE_RESERVED_KEYS = ['storageHash', 'types'] as const; |
There was a problem hiding this comment.
types is SQL-specific, isn't it? It shouldn't have special handling here.
|
|
||
| function isNamespaceEntry(value: unknown): value is Namespace { | ||
| return ( | ||
| typeof value === 'object' && value !== null && 'id' in value && typeof value.id === 'string' |
There was a problem hiding this comment.
This seems like an unreliable condition. Anything with a string id? That's so context-specific the type predicate is pointless
| if (isStoragePlaneReservedKey(key)) continue; | ||
| if (isNamespaceEntry(value)) { |
There was a problem hiding this comment.
Looks like this logic is here only because types is a top-level member alongside namespaces. There should be no top-level members which aren't within a namespace, otherwise this strategy doesn't work
| /** | ||
| * Spread a namespace map into flat storage input (namespace ids as direct keys). | ||
| */ | ||
| export function flatStorageInput<TNamespace>(params: { |
There was a problem hiding this comment.
This function looks like a code smell: is it just converting the new structure back into the old?
|
Closing without merging — this slice took the IR in the wrong direction. The goal was symmetric two-plane IR. Decision (with @wmadden): keep the The one code change here that looked like a standalone fix ( |
## Linked issue Refs [TML-2751](https://linear.app/prisma/issue/TML-2751) ## At a glance **Before** — models lived at the contract root: ```json { "target": "postgres", "models": { "User": { "fields": { "id": { "nullable": false, "type": { "kind": "scalar", "codecId": "pg/int4@1" } } }, "storage": { "table": "user", "fields": { "id": { "column": "id" } } } } }, "storage": { "namespaces": { "__unbound__": { "tables": { "user": { } } } } } } ``` **After** — the application plane is symmetric with storage; models and value objects sit under `domain.namespaces`: ```json { "target": "postgres", "domain": { "namespaces": { "__unbound__": { "models": { "User": { "fields": { "id": { "nullable": false, "type": { "kind": "scalar", "codecId": "pg/int4@1" } } }, "storage": { "table": "user", "fields": { "id": { "column": "id" } } } } }, "valueObjects": { } } } }, "storage": { "namespaces": { "__unbound__": { "tables": { "user": { } } } } } } ``` `storage.namespaces` is unchanged in shape and responsibility. The framework domain plane has **no** `domain.types` bucket (SQL storage types stay on `storage` only). ## Why ADR 221 calls for symmetric plane envelopes: both storage and domain carry a `namespaces` segment so extension packs and multi-namespace authoring compose the same way at each plane. Consumers stop reading flat `contract.models` / `contract.valueObjects`; runtime and DSL code use `contract.domain.namespaces` and helpers such as `contractModels()` / `ContractModelsMap`. ## What changed - Contract IR, canonicalization, and validation emit and require `domain.namespaces`. - SQL and Mongo emitters, validators, and generated `contract.d.ts` include the domain envelope. - `contractModels`, `ContractModelsMap`, and `ContractValueObjectsMap` are the supported access paths; legacy flat domain roots are normalized at the boundary only. - SQL `insert()` / update row typing resolves models via `ContractModelsMap` (fixes `never` rows when only `domain` is populated). - Tests, examples, and tracked contract fixtures updated; `pnpm fixtures:check` passes. ## Alternatives considered [PR #649](#649) explored **flattening storage** (lifting tables out of `storage.namespaces`). That direction was abandoned in favour of this symmetric envelope: keep `storage.namespaces` as-is, add `domain.namespaces` for models/value objects, and avoid `STORAGE_PLANE_RESERVED_KEYS` or other storage reshaping. ## Testing performed - [x] `pnpm fixtures:check` - [x] `test/integration` package `pnpm typecheck` - [ ] Full `pnpm test:packages` — last run: type errors clean; a few PGlite/MMS integration flakes in `@prisma-next/adapter-postgres` and unrelated packages (re-run on CI) - [ ] Full `pnpm test:integration` — not re-run end-to-end in this session after typecheck green; pretest build includes postgis demo (insert typing fixed via sql-builder) ## Checklist - [x] No `domain.types` on framework contracts - [x] No storage flatten / `STORAGE_PLANE_RESERVED_KEYS` - [x] Upgrade instructions not recorded (fixture/hash-only and test changes; no `examples/` or `packages/3-extensions/` source API changes requiring consumer migrations) --------- Signed-off-by: Will Madden <madden@prisma.io> Co-authored-by: Will Madden <madden@prisma.io>
At a glance
Here is a Postgres contract's
storageplane, before and after this change:The
namespacesenvelope is gone. Namespace ids (__unbound__, and — in later slices —public,auth, …) are now direct keys understorage, andstorageHashsits beside them as a reserved sibling key. Reading a namespace goes fromstorage.namespaces.public.tablestostorage.public.tables.Decision
Key the storage plane's namespaces directly under
contract.storage, matching the canonical IR shape described in ADR 221.storageHashbecomes a reserved sibling key (no namespace may be namedstorageHash).This is the first slice of bringing the contract IR to its ADR 221 shape, and it does the storage plane only.
Why
The previous milestone shipped per-namespace storage, but kept every namespace behind a
namespaceswrapper segment that ADR 221 never describes. That wrapper had two costs:storage.namespaces.<id>everywhere — an extra hop that carried no information.StorageBase(foundation) is hash-only; the namespace map lived a layer up. Code that needed to walk namespaces from foundation resorted to a duck-typing shim, because foundation may not import the layer where the map was declared.Dropping the wrapper removes that whole class of accidental complexity: namespace ids are the storage keys, and three generic walk helpers —
storageNamespaceEntries,storageNamespaceValues,getStorageNamespace— replace the ad-hoc reads across both families, targets, and extensions.What changed
lint:castsis net −10 vs base — every wrapper-era cast removed, the few genuinely unavoidable ones converted to justifiedblindCast.contract.json/contract.d.ts, all migrationstorageHashpins, and inline test fixtures regenerated/flattened.extractContractNamespaceIdsstill read the old wrapper, so flat-storage contracts introspected only thepublicschema and cross-namespace tables verified as missing. It now reads namespace ids through the walk helpers.main. Reconciled with the closed-Mongo-$jsonSchema-validators change that landed in parallel; regenerated artifacts carry both.Out of scope (deliberately)
These belong to later slices in the same stack and are not touched here:
contract.models→contract.domain.<ns>.models(next slice).public-by-default — the PSL policy for un-namespaced models."public"."user"at query time.Verification
typecheck(138/138) ·fixtures:check·lint:deps·lint:casts(−10) ·test:e2e(104/104) — all green.test:integrationis green modulo pre-existing PGlite connection-churn flakiness (passes in isolation).Alternatives considered
namespaceswrapper. Rejected: it diverges from ADR 221, taxes every consumer with a dead segment, and is what forced the foundation-layer duck-typing shim in the first place.storageHashas a nested metadata object (e.g.storage.__meta__.storageHash) rather than a reserved sibling key. Rejected as unnecessary ceremony: a single reserved key is simpler, and namespace ids are already validated, so collision withstorageHashis a non-issue.