Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Introduce PgRegistry; codecs own relations; type overhaul #260

Merged
merged 192 commits into from
Apr 4, 2023

Conversation

benjie
Copy link
Member

@benjie benjie commented Mar 29, 2023

First: the term "source" is overused, so we've renamed PgSource to PgResource. I'll try and refer to "resources" rather than "sources" from here on out. We've also renamed PgTypeCodec to simply PgCodec.

We want the schema to be exportable, which means the underlying "resources" need to be exportable. The resources relate to other resources, and sometimes these relations form circular dependencies, e.g. User.posts -> Post and Post.author -> User. When we try and export these we can't, because we can't define const User = ... without first defining Post, and we can't define const Post = ... without first defining User. How to solve?

Initially I solved this with a PgSourceBuilder. This was a precursor to a PgSource (now PgResource) - everything except the relationships - and then relationships could be to either a PgSource directly or to a PgSourceBuilder that would later "become" a PgSource (via PgSourceBuilder.get()). Code throughout the codebase had to say "if it's a PgSourceBuilder, .get() it, otherwise use it directly" and it was ugly. I didn't like it, and concluded that "relations" should be a separate thing (see #117).

From discussion with @jemgillam we realised that we could significantly simplify things by building first the codecs, then the resources, then the relations... But that accessing the relations would be painful, and more importantly the relations might not export along side everything else, which might cause the exported schema to be incomplete. Clearly not viable. We needed something that was interlinked with circular dependencies at runtime, but that was produced without circular dependencies at build time... We needed a registry!

The registry

The idea with the registry is simple: we take a list of codecConfigs, resourceConfigs, and relationshipConfigs (all without circular dependencies - a directed acyclic graph) and then we build from that a structure (the "registry") that contains the built codecs, resources and relationships which can link back to the central registry. So the "resources" don't actually "know" their relationships - they just know the "registry" which stores all the relationship data. And the registry itself is built via a factory function which we can export with the raw configs as input and no circular dependency issues.

graph TD
   PgCodecConfig
   PgResourceConfig
   PgRelationshipConfig
   PgRegistry
   PgCodec
   PgResource
   PgRelationship

   PgCodecConfig --> PgResourceConfig
   PgResourceConfig --> PgRelationshipConfig
   PgCodecConfig --> PgRelationshipConfig
   PgCodecConfig ---> PgRegistryConfig
   PgResourceConfig ---> PgRegistryConfig
   PgRelationshipConfig ---> PgRegistryConfig

   PgRegistryConfig -->|build| PgRegistry

   PgRegistry --> PgCodec
   PgCodec --> PgResource
   PgRegistry --> PgResource
   PgRegistry --> PgRelationship
   PgResource --> PgRegistry
   PgCodec --> PgRelationship
   PgResource --> PgRelationship

This also meant that we could maintain the types much better:

image

(Early screenshot, API has changed slightly.)

And also, the mixture of PgSource/PgSourceBuilder has been cleared up via the following rules:

  • PgResourceOptions is generated during the "gather" phase, you don't deal with PgResource directly during gather (until the very end when the PgRegistry is built)
  • PgResource is consumed during the "schema" phase, you don't deal with PgResourceOptions during the "schema" phase at all.

So no more hacky resolveSource()/sourceBuilder.get() nonsense!

Lots of breaking changes

  • TypeScript v5 required, we use satisfies and const-generics
  • PgSourceBuilder is no more (but it has effectively been replaced with PgResourceOptions)
  • PgSource* -> PgResource*
  • PgTypeCodec* -> PgCodec*
  • PgTypeColumn* -> PgCodecAttribute*
  • listOfType -> listOfCodec
  • The generics on PgResource and PgCodec have changed significantly - the first generic is now a name (string) which allows us to build up the registry nicely
  • Now that codecs own relationships, this has been reflected in the relationship inflectors; they no longer accept {resource, relationName}, instead accepting {registry, codec, relationName}. (Note: registry = resource.registry, codec = resource.codec.)
    • singleRelation
    • singleRelationBackwards
    • _manyRelation
    • manyRelationConnection
    • manyRelationList

Fixes #117

benjie added 26 commits April 3, 2023 16:11
…Codec` to `PgCodec`,

`PgEnumTypeCodec` to `PgEnumCodec`, `PgTypeColumn` to `PgCodecAttribute` (and
similar for related types/interfaces). `source` has been replaced by `resource`
in various of the APIs where it relates to a `PgResource`.

`PgSourceBuilder` is no more, instead being replaced with `PgResourceOptions`
and being built into the final `PgResource` via the new
`makeRegistryBuilder`/`makeRegistry` functions.

`build.input` no longer contains the `pgSources` directly, instead
`build.input.pgRegistry.pgResources` should be used.

The new registry system also means that various of the hooks in the gather
phase have been renamed/replaced, there's a new `PgRegistryPlugin` plugin in
the default preset. The only plugin that uses the `main` method in the `gather`
phase is now `PgRegistryPlugin` - if you are using the `main` function for
Postgres-related behaviors you should consider moving your logic to hooks
instead.

Plugin ordering has changed and thus the shape of the final schema is likely to
change (please use `lexicographicSortSchema` on your before/after schemas when
comparing).

Relationships are now from a codec to a resource, rather than from resource to
resource, so all the relationship inflectors (`singleRelation`,
`singleRelationBackwards`, `_manyRelation`, `manyRelationConnection`,
`manyRelationList`) now accept different parameters (`{registry, codec,
relationName}` instead of `{source, relationaName}`).

Significant type overhaul, most generic types no longer require generics to be
explicitly passed in many circumstances. `PgSelectStep`, `PgSelectSingleStep`,
`PgInsertStep`, `PgUpdateStep` and `PgDeleteStep` now all accept the resource
as their single type parameter rather than accepting the 4 generics they did
previously. `PgClassExpressionStep` now accepts just a codec and a resource as
generics. `PgResource` and `PgCodec` have gained a new `TName extends string`
generic at the very front that is used by the registry system to massively
improve continuity of the types through all the various APIs.

Fixed various issues in schema exporting, and detect more potential
issues/oversights automatically.

Fixes an RBAC bug when using superuser role for introspection.
@benjie benjie changed the title Introduce PgRegistry - PgSourceBuilder is no more Introduce PgRegistry; codecs own relations; type overhaul Apr 4, 2023
@benjie benjie merged commit 4ab5ca5 into planning Apr 4, 2023
@benjie benjie deleted the codec-relations branch April 4, 2023 11:16
@benjie benjie added this to the V5 alpha milestone Apr 13, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Relations should be a separate thing
1 participant