Controlling module resolution of zapatos/schema

[freshtonic]

I'm having difficulties getting my project to build.

Essentially I want to create a publishable node module (let's call it my-db) that exports the zapatos DB instance and the generated schema types and I have another module that consumes it (let's call it my-db-consumer).

The problem is that my-db-consumer does not know how to resolve zapatos/schema. And nor should it have to, really. It should be an implementation detail of my-db.

If anyone has any pointers on how to set this up (package.json & tsconfig.json examples etc), that would be fantastic.

Thanks in advance!

P.S. have you considered creating "meta-schema" types that the generated types must conform to, and instead allowing the database connection to be instantiated with a type argument rather than it resolving a special module name (zapatos/schema) to resolve the types? The special module name strategy for resolving types is extra-linguistic and seems to lead to build environment complications.

[jawj]

Thanks for filing an issue.

Is this something that worked until version 3?

I think the new ambient declaration approach is generally better, because it makes this just a normal library that doesn't have to copy its code into your project, but I can see it may also have downsides.

I think to see if this can be resolved we'll probably need to see a minimal test case. Are you able to share a repo with the setup that now doesn't work?

On your PS, I'm not sure what this looks like. Could you elaborate with a concrete example?

[maxdeviant]

I would think in this case you would want my-db to re-export the schema type, which I think would make it get emitted when you compile the my-db project.

[jawj]

I think #53 may help here.

[ellis]

@jawj I'm facing the same issue. I have a "my-db" library like freshtonic described, but built on zapatos v2, and it works great for this use-case. I was trying to do the same thing in a new project using the latest version of zapatos, but I see that it doesn't work anymore. It'd be great if #53 would solve this...?

[jawj]

@ellis I think the way to check if #53 would solve this is simply to rename all your .d.ts files under zapatos to .ts files, and rebuild your code.

[ellis]

@jawj I tried that, but was unable to import the equivalent for "zapatos/db" in the "my-db-consumer" project, since it's simply not present in the generated zapatos output. Or am I just missing it? Any alternative suggestions?

(Update:) After a bit of further experimentation, I've encountered another issue that I'll mention here for context. My "my-db-consumer" package is a next.js app. Unfortunately, I can't generate the zapatos files directly inside it because of the required isolatedModules option in tsconfig.json, which makes lines like declare module "zapatos/custom" invalid.

(Update2:) I switched to zapatos v2 and things are working as desired. I'll ultimately need to access a second external database as well. Could it be that zapatos v3 doesn't support that either? Just thinking aloud: as elegant as "ambient" modules are, if they don't allow us to modularize our database code into separate packages or access multiple databases, those would seem like serious limitations. Zapatos is everything I've dreamed of in a postgres library, so personally, I'm perfectly happy to deal with the inelegance of keeping generated zapatos v2 code in my git repo, considering how incredibly awesome the zapatos approach is.

[ellis]

@jawj I've been expanding my use of zapatos v2 significantly throughout the projects I'm working on. It's truly fantastic. I was wondering if you could provide some feedback about a concern, however? I'm concerned about only being able to use v2 (for encapsulating the database definitions in their own module, and for accessing multiple databases), while development is progressing in v3. Do you think zapatos will stick with the limitations inherent in ambient modules for the foreseeable future?

[sfsr12]

Does anyone have a working solution for accessing zapatos that is configured in a package that gets imported into another package?

I am working in a project that has a db @sfsr/db that gets imported into another package @sfsr/app

I tried exporting everything as usual from @sfsr/db

// db   index.ts 
import * as db from 'zapatos/db';
import type * as s from 'zapatos/schema';
import pool from './pg-pool';

export { db, s, pool };

and importing in @sfsr/app:

// app   main.ts
import { db, pool, s } from '@sfsr/db';

db and pool work as expected - but the import s (the schema) does not.

I then tried renaming all .d.ts files to .ts as suggested on other issues and that does not help - the import of s is still not resolving.

I have seen references to including a line such as /// <reference path="./zapatos/schema.d.ts" /> but I'm not sure where that would go since my @sfsr/app package has no zapatos directories.

Maybe I am going about this all wrong - but these .d.ts files have thrown me for a loop and I'm not sure where to go with it.

[sfsr12]

I dug into the issues a bit more and have a working solution that can be found here:
https://github.com/sfsr12/zapatos-example

Maybe it will be helpful to others - It has some quirks I outline in the readme - maybe someone has some input on ways to resolve them?

[jawj]

@jawj I've been expanding my use of zapatos v2 significantly throughout the projects I'm working on. It's truly fantastic. I was wondering if you could provide some feedback about a concern, however? I'm concerned about only being able to use v2 (for encapsulating the database definitions in their own module, and for accessing multiple databases), while development is progressing in v3. Do you think zapatos will stick with the limitations inherent in ambient modules for the foreseeable future?

Thanks @ellis, it's a good question. I thought the v3 approach (ambient modules) was, on balance, better than the v2 approach (copying the complete source into your project). I was keen not to need to maintain both approaches, which is why I didn't make the new approach optional in v3. However, I didn't anticipate the issues it has caused people.

If it were possible to find a third approach that fixes your issues with v3, that would be excellent (and even better if it didn't involve another painful upgrade process like v2 -> v3). Any ideas?

[ellis]

@jawj I've been thinking about this some, and I don't see a great way to have the benefits of both approaches. It seems like more extensive use of generics could allow for 1) merely copying a minimal amount code instead of the complete source code, or 2) constructing the d object from a function. But that would also lead to more complex types, which has its own set of drawbacks, and there might be problems if a person tries to use two different database libraries that were based on different versions of zapatos.

I really don't think that copying the complete source into a project is such a bad thing. Especially since the amount of code is limited. There are tons of other great projects that generate lots more code and boilerplate. Database access is one of the areas where programmers appreciate the least amount of surprise, so having the code checked into the repo also helps minimize unwanted feature changes.

[jawj]

@ellis Hmm.

I think the ambient approach is here to stay, because it wouldn't be fair to make everyone go through the v2 -> v3 migration again in reverse, and because it's nice and easy for simple use cases which I imagine (but have annoyingly little evidence!) are probably the majority.

I can see there are situations like yours where this really isn't workable, though. So perhaps, if nobody can think of a third way that marries the advantages of ambient modules and code-copying, we should consider taking a split approach, making the v2 code-copying approach available again behind a config option.

I kinda feel like we ideally need a survey of users ... (and also like this would be a nice GitHub/npm feature, or a new startup for someone — it's really hard to get a feel for how most people are using a library, since only the ones with issues file issues).

[TheNickmaster21]

I support a split approach. I currently have a project that is split between two monorepos. I currently have to make a sub project that exports the typings from the exported module to be used across the project (or I am doing something totally wrong and don't actually need to do that, but it is the only thing I found worked). I have the same setup in both repos and I just set the config to pull all schemas in the project that requires the files. I'd love to instead have a way to reference the first project so I don't need to double declare all of the common definitions between the two projects.

[ellis]

@jawj Thanks for your consideration of this use-case. Anything that could again allow for zapatos to be used from a separate library and with multiple databases would be fantastic. Seems to me like your idea of adding a config flag to support the code-copying approach of v2 is probably the best way to support this.

[TheNickmaster21]

Just an update to this, I ended up adding the zapatos/schema.d.ts to the files in our base tsconfig for the monorepo and that has worked great for us.

[puckey]

The only way I was able to get functioning shared schema types across packages in our yarn 2 monorepo was to:

• duplicate the schema.d.ts file to a schema.ts file
• remove the declare module 'zapatos/schema' { and closing } lines
• import * as schema from './schema' (and then export it again)

A bit of a hassle to have to do this every time the schema is updated.. But this library is well worth a bit of hassle – amazing work.

Update: actually above does not fully work.. The types of things like db.selectOne('thing', db.all) do not come through..

[jawj]

Thanks @puckey. It does seem there are enough users that need this to justify revisiting it. I'm not sure when I'll find time, but it's on my TODO list.

[puckey]

Thanks @jawj!

Since my earlier solution didn't work, I took another look at @sfsr12's solution and was able to simplify it a bit: sfsr12/zapatos-example#1

This has zapatos generate its type definitions in a types directory in the root. By referencing this types directory in tsconfig.json ->typeRoots of the workspaces the types are then correctly imported.

[Harrisonl]

Love the library - the only one I've actually enjoyed using with typescript. However I'm now running into this issue where we currently have 2 different db's in the same codebase (long story) - so definitely a 👍 for me - however no rush, for now I'm happy with a workaround. I haven't be able to get any working as of yet, is there a known workaround for that situation? The types work fine, but the db.selectOne("xyz",...) functions complain.

[jawj]

@Harrisonl Glad the library suits you! I think this is one of those cases where you need to either stick to v2 or wait for the v2 code-copying functionality to come back to v3+, as discussed above.

[sahil87]

Looking forward to this!
This is the one thing preventing us from moving beyond v2.

[javier-garcia-meteologica]

Why not turn it upside down and use a modular approach?

Declare a structured interface for each table and use it to call sql or shorthand functions.

export namespace my_table {
  export interface StructuredSQL {
    Table: Table
    Selectable: Selectable
    Updatable: Updatable
    Insertable: Insertable
    Whereable: Whereable
    JSONSelectable: JSONSelectable
    UniqueIndex: UniqueIndex
    Column: Column
  }
}

These interfaces are not generated inside zapatos/schema. Instead, they are generated in regular typescript files. This means that multiple DBs are supported and the schema can be exported as regular typescript definitions.

From these interfaces it's trivial to recover SQLExpresssion and SQL types.

export interface GenericStructuredSQL {
  Table: string
  Selectable: object
  Updatable: object
  Insertable: object
  Whereable: object
  JSONSelectable: object
  UniqueIndex: string
  Column: string
}

export declare type SQLExpressionForStructure <S extends GenericStructuredSQL> = GenericSQLExpression | ColumnNames<S['Updatable'] | (keyof S['Updatable'])[]> | ColumnValues<S['Updatable']> | S['Table'] | S['Whereable'] | S['Column']; 
export declare type SQLForStructure <S extends GenericStructuredSQL> = SQLExpressionForStructure<S> | Array<SQLExpressionForStructure<S>>;

Use these types with sql to get type safety.

sql<SQLForStructure<s.my_table.StructuredSQL>>(`SELECT * FROM ${'my_table'};`)

Change the signature of sql to make it even easier

export declare function sql<S = GenericStructuredSQL, RunResult = pg.QueryResult['rows'], Constraint = never>(literals: TemplateStringsArray, ...expressions: NoInfer<SQLForStructure<S>>[]): SQLFragment<RunResult, Constraint>;


db.sql<s.my_table.StructuredSQL>(`SELECT * FROM ${'my_table'};`)

For insert and update shorthand methods, it requires more work, but it's doable. It looks like this

db.sql<s.my_table.StructuredSQL>('my_table', [{
  id: '123',
  name: 'Foo'
}])

This example required the following changes in the signature of insert.

interface InsertSignatures {
  <S extends GenericStructuredSQL>(table: S['Table'], values: InsertableForStructure<S>[], options?: ReturningOptionsForStructure<S>): SQLFragment<ReturningTypeForTableForStructure<S>[]>
}

type InsertableForStructure<S extends GenericStructuredSQL> = S['Insertable'];
type ColumnForStructure<S extends GenericStructuredSQL> = S['Column'];
interface SQLFragmentOrColumnMapForStructure<S extends GenericStructuredSQL> {
    [k: string]: SQLFragment<any> | ColumnForStructure<S>;
}
type ExtrasOptionForStructure<S extends GenericStructuredSQL> = SQLFragmentOrColumnMapForStructure<S> | undefined;
type ColumnsOptionForStructure<S extends GenericStructuredSQL> = ColumnForStructure<S>[] | undefined;
interface ReturningOptionsForStructure<S extends GenericStructuredSQL> {
    returning?: ColumnsOptionForStructure<S>;
    extras?: ExtrasOptionForStructure<S>;
}
type ExtrasResultForStructure<S extends GenericStructuredSQL> = {
    [K in keyof SQLFragmentOrColumnMapForStructure<S>]: SQLFragmentOrColumnMapForStructure<S>[K] extends SQLFragment<any> ? RunResultForSQLFragment<SQLFragmentOrColumnMapForStructure<S>[K]> : SQLFragmentOrColumnMapForStructure<S>[K] extends keyof S['JSONSelectable'] ? S['JSONSelectable'][SQLFragmentOrColumnMapForStructure<S>[K]] : never;
};
type JSONOnlyColsForStructure<S extends GenericStructuredSQL, C extends any[]> = Pick<S['JSONSelectable'], C[number]>;
type ReturningTypeForTableForStructure<S extends GenericStructuredSQL> = (undefined extends ColumnsOptionForStructure<S> ? S['JSONSelectable'] : ColumnsOptionForStructure<S> extends ColumnForStructure<S>[] ? JSONOnlyColsForStructure<S, ColumnsOptionForStructure<S>> : never) & (undefined extends ExtrasOptionForStructure<S> ? {} : ExtrasOptionForStructure<S> extends SQLFragmentOrColumnMapForStructure<S> ? ExtrasResultForStructure<S> : never);

[bryanjswift]

Piling onto this discussion; I'd like to be able to use the types generated by Zapatos with a tool like nexus for generating a GraphQL schema. I don't think the particulars of my setup are important but as it stands nexus is unable to find the type definitions as a result of the ambient module approach. However, if there was a type export for the generated schema I believe this would work just fine.

Love the approach of Zapatos even if I can't use it in its current form for this particular project!

[jawj]

@bryanjswift Great — I haven't been able to find much time to work on Zapatos lately, but this is still on my TODO list.

[wanhose]

Hey @jawj,

Any updates on this? I'm having the same problem as @ellis commented above and I don't know what to do 😅

Thank you in advance for this great work!

[pocin]

not exactly sure how, but i managed to get yarn workspace with a next.js app working with the use of sfsr12/zapatos-example#1 and probably this https://github.com/belgattitude/nextjs-monorepo-example

[ellis]

I'm very excited to say that I found a workaround that allowed me to upgrade to the latest version by getting rid of the ambient module stuff. Here are the steps:

1. In zapatosconfig.json, set "outExt": ".ts" (I don't know if this is actually necessary)
2. Copy the files node_modules/zapatos/dist/db/* to $OUTDIR/zapatos/src (where $OUTDIR is the outDir value in zapatosconfig.json)
3. Replace references to 'zapatos/schema' with '../schema' in $OUTDIR/zapatos/src/*.d.ts
4. Generate the schema using the zapatos CLI
5. Patch the generated $OUTDIR/zapatos/schema.ts file as follows:
   • remove the declare module line and its starting and ending braces
   • replace zapatos/db with ./src

Below is a script that does it all in one go; it's not perfect, but it's working for my setup.

#!/usr/bin/env bash

# This script generates and copies zapatos source code, and modifies it so that it works as a normal (non-ambient) module

set -e
set -x

# Extract "outDir" from zapatosconfig.json
OUTDIR=$(jq --raw-output .outDir zapatosconfig.json)
OUTDIR=${OUTDIR:-.}
SRCDIR="$OUTDIR/zapatos/src"

# Copy files from zapatos distribution
rsync -arv node_modules/zapatos/dist/db/ $SRCDIR
# Replace references to 'zapatos/schema'
sed -i -e 's/zapatos\/schema/..\/schema/g' $SRCDIR/*.d.ts

# Generate $OUTDIR/zapatos/schema.ts
./node_modules/.bin/zapatos
# Patch schema.ts
sed -i \
"s#declare module 'zapatos/schema' {##
s#zapatos/db#./src#
s#^}\$##" \
  $OUTDIR/zapatos/schema.ts

[sahil87]

Looking forward to this!

This is the one thing preventing us from moving beyond v2.

We have finally switched to v6. The solution was to ensure only one project with Zapatos ever gets used. The introduction of schema name into Zapatos in the recent versions allowed us to do this.

So even if you have multiple dbs/schemas, the Zapatos type files for all those schemas need to created in a single repo. You can then choose downstream which schema to use in which project.