Multiple files support

[fabien0102]

Feature description

So far, we are just supporting types declared in one file. The plan will be to follow any import statement.

To solve different use cases, we should provide different resolving options:

• "flat" -> all the dependencies are flatten in one output file (perfect for Mark types for schema generation #30 usecase)
• "multi" -> we have a 1-1 mapping between (perfect for Circular Dependencies Error when there are none #70 usecase)

Input

// bar.ts
import { Foo } from "./foo"

export type Bar {
  foo: Foo
}

// foo.ts
export type Foo = number

Output (flat)

import { z } from "zod"

export const fooSchema = z.number()
export const barSchema = z.object({foo: fooSchema})

Output (multi)

// bar.zod.ts
import { z } from "zod"
import { fooSchema } from "./foo"

export const barSchema = z.object({foo: fooSchema})

// foo.zod.ts
import { z } from "zod"
export const fooSchema = z.number()

Related issues

#30 #70

[then3rdman]

Wondering if any work has been undertaken on this? Would potentially be be happy to spend sopme time on it.

[nlundquist]

This is extremely important because it currently limits the use of ts-to-zod to only the simplest files.

There's no way for users to produce a single file containing all the types you'd want to pass to ts-to-zod because the rollup functionality of api-extractor produces a .d.ts file. When tying to use a d.ts ts-to-zod silently fails, producing bad output by incorrectly including the declare keyword in the output.

[fracalo]

Are there any updates on multifile support? I may be capable to develop this feature, but would like to discuss it first.
@fabien0102 Do you have any ideas on how this should be done?

[fabien0102]

I started something on my last plane trip but needed more time to make something usable.
It’s actually not an easy problem, the main challenge is to compute the dependencies tree.

I’m also not totally sure what to do if you have some types picked from node_modules for example. But keeping this edge case aside, the idea is:

1. Transforms all the import type statements to generated schema imports
2. List all the files used by the entry point, so we can add them to a generation queue
3. Make sure the type validator works (nice to have, but still)
4. Enjoy! 🎉

[mattfysh]

I had a quick look at the pull request and it looks like you're heading down the "build a bundler" route. Can I suggest to keep things simple that you keep ts-to-zod as a compiler only? This is how typescript compiler itself works, and does not provide bundling. It works because the output directory has a mirrored structure of the input directory.

Also not sure if it's helpful, but this is the script I've built using a glob pattern to produce a generated dir. I am temporarily replacing any imported identifiers with a string literal (a uuid) and after ts-to-zod has done it's things, I go back through and replace the z.literal("<uuid>") call expressions with the original identifier

https://gist.github.com/mattfysh/6a071d5b2150bdb6fb10be5be715402c

[fracalo]

I had a quick look at the pull request and it looks like you're heading down the "build a bundler" route. Can I suggest to keep things simple that you keep ts-to-zod as a compiler only? This is how typescript compiler itself works, and does not provide bundling. It works because the output directory has a mirrored structure of the input directory.

Actually most of the work in this PR is to retrieve the relevant types from the import clauses and aggregate the schema in order not to modify the return of the generate function.
So the result of having all the schemas bundled in one file is more a convenience decision rather than an architectural one.
IMO, for a first iteration, this could be good enough.

I do agree with you that splitting the schemas up in multiple file would be a nice option to offer in the future.
To do so we would probably need to generate a dependency graph associating the paths to the relative schemas, coordinate the imports between files, and then dump everything to disk (adjusting integrations test, etc.).
It is certainly not as easy/linear as writing everything to a single file.

Also not sure if it's helpful, but this is the script I've built using a glob pattern to produce a generated dir. I am temporarily replacing any imported identifiers with a string literal (a uuid) and after ts-to-zod has done it's things, I go back through and replace the z.literal("<uuid>") call expressions with the original identifier

https://gist.github.com/mattfysh/6a071d5b2150bdb6fb10be5be715402c

Thanks for sharing, your script is interesting but it would need some polishing to be used in production.

[anthony-dandrea]

Any update here? I saw a few PRs(#148, #135, #118) that appear to be related to this issue. This feature is really the one big thing holding us back from adopting ts-to-zod(and zod).

Great library by the way. Thanks for your work!

[fabien0102]

Quite a busy life those days, I will try to have a look when this is getting calmer. If one of those PR are working for you, please try them on your projects (the easiest is to clone the project, add your name as prefix in the package name and publish to npm 😉 )

Every feedbacks are appreciate :) Thanks for the support 😃

[tvillaren]

For some reasons, I had missed this issue so started working on 2 successives PRs:

• Add import handling for external classes #135 supports any import statement and will handle it as an third party import, outputting the same import statement in the generated file + a const importSchema = z.instanceof(importType) statement
• Handling imports of generated types #148 will add support for the multi case mentioned in this issue, but will not do it systematically for all imports, only for those referenced in the ts-to-zod.config.js file. I think it's a non-breaking good first step to handle zod file imports: one can decide to add it or not to the config, it's less intrusive than a global holistic approach.

[tvillaren]

Import support has been added to ts-to-zod in 3.7 (and fixed in 3.7.3 😅).
Right now, it relies on having all the files in the configuration files and generation must be ordered "manually".

[efstajas]

Thanks for building this great package!

In our project, we automatically generate types for GQL queries adjacent to their definitions in __generated__/gql.generated.ts files. Now, I'm looking for a way to take these generated files, and additionally generate a __generated__/schemas.generated.ts file for each generated types file, containing generated zod types based on the existing TS types.

That way, we always have a zod schema available to parse GQL responses with — which we need specifically because we sometimes cache query results on redis, and don't necessarily want to blindly trust that the cached data has the required shape at runtime.

As-is, I understand that the only way to do this with this package would be to dynamically generate an ephemeral config that lists all the generated files explicitly, and discarding it after the zod types were generated. That's a bit convoluted. I was hoping that I could just configure the package to run for all files matching a glob, and an output filename relative to the matched input file's location.

Does this match your vision?

[schiller-manuel]

glob sounds like a nice idea

[tvillaren]

My vision for the next step (because that would ease my work 😅) was to generate for all files in a given directory (or set of files), with ts-to-zod being able to work out the right generation order based on a toposort of import dependencies.

I was hoping that I could just configure the package to run for all files matching a glob, and an output filename relative to the matched input file's location.

That would be an interesting next step: it would work out-of-the-box only for "independent" files. If the files matching the glob reference each other in a way, then validation would fail.
It would look something like ts-to-zod __generated__/*.generated.ts outputFolder/ ?

[efstajas]

It would look something like ts-to-zod generated/*.generated.ts outputFolder/ ?

In our case, the files are spread all over the repo, because they're generated adjacent to the query definitions. So the glob would be something like src/**/__generated__/gql.generated.ts, and the output files would have to be named sth like schemas.generated.ts and placed next to the respective matched file.

If the files matching the glob reference each other in a way, then validation would fail.

Again in our case, the files don't reference each other, but they do reference a "global types file" which has the basic GQL types in it, like scalars etc. I'm not sure how unique this setup is, but it's the result of using graphql-codegen with the near-file-preset preset, which seems pretty popular.

The graphql codegen config in question is here

... as I'm writing this, I'm starting to realize that having a graphql-codegen plugin that uses this package to transform types to schemas would probably be the neatest solution, and would offload all the complex glob matching logic to graphql-codegen itself. For our specific usecase at least.