RFC: Refactoring transformers and Zod 3

[colinhacks]

TLDR

Transformers are poorly designed. I want to reimplement them but there will be breaking changes. Under this proposal, Zod 2 will never come out of beta and we'll jump straight to Zod 3 with a better implementation of transformers. Skip to "New Implementation" for details.

The reason Zod 2 has been in beta for so long (well, one of the reasons anyway!) is that I’ve been increasingly unsatisfied with Zod’s implementation of transformers. Multiple issues have cropped up that have led me to re-evaluate the current approach. At best, transformers are a huge footgun and at worst they’re fundamentally flawed.

Context

Previously (in v1) the ZodType base class tracked the inferred type in a generic parameter called Type. But transformers accept an Input of one type and return an Output of a different type. To account for this, Type was renamed to Output and a third generic property was added (Input). For non-transformers, Input and Output are the same. For transformers only, these types are different.

Let's look at stringToNumber as an example:

const stringToNumber = z.transformer(z.string(), z.number(), val => parseFloat(val));

For stringToNumber, Input is string and Output is number. Makes sense.

What happens when you pass a value into stringToNumber.parse?

1. The user passes a value into stringToNumber.parse
2. The transformer passes this value through the parse function of its input schema (z.string()). If there are parsing errors, it throws a ZodError
3. The transformer takes the output from that and passes it into the transformation function (val => parseFloat(val))
4. The transformer takes the output of the transformation and validates it against the output schema (z.number())
5. The result of that call is returned

Here's the takeaway: for a generic transformer z.transformer(A, B, func), where A and B are Zod schemas, the argument of func should be the Output type of A and the return type is the Input type of B. This lets you do things like this:

const stringToNumber = z.transformer(z.string(), z.number(), val => parseFloat(val));
const numberToBoolean = z.transformer(z.number(), z.boolean(), val => val > 25);
const stringToNumberToBoolean = z.transformer(stringToNumber, numberToBoolean, val => 5 * val);

The problems with transformers

After implementing transformers, I realized transformers could be used to implement another much-requested feature: default values. Consider this:

const stringWithDefault = z.transformer(z.string().optional(), z.string(), val => val || "trout");
stringWithDefault.parse("marlin") // => "marlin"
stringWithDefault.parse(undefined) // => "trout"

Voila, a schema that accepts string | undefined and returns string, substituting the default "trout" if the input is ever undefined.

So I implemented the .default(val:T) method in the ZodType base class as below (partially simplified)

default(def: Input) {
  return ZodTransformer.create(this.optional(), this, (x: any) => {
    return x === undefined ? def : x;
  });
}

Do you see the problem with that? I didn’t. Neither did anyone who read through the Transformers RFC which I left open for comment for a couple months before starting on implementation.

Basically this implementation doesn’t work at all when you use it on transformers.

Side note: should the .default method for stringToNumber accept a number or a string? As implemented above it should accept a string (the Input). But already this is unintuitive to many people.

stringToNumber.default("3.14")

// EQUIVALENT TO
const defaultedStringToNumber = z.transformer(
  stringToNumber.optional(),
  stringToNumber,
  val => val !== undefined ? val : "3.14"
)

defaultedStringToNumber.parse("5")
/* { ZodError: [
  {
    "code": "invalid_type",
    "expected": "string",
    "received": "number",
    "path": [],
    "message": "Expected string, received number"
  }
] */

Let’s walk through why this fails. The input ("5") is first passed into the transformer input (stringToNumber.optional()). This converts the string "5" to the number 5. This is then passed into the transformation function. But wait: val is now number | undefined, but the transformer function needs to return a string. Otherwise, if we pass 5 into stringToNumber.parse it’ll throw. So we need to convert 5 back to "5". That may seem easy in this toy example but it’s not possible in the general case. Zod can’t know how to magically undo the transformation function.

In practice, the current definition of default in ZodType shouldn’t have even been possible. The only reason the type checker didn’t catch this bug is because there are a regrettable number of anys floating around in Zod. It’s not a simple matter to switch them all to unknowns either; I’ve had to use any in several instance to get type inference and certain generic methods to work properly. I’ve tried multiple times to reduce the number of anys but I’ve never managed to crack it.

It’s possible this is a one-off issue. I could find some other way to implement .default() that doesn’t involve transformers. Unfortunately this isn’t even the only problem in Zod’s implementation.

The .transform method

Initially the only way to define transformers was with z.transformer(A, B, func). Eventually I implemented a utility function you can use like this:

z.string().transform(z.number(), val=>parseFloat(val));

 // equivalent to 
z.transformer(z.string(), z.number(), val=>parseFloat(val));

Some users were executing multiple transforms in sequence without changing the actual data type:

z.string()
  .transform(z.string(), (val) => val.toLowerCase())
  .transform(z.string(), (val) => val.trim())
  .transform(z.string(), (val) => val.replace(" ", "_"));

To reduce the boilerplate here, it was recommended that I overload the method definition to support this syntax:

z.string()
  .transform((val) => val.toLowerCase())
  .transform((val) => val.trim())
  .transform((val) => val.replace(" ", "_"));

If the first argument is a function instead of a Zod schema, Zod should assume that the transformation doesn’t transform the type. In other words, z.string().transform((val) => val.trim()) should be equivalent to z.string().transform(z.string(), (val) => val.trim()). Makes sense.

Consider using this method on a transformer:

stringToNumber.transform(/* transformation_func */);

What type signature do you expect for transformation_func?

Most would probably expect (arg: number)=>number. Some would expect (arg: string)=>string. Neither of those are right; it’s (arg: number)=>string. The transformation function expects an input of number (the Output of stringToNumber) and a return type of number (the Input of stringToNumber). This type signature is a natural consequence of a series of logical design decisions, but the end result is dumb. Intuitively, you should be able to append .transform(val => val) to any schema. Unfortunately due to how transformers are implemented, that's not always possible.

More complicated examples

The fact that I incorrectly implemented both .transform and .default isn't even the problem. The problem is that transformers make it difficult to write any generic functions on top of Zod (of which .transform and .default are two examples). Others have encountered similar issues. #199 and #213. are more complicated examples of how the current design of transformers makes it difficult to write any generic functions on top of Zod. Nested transformers in particular are a minefield.

A path forward

When I set out to implement transformers I felt strongly that each transformer should have a strongly defined input and output transformer. This led to me implementing transformers as a separate subclass of ZodType (ZodTransformer) in an attempt to make transformers compose nicely with other schemas. This is the root of the issues I’ve laid out above.

Instead I think Zod should adopt a new approach. For the sake of differentiation I’ll use a new term "mods" instead of "transformations". Each Zod schema has a list of post-parse modification functions (analogous to Yup’s transform chain). When a value is passed into .parse, Zod will type check the value, then pass it through the mod chain.

const schema = z.string()
  .mod(val => val.length)
  .mod(val => val > 100);

type In = z.input<typeof schema> // string
type Out = z.input<typeof schema> // boolean

Unlike before, Zod doesn’t validate the data type between each modification. We’re relying on the TypeScript engine to infer the correct type based on the function definitions. In this sense, Zod is behaving just like I intended; it’s acting as a reliable source of type safety that lets you confidently implement the rest of your application logic — including mods. Re-validating the type between each modification is overkill; TypeScript’s type checker already does that.

Each schema will still have an Input (the inferred type of the schema) and an Output (the output type of the last mod in the mod chain). But because we’re avoiding the weird hierarchy of ZodTransformers everything behaves in a much more intuitive way.

One interesting ramification is that you could interleave mods and refinements. Zod could keep track of the order each mod/refinement was added and execute them all in sequence:

const schema = z.string()
  .mod(val => parseFloat(val))
  .refine(val => val > 25, { message: "Too short" })
  .mod(val => `${val}`)
  .refine(val => /^\d+$/.test(val), { message: "No floats allowed" });

Compatibility

I was using the "mod" terminology above to avoid confusion with the earlier discussion. In reality I would implement the "mod chain" concept using the existing syntax/methods: .default, .transform, etc. In fact I think I could switch over to the "mod" approach without any major API changes.

• A.transform(func): instead of returning a ZodTransformer, this method would simply append func to the "mod chain"
• A.transform(B, func): this would return A.transform(func).refine(val => B.safeParse(val).success)
• z.transformer(A, B, func): this could just return A.transform(func).refine(val => B.safeParse(val).success)
• A.default(defaultValue): this is trickier but still possible. This function would instantiate A.optional().mod(val => typeof val !== "undefined" ? val : defaultValue). Then all the mods of A would be transferred over to the newly created schema

Under the hood things would be working very differently but most projects could upgrade painlessly unless they explicitly use the ZodTransformer class in some way (which isn’t common).

I would still consider this to be a breaking change of course. If/when I make these changes, I plan to publish them as Zod v3. In this scenario Zod 2 would never leave beta, we’d jump straight to v3.

This transformers issue has caused me a lot of grief and headaches but I’m confident in the new direction; in fact I already have most of it working. I want to put this out for feedback from the community. The issues I’m describing are pretty subtle and not many people have run into them, but I believe the current implementation is currently untenable.

[o-alexandrov]

The only regret I have about zod is:

• type-related methods in classes, for example of z.number()'s:
  • min, max, etc.

The problem with them is:

• if unused, they cannot be removed as part of the dead code removal (for example w/ terser) because it's possible class methods are dynamically requested
  • btw, that was one of the reasons why react's team claimed classes are complicated for minifiers

To shake off a lot of dead code in production, zod could:

• remove all type-related methods like min, max etc
• export them as ready-to-use either transform's or refine's functions
  • you could still protect the user from unwanted use of min for example for z.object(), by assigning proper typings, for example:
    • z.ZodTransformer<z.ZodNumber, z.ZodNumber>

[colinhacks]

That should be it's own issue, it's not really related to this discussion

[o-alexandrov]

Sorry, I thought it could be part of the transformers change.
I'll keep an eye on this issue then and will try to express the thoughts above clearer once this issue is merged

[colinhacks]

This is just a proposal at this stage, there may be a good reason not to do this that I'm not aware of.

I understand what you're proposing above and I'm against switching everything to functions, mostly because you lose autocomplete. I don't want to degrade the developer experience for everyone to appease a tiny minority who care about a couple extra kb in their bundle.

[colinhacks]

@o-alexandrov I just published a v3 branch with the new transformers implementation. There aren't many changes outside of ZodTransformer. I'd like to publish the v3 beta once we have ESM working.

[carlpaten]

To provide a data point: when I saw that Zod v2 had transformers I expected them to work pretty much exactly how you describe here, and I was confused when they didn't.

This proposal sounds grand to me. The only thing I'd like to flag is that it sounds like you're going to be implementing multiple transforms as lists of functions. The alternative would be to use function composition, which I suspect would be simpler and more performant. Have you considered that approach?

[jstewmon]

👋 Hi, was trying out the proposed interface in 3.0.0-alpha.2, and found a couple of issues using a simple string to Date transform...

A.transform(B, func): this would return A.transform(func).refine(val => B.safeParse(val).success)

The latter works, but the former results in an error:

> z.string().transform(d => new Date(d)).refine(d => z.date().safeParse(d).success).parse('2020-12-30T22:01:06.839Z')
2020-12-30T22:01:06.839Z

> z.string().transform(z.date(), d => new Date(d)).parse('2020-12-30T22:01:06.839Z')
Uncaught TypeError: effect.mod is not a function
    at ZodTransformer.parse (/home/node/app/node_modules/.pnpm/zod@3.0.0-alpha.2_@types+node@14.14.17/node_modules/zod/lib/cjs/parser.js:860:41)

z.transformer(A, B, func): this could just return A.transform(func).refine(val => B.safeParse(val).success)

The former returns the result of A.parse:

> z.transformer(z.string(), z.date(), d => new Date(d)).parse('2020-12-30T22:01:06.839Z')
'2020-12-30T22:01:06.839Z'

[colinhacks]

Sorry I haven't implemented the A.transform(B, func) and z.transformer syntax yet. I think I might just get rid of those syntax options entirely after all, they're too confusing.

[jstewmon]

The refine interface is a bit awkward to work with following a transformation because there doesn't seem to be a way to leverage the fidelity of schemas thereafter.

Here's an example of a transform for parsing duration strings:

const duration = z
  .string()
  .transform(ms)
  .refine((val) => z.number().safeParse(val).success, {
    message: "invalid duration",
  });

The first thing I noticed is that I couldn't figure out how to use the existing options from z.number(). Without providing options, the ZodError will have { code: 'custom', message: 'Invalid value.'}

If I want to further refine the schema (e.g. range constraints), I can't use .min(x).max(y) - I have to wrap each one with refine and provide the options.

Would it be possible to pass a ZodType instance to refine, so that the return value of refine could be a ZodType that can be leveraged so that we can continue chaining refinements?

For example:

const duration = z
  .string()
  .transform(ms)
  .refine((z.number());

const durationRange = duration.positive().max(30_000);

[colinhacks]

It's an interesting idea but I think it would get confusing fast. durationRange would be a ZodNumber instance that only accepts strings...which I don't like.

Also you can use the ZodNumber methods like this:

const duration = z
  .string()
  .transform(ms)
  .refine((val) => z.number().positive().max(30_000).safeParse(val).success, {
    message: "invalid duration",
  });

It's not quite what you're looking for but it makes everything more explicit.

To make this less verbose I could introduce a .refineWithSchema method that you use like this:

const duration = z
  .string()
  .transform(ms)
  .refineWithSchema(z.number().positive().max(30_000));

It's basically syntactic sugar over your .safeParse(val).success approach except it would automatically catch any ZodError thrown during parsing and return that, instead of throwing the generic { code: 'custom', message: 'Invalid value.'} error.

[jstewmon]

durationRange would be a ZodNumber instance that only accepts strings...which I don't like.

This seems like an incidental detail to me. Is the instance type significant to calling code?

I have been reasoning about a schema as something that takes an input and returns a result of some type iff the input meets all the validation criteria. Knowing the type of the input isn't necessarily useful, since often comes from deserializing an input.

Having read back over the examples in the proposal, I think the problem isn't with the interface / behavior of transform, though there is probably room for improvement.

IIUC, the existing transform method properly supports chaining / pipelining / composition of schemas, but the current proposal does not offer a substitute.

Let's break down the original problem case:

const stringToNumber = z.transformer(
  z.string(),
  z.number(),
  val => parseFloat(val)
);

stringToNumber.default("3.14")

const defaultedStringToNumber = z.transformer(
  stringToNumber.optional(),
  stringToNumber,
  val => val !== undefined ? val : "3.14"
)

stringToNumber is a parse pipeline that takes a string and returns a number because z.string() is the head and z.number() is the tail.

I actually would have expected stringToNumber.default("3.14") to produce a TS error b/c the parameter type should be number. This seems like an opportunity for improvement.

Still, if we recognize that transform is roughly equivalent to pipe, we can reason that specifying a default must be done before a value is required, which will typically be at the head of the pipeline:

z
  .string()
  .default("3.14")
  .transform(stringToNumber, val => val)
  .parse(undefined); // 3.14

We were able to reuse our stringToNumber schema through composition with transform.

If parsing a value is expensive, so we'd prefer to provide a default for the tail instead of the head, we can do that with union and transform:

const shortCircuit = z.union([
  z.literal(undefined).transform(z.number(), x => 3.14),
  stringToNumber
]);
shortCircuit.parse(undefined); // 3.14
shortCircuit.parse("10"); // 10

The composition facilities of transform create a lot of leverage that is missing from the proposal.

Going through these examples, the only significant fault I found with transform is that the return type doesn't match the tail schema.

I think having an accurate transform return type and clarifying docs would be sufficient to avoid confusion while retaining the power of the existing functionality.

[jstewmon]

Going through these examples, the only significant fault I found with transform is that the return type doesn't match the tail schema.

Er... I forgot to try my original use case:

stringToNumber.positive().max(5).parse("10")
// Property 'positive' does not exist on type 'ZodTransformer<ZodString, ZodNumber>'

That doesn't work, presumably for the same reason why default doesn't have the signature I expected.

I actually would have expected stringToNumber.default("3.14") to produce a TS error b/c the parameter type should be number

Correction, I think the type default parameter type should be never, since z.number() cannot use a default value.

[carlpaten]

[Content status: strong opinions held weakly; not passing value judgments, just offering suggestions]

I have been reasoning about a schema as something that takes an input and returns a result of some type iff the input meets all the validation criteria. Knowing the type of the input isn't necessarily useful, since often comes from deserializing an input.

To concur: IMO, a parsing library is (morally) a toolbox for building functions of type A => B | ZodError, with the contents of ZodError reflecting A's structure. In a perfect world you'd want the library to permit composition of A => B | ZodError with B => C | ZodError (I believe this is called a "functorial" interface in functional programming). The hard part is keeping ZodError legible under a variety of potential combinations, and indeed that severely restricts the space of practical solutions.

If I understand correctly, @colinhacks proposes factoring A => B | ZodError into A => A | ZodError and A => B. @jstewmon would like to be able to compose this with a B => B | ZodError and possibly a B => C | ZodError. The thing is that unless you can somehow provide a reverse mapping from downstream validation errors to the original value, this will net you utterly incomprehensible error messages!

The question of how to reverse-map downstream errors into a form that reflects the original input is an interesting one, but I don't think it should be a blocker for work on transformers. Just surfacing "validate, then transform" solves a ton of real-world use cases. We can talk about performing validation on the transformed values in a later proposal - IMO in the vast majority of cases it's going to be an anti-pattern because it will completely wreck your error messages and perform likely unnecessary work on invalid inputs.