A minimal custom transformer plugin proposal

[jakebailey]

In the TypeScript compiler, we have the concept of a "transformer". A transformer is a function which "transforms" one AST to another. TypeScript ships many such transforms, including all of the various transformation steps needed to downlevel newer syntax to older syntax, to convert from import/export to require/module.exports, and so on. Even the declaration emitter is a transformer, stripping away function bodies, filing in types, etc.

Since TypeScript 2.3, TypeScript has had the following API:

interface Program {
    emit(
        targetSourceFile?: SourceFile,
        writeFile?: WriteFileCallback,
        cancellationToken?: CancellationToken,
        emitOnlyDtsFiles?: boolean,
        customTransformers?: CustomTransformers, // <--- 👀
    ): EmitResult;
}

interface CustomTransformers {
    before?: (TransformerFactory<SourceFile> | CustomTransformerFactory)[];
    after?: (TransformerFactory<SourceFile> | CustomTransformerFactory)[];
    afterDeclarations?: (TransformerFactory<Bundle | SourceFile> | CustomTransformerFactory)[];
}

This API allows users to provide a set of "custom transformers" at emit time. These transformers run alongside TypeScript's own transformers, either before or after, depending on where they are placed in CustomTransformers.

However, while this API exists and does work, a major gotcha is that custom transformers can only be provided to the API. There is no way to use custom transformers if you want to use tsc alone.

This has been a long-standing issue; #14419 is the fourth most upvoted issue on the repo.

What do people use custom transformers for?

This list can go on and on, but transformers are commonly used these days for:

• Generating serialization / deserialization / runtime type checks
  • typia,ts-runtime-checks, which top the benchmark charts, but there are many more
• Powering internationalization
  • formatjs
• Generating database code
  • ts-graphql-plugin
• Transforming paths (please don't do this one, though 😅)

I also know of many ideas which want to be implemented via plugins, but are not willing to do so without official support, including other runtime type checks, tracing code, and so on.

How do users currently use custom transformers?

Given the current constraints of custom transformers, there are two ways that downstream users have been able to use them:

1. Build TypeScript code via the API, either directly or by using some system which uses the API.
   • webpack, rollup's TS plugins, nx, all provide configuration which allows users to specify a set of custom transformers.
2. Patch TypeScript.
   • ttypescript, ts-patch wholly replace TypeScript in a user's workflow.

The first one is obviously fine. The latter, eek!

What changed?

In TypeScript 5.0, we shipped the biggest architectural / packaging change in TypeScript's history; conversion of the entire project to modules, including the bundling of our code. The relevant side-effects are twofold:

1. The structure of tsc.js (and other outputs) changed massively. This broke anyone who was patching the contents of the TypeScript package.
2. The API is no longer runtime patchable; we use esbuild to bundle our code and the objects it creates are not "configurable", correctly emulating how "real" ES modules would behave. Any runtime patches were already extremely fragile, only functioning due to the structure of our old namespace emit.

In response to this, I did an analysis of as many TS "patchers" as I could find, and found that there are actually very few reasons to patch TypeScript at all:

• ttypescript/ts-patch, which seem to be almost exclusively used to enable the use of custom transformers.
• Language service plugins, mainly injecting their own module resolution and custom source file types (vue/volar, css, etc).
• Build systems like heft.
• Yarn PnP.
• Those who want to tree shake out our parser (prettier).

For 5.0, I was able to fix many of these projects (or at least guide / warn maintainers ahead of time), but many of these patchers have no viable alternative.

We have been seriously considering ways that we can approach the problems that remain in hopes that we can eliminate the need for people to patch TypeScript. This includes (no promises):

• Custom transformers.
• Custom module resolution.
• A future API that actually allows a consumer to tree shake out just a parser (or similar).

This issue intends to address the first bullet.

A conservative, targeted proposal

In short, my proposal is to add to TypeScript the ability to define "custom transformer plugins". This new functionality is targeted and minimal. Its only intent is to satisfy those who just want to be able to add custom transformers.

Many details still have yet to be expanded on, but in short:

• These plugins are placed into the plugins array of compilerOptions, with a discriminator like "type": "transformer".
  • This is the approach taken by ttypescript/ts-patch, and offers up a future base for additional tsc plugins (e.g. file watching plugins, module resolution plugins).
  • The choice in discriminator ("type": "transformer", for now) is intended to avoid conflicting with existing plugins defined by ttypescript or ts-patch.
• Plugins would take a form similar to language service plugin, receiving the ts object, returning a factory that can be used to create a CustomTransformers object. The factory would receive the Program, which is required for any type-using plugins.
  • The ts object must be a part of the API for similar reasons to the language service plugins; the APIs available within the core compiler are very, very limited, and the bundle is not the same as typescript.js.
  • The plugin object entry is passed to the plugin for extra configuration, like LS plugins.
• A new tsclibrary.d.ts file is added, describing the limited API within tsc. This is similar to tsserverlibrary.d.ts.
• When invoking tsc, you must pass --allowPlugins to opt into executing plugins. When using the API or tsserver, plugins are enabled by default.
  • No sandbox is present otherwise. If you currently use a patching-based method to use plugins, you're already taking matters into your own hand. If you currently use webpack or rollup, your config files are executable anyway.
  • --allowPlugins is also a part of the watch plugin PR.
• If multiple plugins are used, their returned CustomTransformers are merged.

There are almost assuredly other things that people want to be able to do with custom transformers or other plugins, however, I believe that we can serve the vast majority of those who want custom transformers with this API, and I do not want to let perfect be the enemy of the good.

In the future, we can expand plugins to do more, allowing the customization of other aspects of TypeScript, or simply add on to what custom transformer plugins can do. Prior research has shown that there are more interested parties, which I believe that we can eventually support.

An example

With the above proposal, a tsconfig.json would look like this:

{
    "compilerOptions": {
        "plugins": [
            { "type": "transformer", "path": "@jakebailey/cool-transformer" }
        ]
    }
}

The transformer would look something like:

import type * as ts from "typescript/tsclibrary";

const factory: CustomTransformersModuleFactory = ({ typescript }) => {
    return {
        create: ({ program }) => {
            return {
                before: [(context) => (file) => {/* ... */}],
            };
        },
    };
};

export = factory;

Of course, the details may change.

Unsolved problems

• It's very unfortunate to have to define yet another random d.ts API used for one thing.
  • Can we do better?
  • What if the executables were ESM?
• Like LS plugins, tsc plugins would need to be CJS.
  • Can we instead make use of import?
  • How do we do async? (This may already be "solved" in a prior WIP for plugins)
• Should we do any sort of performance metric counting for plugins?
  • Like LS plugins, can we just say "no warranty" when plugins are involved?
• How much of a problem is it going to be to need to use the typescript namespace passed into the plugin?
  • What happens to existing transformers which mistakenly import typescript.js's API directly, even though that may not have always worked?
  • Will plugins unknowingly use the typescript.js API and rely on it, assuming users won't use tsc?
  • Should we try and attempt converting our binaries/libraries to ESM first?
• Is the proposed API too factory-y?
• Should we even do this?

Please give feedback!

If you are currently using transformers, creating transformers, or otherwise, I would appreciate your feedback.

[jakebailey]

Preemptive @nonara ping; I suspect you have feedback! 😄

[timocov]

Hey @jakebailey,

Thank you for this proposal! I personally like the approach (it feels similar to ttypescript), but I have a question - it seems it is not possible to provide options to a "transformer" from the config, right? Would it be possible to allow to define something like options/params/config next to type and path fields that the compiler will pass as-is (without any transformation/validation) to the transformer function?

Quick intro where I came from

I'm the author of 2 optimization-related transformers:

• https://github.com/timocov/ts-transformer-minify-privates
• https://github.com/timocov/ts-transformer-properties-rename

They both allow you to minify properties of objects (either private or internal) and it is achieved by adding a prefix to "all properties that should be minified" so you can use these prefixes in tools like terser or uglifyjs to minify i.e. mangle these properties. While the default prefixes (_private_ and _internal_) are used in majority cases, there are few cases when people prefer to use different prefixes for whatever reason (e.g. to sync values in their config files). Thus it would be quite beneficial to have such option.

Thanks!

[jakebailey]

Would it be possible to allow to define something like options/params/config next to type and path fields that the compiler will pass as-is (without any transformation/validation) to the transformer function?

Oops, I should have explicitly defined that; yes, any other options are passed along to create, just like language service plugins.

export type CustomTransformersModuleFactory = (mod: { typescript: typeof ts }) => CustomTransformersModule;

export interface CustomTransformersModuleWithName {
    name: string;
    module: CustomTransformersModule;
}

export interface CustomTransformersModule {
    create(createInfo: CustomTransformersCreateInfo): CustomTransformers;
}

export interface CustomTransformersCreateInfo {
    program: Program;
    config: any;
}

[patdx]

Why wouldn’t it support esm? Is it because the initializing code of tsc is synchronous? It may be nice if it could support esm from the start.

[jakebailey]

Why wouldn’t it support esm? Is it because the initializing code of tsc is synchronous? It may be nice if it could support esm from the start.

Yes, correct, in its current form. The same currently applies to tsserver plugins.

We'd need dynamic import and that doesn't fit nicely into our current architecture. But, I need to go check what was done in the previous attempt as I believe it may have been solved there.

[samchon]

It seems great. TS 5.2 update is expected.

[samchon]

This is just my opinion - @jakebailey

Support both TransformerFactory and CustomTransformerFactory

Looking at changed codes, I think it may possible to support both TransformerFactory and CustomTransformerFactory. There're some useful libraries using ttypescript, but some of them had stopped maintaining. I think supporting both interfaces maybe helpful for them.

--allowPlugins to be default true or just remove it

To use transformation based libraries like my typia (or nestia), users must configure tsconfig.json file with plugin property. Configuring the plugin property by themselves mean that they have already agreed to using plugin transformation. In my opinion, typing --allowPlugins CLI is just annoying thing for users.

I hope it to be default true, or just remove it.

[jakebailey]

Looking at changed codes, I think it may possible to support both TransformerFactory and CustomTransformerFactory. There're some useful libraries using ttypescript, but some of them had stopped maintaining. I think supporting both interfaces maybe helpful for them.

Those transforms are unlikely to work because they import typescript directly. The APIs available within tsc are very limited, no methods on Node, Type, etc, and it's very possible for a user to accidentally get into a setup where a plugin may import a different version of typescript than is loaded. These plugins need to be loaded in the language service as well, which for most users is guaranteed to not be TypeScript within the workspace, but instead be the one included in VS Code and therefore cause two different versions of TypeScript to be active in memory and potentially have mismatching behavior.

To use transformation based libraries like my typia (or nestia), users must configure tsconfig.json file with plugin property. Configuring the plugin property by themselves mean that they have already agreed to using plugin transformation. In my opinion, typing --allowPlugins CLI is just annoying thing for users.

This comes down to the security tradeoffs. In the API, you're already running code, so the flag isn't needed. In the language service, we already load plugins automatically and in VS Code require that you "trust" the workspace before starting tsserver. There is no such equivalent for tsc and without --allowPlugins, this would be the first instance of tsc arbitrarily executing code.

---

I should also note that it's possible that this proposal is not implemented at all. There are still strong arguments against doing this sort of thing whatsoever. The use cases may not outweigh the downsides; it may also be the case that this feature would be avoided by most because this means they cannot use any transpilers except TypeScript itself.

[canonic-epicure]

--allowPlugins to be default true or just remove it

To use transformation based libraries like my typia (or nestia), users must configure tsconfig.json file with plugin property. Configuring the plugin property by themselves mean that they have already agreed to using plugin transformation. In my opinion, typing --allowPlugins CLI is just annoying thing for users.

I hope it to be default true, or just remove it.

+1 for this, --allowPlugins is clearly an overkill. Its not a nuclear rocket launch, where you have to use 2 different keys from 2 persons, rotate them simultaneously etc. If user has specified the transformer in the "tsconfig.json" I believe his/her intention is clear enough.

[sosoba]

ttypescript passed another useful argument to the plugin. Method addDiagnostic which allows plugins to report errors and warnings to a place in the code in a standard way:

    helpers?: { ts: typeof ts; addDiagnostic: (diag: ts.Diagnostic) => void }

It seems to me that this feature should be in CustomTransformersCreateInfo too.

[amanteaux]

Another use case for custom transformer is to provide dependency injection at compile time. This is what is partially done in https://github.com/wessberg/DI-compiler

[43081j]

I've actually pushed for this many times over the years as I've written a fair few transforms, most of which are for codegen stuff (e.g. replacing decorators with equivalent code).

However, I think i'd vote against it these days.

If we open the flood gates here, we're opening ourselves up to inconsistent behaviour and syntax across any two typescript projects - which seems high risk to me.

The usual culprits that already implement a lot of "magic" (frameworks, codegen projects, gql, etc) i'm sure would make use of it and build their own "flavour" of typescript soon enough. Which would lead to fragmentation of an otherwise pretty concrete ecosystem/community.

On top of that, typescript was always meant to be "javascript with types" (i.e. just some annotations we can strip away). That kind of goes away if we open up to custom syntax.

I'd love to be able to simplify some of my builds by this being possible, but i would rather put up with needing my own build script to avoid the above.

anyone who really wants to transform the AST can still throw a small build script together. but that slightly higher bar is why we still have a consistent syntax overall.

[jakebailey]

I don't understand the above; clearly all type information is available within tsc, it's just not as convenient as how it is within the wider services API. If we were to do this, we would of course want to ensure that things are accessible; that is the point after all. I mentioned some of this preciously.

I'll also say that I don't understand how what you're saying is accurate when the patching libs still operate within the same bounds as tsc already has. How are they getting the methods, when such methods don't even exist within tsc?

[samchon]

@jakebailey This is a list how TS 5.2 transform API breaks transformer libraries:

1. As ts.Node.getSourceFile() and ts.SourceFile.getLineAndCharacterOfPosition() be erased, it became impossible to show exact (3rd party library defined) compile error position
2. As ts.Node.getSourceFile() be erased, it became impossible to support special transformation function in 3rd party library.
3. As ts.Node.getSourceFile() be erased, it became impossible to analyze import relationship in 3rd party library.
4. As ts.Type.isTypeParameter() be eerased, it became impossible to find or filter out generic argument.
5. As ts.Type.isLiteral() and ts.LiteralType.value be erased, unable to analyze literal type.
6. As ts.Type.isClass() be erased, unable to check class type.
7. As ts.Declaration.getChildren() and ts.Type.get(Apparent)Properties() be erased, unable to list up properties.
8. As ts.JsDocTagInfo, ts.Symbol.getDocumentationComment(), and ts.Symbol.getCommentTags() be erased, unable to analyze comments and comment tags.
9. As ts.Type.isUnionType() and ts.Type.isIntersection() be erased, unable to analyze union type.
10. As ts.Type.getFlags() be erased, unable to analyze exact value type like number.
11. As ts.Type.getSymbol() be erased, unable to get exact ts.Symbol value.
12. As ts.Type.isTupleType() be erased, unable to analyze tuple type.
13. As ts.Type.isArrayType(), ts.Type.isArrayLikeType() and ts.Type.getNumberIndexType() be erased, unable to analyze array type.
14. As ts.Node.getFullText() be erased, unable to get full name.

Also, you're saying that all type informations are available in typescript/lib/tsclibrary, and additional properties and methods in typescript is just for convenience. However, note that, 3rd party transformer libraries had used only typescript for many years, not typescript/lib/tsclibrary. It is not just a problem of convenience, but 3rd party transformer library developers don't know typescript/lib/tsclibrary at all.

Furthermore, as you're a member of TypeScript compiler developer, you can feel that everything is possible in typescript/lib/tsclibrary, and typescript is just for convenience. However, you know what? TypeScript had not provided any detailed documents about compiler API. Therefore, third-party developers just looked at the names of objects, methods, and properties of APIs that existed in typescript, and learned how to use them by randomly inferring and experimenting, especially depending on TypeScript AST viewer. Even typescript was terribly difficult for me, because everything was black-box.

In such circumstance, you're just going to adjust breaking change that 3rd party libraries's developers never had experienced. I don't know how much performance benefit would be in the typescript/lib/tsclibrary, but I hope you to respect legacy 3rd party libraries had utilized ttypescript and ts-patch. We had utilized typescript instead of typescript/lib/tsclibrary too much time.

• Summary
  • TypeScript had not supported any detailed documents about compiler API
  • For many years, 3rd party libraries have utilized not typescript/lib/tsclibrary but typescript
  • Therefore, TS 5.2 transform API would become break change, even without any detailed documents
    • You're saying that everything is possible in typescript/lib/tsclibrary
    • And typescript is just for convenience
    • However, even typescript was terribly difficult for me, due to blackbox development
  • Hope to consider such 3rd party circumstances, and please just provide typescript transform API

p.s.) Just providing more convenient methods and properties, it really affects to the performance? I can't understand it.

[jakebailey]

re: #54276 (comment)

1. Use ts.getSourceFileOfNode and ts.getLineAndCharacterOfPosition.
2. Use ts.getSourceFileOfNode.
3. Use ts.getSourceFileOfNode.
4. Check type.flags & TypeFlags.TypeParameter.
5. Check type.flags & (TypeFlags.StringLiteral | TypeFlags.NumberLiteral | TypeFlags.BigIntLiteral). LiteralValue's value property is public.
6. This can be checked via objectFlags. We could export a helper for this.
7. I'm not convinced getChildren should be used in a transform at all. Our transforms do not use this helper. You probably want to be using explicit visit functions. For the properties, call the checker, which has getPropertiesOfType and getAugmentedPropertiesOfType.
8. I don't think a transform should need getDocumentationComment. That's for the language service for displaying in tooltips and such. There are other ways to obtain JSDoc info, too.
9. Check type.flags & TypeFlags.Union or type.flags & TypeFlags.Intersection.
10. Just access type.flags.
11. Just access type.symbol.
12. Call typeChecker.isTupleType.
13. Call typeChecker.isArrayType, typeChecker.isArrayLikeType, or typeChecker.getIndexTypeOfType.
14. Call ts.getSourceFileOfNode and slice the text by node.pos and node.end.

When you operate within the core compiler, you have to play by the core compiler's rules. These are all things that we do on a regular basis. If there is an API missing, then we can add it.

But pushing FUD on multiple issue trackers is really not helping, in fact, it's mostly convincing me that I shouldn't have made this proposal at all.

[samchon]

Okay, I will argue only in here issue. Sorry for multiple articles' comments

[tobias-tengler]

@tobias-tengler All this proposal is for is to directly expose a feature that already exists. You can write a TS transformer right now to test whether it can do what you want. See the section "How do users currently use custom transformers?" in the issue. There's at least 5 ways to do it! See this handbook as well

I don't think I fully understand everything in your example, but I don't think it can achieve what you want. The TS compiler phases are, in order: parse, bind, check, transform, emit. Type checking has already finished before the code is transformed, and it seems like you'd want that to be the other way around. Also yes, it looks like you'd need an IDE plugin.

Thank your for your answer and input :)

Very contrived I basically want to be able to extend TypeScript in a way that would allow to dynamically look up a type in another file to use as return type of a function, based on the inputs of the function call. Everything from the type checking to the IDE experience should just behave like I would've annotated the function by hand.
I think there are many other cases where something like this would be useful.
Do you know of a proposal for a feature or even an existing feature like this?

[lorenzodallavecchia]

Very contrived I basically want to be able to extend TypeScript in a way that would allow to dynamically look up a type in another file to use as return type of a function, [...]

A possible problem here is that TypeScript would not know about that additional file and, under incremental compilation, may not recompile the main file if you change the imported one. Am I correct?

[matthew-dean]

It would be great if this allowed the use of ts-macros without requiring the use of ts-patch!

[rbalicki2]

I would like to add an additional use case here. I'm the author of Isograph (https://github.com/isographlabs/isograph). One workflow is:

• write iso literal (thing graphql literal)
• compiler generates files
• babel transforms that iso literal into a require(path_to_generated_file) statement

This is great, and it works, but the types are lost, and users are required to import the generated file for types to work:

// "any" types inferred
const {data} = useLazyReference(iso`Query.home_page`, {})

// proper types inferred
const HomePageEntrypoint = import '../__isograph/Query/home_page/entrypoint'
const {data} = useLazyReference<typeof HomePageEntrypoint>(iso`...`, {})

In this workflows (and others in Isograph), the ability to have typescript understand the results of the babel transform would substantially improve the DevEx!

EDIT: TBH I'm unsure whether @michaelboyles comment indicates that this is currently possible. Looking into it! But feel free to let me know either way ;)

[TylerOliverMCG]

Hi all! I was looking into this issue because it's common practice with vite to use @/ as an alias to the src folder. However, as far as I can tell, doing this in typescript when building for a library leads to broken paths in the emitted files. It looks like this issue would result in the ability to include a transformer to handle the above file path resolution.

If this issue is still active, what can I do to help it continue forward? The thread states the goal was 5.2.0, but we're well past that release and this pr is still WIP. Very interested in resolving this issue, or any solutions that don't involve wrapping Typescript to resolve a path alias :)

[jakebailey]

Path aliases are not recommended to be used in that way (in fact, we now explicitly say not to do so in the docs). package.json import aliases fit that need but do work at runtime, though I'm not quite sure why you need the code to work at runtime if you're using vite (a bundler).

I would not pursue this issue as a solution to path aliases being emitted as-is.

[TylerOliverMCG]

Don't want to derail this thread, but I'm building a Vue Component library and then looking to emit typing for non-component entities (composables, utilities, etc). Those utilities reference shared type information from our types folder, and we're using a path alias to reduce the overhead. Runtime works great (Storybook resolves all the aliasing), and the local builds look good. When we build the library, the emitted type files still contain the unresolved paths (@/types, etc).

Is the intended workflow then that we just never use a path alias in Typescript for libraries? I'm re-reading this documentation but struggling to understand what's incorrect about what I'm doing in concept. Or is there some documentation I'm missing about how to sync up my tsconfig with the vite bundler so that vite resolves the aliasing before type information is built.

Thanks so much for responding, even knowing that I'm on the wrong path is super helpful :)

[jakebailey]

If you're using a bundler, or those paths actually do exist at runtime, then that can be fine; see https://www.typescriptlang.org/tsconfig#paths.

But, for shortening import specifiers, import maps do the same thing without needing to "lie" to TypeScript: https://nodejs.org/api/packages.html#subpath-imports

[michaelboyles]

@M-jerez I don't see how that's productive or on-topic for this issue. The TS team have already said they won't do that: #3628 (comment)

Transformers have already been implemented. The decision is purely whether to expose them, and how. The decision is not to bin them in favour of a feature that's already been declined.

People have already implemented reflection, and arguably the most relevant use-case for reflection, runtime validation... They're both implemented using custom transformers. Resolving this issue would make it easier to use both of those libs.

[marcj]

Transformers opens the gates to getting potatoes instead javascript as output. Nowadays they are rarely used

@M-jerez transformers are very commonly used, they are all over the place, even without people knowing: React with JSX, Angular with their decorators that emit runtime data for dependency injection, NestJS with legacy decorators, all these down-level transformers to support new JS features, and many more. Transformers are the backbone of modern web development and probably 99% of all TS rely on them, most unknowingly. Making them available in a more straightforward "official" way would only be a natural next step.

[sorenbs]

Prisma is a Typescript ORM with 3M monthly downloads. We have long wanted to implement LINQ-style queries, and this feature would enable us to deliver that feature with excellent DX. We'd love to prototype something on a pre-release version if that is helpful.

[samchon]

Prisma is a Typescript ORM with 3M monthly downloads. We have long wanted to implement LINQ-style queries, and this feature would enable us to deliver that feature with excellent DX. We'd love to prototype something on a pre-release version if that is helpful.

https://github.com/samchon/typia/blob/f1688280397fbc1fdda9a52f7056a519ba537ab2/package.json#L117

@sorenbs Above URL address may work. By the way, prisma is planning support safe raw query expression validated by the compiler?

[sorenbs]

Thanks @samchon!

By the way, prisma is planning support safe raw query expression validated by the compiler?

Yes - that's what we would like to do. If we can do it in a way that delivers excellent DX. We do not want to push specific bundlers or something like ts-patch on our users.

[alshdavid]

It would be great to also have support for transformers that could run before the core TypeScript ones do - enabling preprocessor steps (like macros).

Would the LSP consider transformer plugins or would there be an alternate pathway for adding support for custom syntax?