New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Plugin Support for Custom Transformers #14419

Open
MichaReiser opened this Issue Mar 2, 2017 · 35 comments

Comments

Projects
None yet
@MichaReiser

MichaReiser commented Mar 2, 2017

Since #13764 has landed, it's easy to write custom transformers. However, if I understand the API correctly, it is needed to duplicate the whole tsc command line just to add a single transformer. This leads to incompatibilities since these, on typescript depending, applications do not support the same features as the tsc command line tool.

It, therefore, would be favored to have a plugin system that allows loading Custom Transformers from third party node modules as this is the case for the language service proxies #12231. These transformers then easily integrate into existing build tools / build workflows.

If someone experienced has inputs on how to implement the changes, I'm willing to create a PR as it would simplify my project tremendously.

@mhegazy

This comment has been minimized.

Contributor

mhegazy commented Mar 2, 2017

We do not plan on exposing compiler plugin system in the short term. The transforms are exposed as part of the public API as you noted, and we are in the process of writing documentation and samples for these.

@MichaReiser

This comment has been minimized.

MichaReiser commented Mar 2, 2017

I don't want to have an exposed plugin API. Instead, I would prefere way to register my custom transforms by just specifying them in the tsconfig.json instead of having to use the TS-Compiler API. Because using the TS-Compiler API implies that I can no longer use the tsc command line tool which further can imply that it does no longer integrate nicely into existing build tools and workflows.

@WanderWang

This comment has been minimized.

WanderWang commented Apr 17, 2017

anything update about documentation and samples ? @mhegazy

@longlho

This comment has been minimized.

longlho commented May 3, 2017

@MichaReiser you can write a simple compiler based on the API (sample here). We actually use TS very heavily within Yahoo Finance and the transformer public API have been awesome.

Those are some of the transformers we wrote after it became public:
https://github.com/longlho/ts-transform-system-import
https://github.com/longlho/ts-transform-img
https://github.com/longlho/ts-transform-react-intl
https://github.com/longlho/ts-transform-css-modules-transform

@mhegazy lmk if you need help documenting/gathering samples

@MichaReiser

This comment has been minimized.

MichaReiser commented May 3, 2017

@longlho
Thanks for your sample.

That is what I'm currently doing. However, it makes it impossible to use other build tools created for typescript, e.g. web pack loaders, jest loaders. Besides, how can I use one of your plugins together with one of mine when each of us uses a different frontend?

Therefore, I do believe that the current approach is a good start but not sufficient for a plugin ecosystem. Because using a plugin is too much effort for the user and requires more than just writing the transformation code for the author.

I believe a plugin system like the one of Babel is essential for typescript if the goal is to encourage the community to create and use custom transformers.

@longlho

This comment has been minimized.

longlho commented May 3, 2017

@MichaReiser yup. I'm not saying it's sufficient for the ecosystem, just good enough for the 1st step towards it.

@ivogabe

This comment has been minimized.

Contributor

ivogabe commented May 9, 2017

I'd like to add support for transforms in gulp-typescript, but I want to prevent that all TypeScript plugins (for gulp, webpack etc) propose a different API. And TypeScript might add even a different way for configurating this later on. So do you currently have plans to add this in the near or far future?

@RubaXa

This comment has been minimized.

RubaXa commented Jun 6, 2017

Hi all, I try to write a preprocessor, but nothing comes out :[

In fact, I have two questions:

  • How to create AST-fragment from string?
  • How to add import?
// 1. Input
class Foo {
    templateString = 'some value';
}

// 2. After transformation
import __LIB__ from '@external/lib';

class Foo {
    templateString = (function compiledTemplate(deps) {
        // ...
        return result;
    })({lib: __LIB__});
}

// 3. Expected result
var lib_1 = require("@external/lib");
var Foo = (function () {
    function Foo() {
        this.templateString = (function compiledTemplate(deps) {
            // ...
            return result;
        })({ lib: lib_1 }); 
    }
    return Foo;
}());

A Simplified Example: https://github.com/RubaXa/typescript-api-questions/tree/master/import-add

@RubaXa

This comment has been minimized.

RubaXa commented Jun 8, 2017

⬆️ ⬆️ ⬆️
@longlho, @mhegazy Can you give any hint?

@rbuckton

This comment has been minimized.

Member

rbuckton commented Jun 8, 2017

@RubaXa Since you added the import declaration in a transform, it isn't bound or type checked. Since it wasn't bound or type checked, we cannot resolve __LIB__ in your expression to the __LIB__ in the declaration.

One option is to use a namepace import instead of a default import, so that your emit is something like:

import * as __LIB__ from "@external/lib"

As there is no aliasing that can occur.

The other hold onto a generated identifier for the import declaration, as per the attached zip

@rbuckton

This comment has been minimized.

Member

rbuckton commented Jun 8, 2017

@RubaXa if your goal is to pass in the entire module object, you probably want to use the import * as __LIB__ from "@external/lib" syntax (which does not require aliasing) and not import __LIB__ from "@external/lib" as the latter imports the default export rather than the entire module object.

@longlho

This comment has been minimized.

longlho commented Jun 9, 2017

yeah we're doing import * as foo in our CSS modules transformer as well to prevent aliasing. Although it'd be nice to tap into that to inline certain named exports

@RubaXa

This comment has been minimized.

RubaXa commented Jun 9, 2017

@rbuckton O, thanks a lot!
Still interested in how to create an arbitrary AST-fragment from string?

function createFragmentFromString(code: string) {
  // ????
}

function visitPropertyDeclaration(node) {
	if (ts.isIdentifier(node.name) && node.name.text === "templateString") {
		// ...
		return ts.updateProperty(
			node,
			ts.visitNodes(node.decorators, visitor),
			ts.visitNodes(node.modifiers, visitor),
			ts.visitNode(node.name, visitor),
			ts.visitNode(node.type, visitor),
			createFragmentFromString('(function compiledTemplate() { /* ... */ })()')
		);
	}
	return node;
}
@rightisleft

This comment has been minimized.

rightisleft commented Oct 21, 2017

This is what i was hoping the transformer support and workflow would behave like in TypeScript.

https://www.dartlang.org/tools/pub/transformers

@chandu0101

This comment has been minimized.

chandu0101 commented Dec 11, 2017

I am also very much interested in this feature.

as @MichaReiser mentioned
If someone experienced has inputs on how to implement the changes, I'm willing to create a PR as it would simplify my project tremendously.

love to see inputs from experts/typescript compiler DEV .

@MeirionHughes

This comment has been minimized.

MeirionHughes commented Jan 30, 2018

Looks like someone has wrapped typescript to expose this functionality. https://github.com/cevek/ttypescript

Also looks like ts-loader has it: https://www.npmjs.com/package/ts-loader#getcustomtransformers-----before-transformerfactory--after-transformerfactory----

@jcimoch

This comment has been minimized.

jcimoch commented Feb 1, 2018

I think this is something that could land in typescript 2.8 release or at least in roadmap at some point, @ahejlsberg @mhegazy @DanielRosenwasser what do you think? In this way custom transformers might be more popular and therefore more powerfull. Having option to plugin in transfromer from tsconfig.json perspective would simplify life a lot.

@mhegazy

This comment has been minimized.

Contributor

mhegazy commented Feb 1, 2018

We have no plans to expose plugins in the short term, as noted earlier.

@raveclassic

This comment has been minimized.

raveclassic commented Feb 1, 2018

@mhegazy Is it a considered decision or is it out of scope just because of low community interest?

@zerkalica

This comment has been minimized.

zerkalica commented May 7, 2018

Please, stop transformer plugin api fragmentation. Stabilize plugin api on top of transfomer api and expose them in tsconfig.json.

ts-loader, parcel-plugin-typescript, rollup-plugin-typescript2, ts-transformer-keys, ttypescript and others.

Each of them provides custom plugin registration method and custom plugin entry point format. It's a way to ts plugin hell.

@zerkalica

This comment has been minimized.

zerkalica commented May 13, 2018

Now cevec/ttypescript supports all common transformer plugin formats. It's a drop-in wrapper on top of all typescript modules and tsc, tsserver commands (just small runtime ts.createProgram patch). Webpack ts-loader and rollup-plugin-typescript2 can be easily configured with ttypescript.

TTypescript suggest universal transformer plugin format, but existing transformers supported too.

{
    "compilerOptions": {
        "plugins": [
            { "transform": "ts-transform-graphql-tag" },
            { "transform": "ts-transform-css-modules", "type": "config" },
            { "transform": "ts-nameof", "type": "raw", "after": true}
            { "transform": "./transformers/my-transformer.ts", "someOption1": 123, "someOption2": 321  }
        ]
    },
    "exclude": ["node_modules", "transformers/**/*"]
}
@therealmikz

This comment has been minimized.

therealmikz commented Jul 21, 2018

Any news on this one? Seems to me like TypeScript authors somehow don't want to allow custom transformers on vanilla TypeScript - that's a shame, it would be an awesome feature.

@MeirionHughes

This comment has been minimized.

MeirionHughes commented Jul 22, 2018

they don't want another api/surface to maintain for fear of leading to difficulty breaking things in the internals. Ironically, all these 3rd-party plugin solutions are, because there is no standard by the ts-team, all somewhat different and will break eventually.

@pcan

This comment has been minimized.

Contributor

pcan commented Jul 22, 2018

they don't want another api/surface to maintain for fear of leading to difficulty breaking things in the internals.

After 3 years working with typescript and watching its evolution, I have come to a simple conclusion. Microsoft has open-sourced Typescript in the sense of the code, but not in the sense of the process. In the majority of the cases, open-source projects are more or less community-driven both in the decisions and in the coding, and this should be the natural way to evolve. Do you remember the IO.js fork of node.js? The community realized that the company interests were not so aligned with the common open source governance model, and they simply forked. I hope this won't happen for TypeScript, but Microsoft should take into account this possibility.
To be clear, I'm not blaming the devs, anyway, they do a great job, really.
Just my 2c, sorry for the OT.

@kitsonk

This comment has been minimized.

Contributor

kitsonk commented Jul 22, 2018

Microsoft has open-sourced Typescript in the sense of the code, but not in the sense of the process.

I wholeheartedly disagree with this. I have never worked for Microsoft, but I have had direct influence on several features of TypeScript over the past 4 years. I was representing an open source library built in TypeScript, and we arranged to have some direct conversations, but all of the "debate" of any features was done open, in public, here. The core team publishes their design meeting notes. The only "insider" I got, representing an open source library, was an opportunity to argue my case for some features in person and get to meet the core team. It made me realise that the team acts as a team. One feature we talked about, Anders basically said the problem was too hard and that it would take a lot of time to chip away at it and solve the "real" problems people faced. That was two years ago and in TypeScript 3.0 we finally get it. But the community has a voice in the design process, but like any community, we have factured and various voices, and no team anywhere could please everyone, and if they did, TypeScript would be a really bad tool.

You also want to label the core team "Microsoft". The core team was the least Microsoft team I have ever encountered. They fought battles to get total control of their release cadence, after they were founded on full control of the their release content. They also fought for openness and transparency. The fought to move to GitHub from CodePlex. Admittedly they aren't that unique anymore in Microsoft as several other teams have adopted their model, but they were, as far as I am aware, the ones who started it all, with the VSCode team following quickly after.

So just because the core team pick and choose their battles, stick to their Design Principles doesn't mean that the community doesn't have a voice, but as a voice we suffer from psychotic multiple personality disorder. We are a difficult customer to serve.

@mrmckeb

This comment has been minimized.

mrmckeb commented Nov 22, 2018

My understanding is that this is the same as #16607, or would this achieve a different goal? As a very recent plugin-author myself, I'd also like to see this exposed - including when working with TypeScript using the Babel plugin.

@MeirionHughes

This comment has been minimized.

MeirionHughes commented Nov 22, 2018

@mrmckeb this issue is for standardizing and exposing typescript AST transformation, which already exists, while the linked issue seems to be discussing whole file transformation in a very broad (undefined) scope.

@FredyC

This comment has been minimized.

FredyC commented Dec 9, 2018

I want to add my 2 cents here. Some of you may have heard about the brilliant project babel-plugin-macros. Basically, instead of having thousands of different Babel plugins, there can be the only one which is then powered by a user code to do a transformation. An awesomely transparent system without a need to mess with configuration all the time. This is especially great for projects like CRA which supports macros but forbids additional configuration.

Today I've opened the discussion on how to possibly get this to TypeScript world. If you have some insight, please do come and share.

Ultimately, I hope that if we somehow succeed, there won't be a need for this as only one transform will be necessary.

@dsherret

This comment has been minimized.

Contributor

dsherret commented Dec 9, 2018

@FredyC that's a cool project, but we'd need this issue to be resolved in order to easily inject something like that into the compiler when only using tsc. For example, with babel-plugin-macros it still needs to be specified in .babelrc and it would be nice to specify something like that in typescript in tsconfig.json.

Actually, are you suggesting that it would be nice to have babel-plugin-macro-like behaviour built into TypeScript and no configuration would be necessary? That might be nice, but personally I would prefer a configurable solution that allows specifying custom transformers in tsconfig.json and then a version of babel-plugin-macros that supports typescript ast nodes could be specified.

@FredyC

This comment has been minimized.

FredyC commented Dec 9, 2018

@dsherret Yea, I don't expect it would be a part of TypeScript itself. However, I can imagine that once the transform is ready, it could be easy to prepare a tsc-like tiny wrapper that would be injecting only that transform for macros and rest could be done from a user code.

Edit: Actually, the tiny wrapper is already there, with above mentioned https://github.com/cevek/ttypescript it would be easy to instruct to use that along with the universal macros plugin and it's a win-win.

I just need to find people who are well versed in Typescript transforms to figure some basic prototype. I can discuss and talk, but implementing it is above my current skillset.

@pietrovismara

This comment has been minimized.

pietrovismara commented Dec 13, 2018

they don't want another api/surface to maintain for fear of leading to difficulty breaking things in the internals.

After 3 years working with typescript and watching its evolution, I have come to a simple conclusion. Microsoft has open-sourced Typescript in the sense of the code, but not in the sense of the process. In the majority of the cases, open-source projects are more or less community-driven both in the decisions and in the coding, and this should be the natural way to evolve. Do you remember the IO.js fork of node.js? The community realized that the company interests were not so aligned with the common open source governance model, and they simply forked. I hope this won't happen for TypeScript, but Microsoft should take into account this possibility.
To be clear, I'm not blaming the devs, anyway, they do a great job, really.
Just my 2c, sorry for the OT.

I agree. It is clear that despite strong interest from the community, devs are simply rejecting the idea of custom transformers/macros without actually providing a reason for the rejection except something on the lines of "People will misuse it", which also sounds offending towards the community as it implies it is a community of low quality devs.

I would accept a rejection if a reasonable justification other than "We won't simply do it" could be provided, but it seems despite all the requests from the community nobody has yet took the effort to provide that.

@mrmckeb

This comment has been minimized.

mrmckeb commented Dec 13, 2018

I think this is unfair criticism @pietrovismara. I agree that this is a must-have feature, but this isn't "Microsoft" blocking this feature. The team, as I understand it, are saying that this is not easy to support and that's why they haven't added such a feature yet.

Again, I want this feature as much as anyone else - but we need to discuss this issue using facts. The facts are that the team aren't ready to support it yet. So let's keep voicing interest, but also keep the conversation on the right track.

@pietrovismara

This comment has been minimized.

pietrovismara commented Dec 13, 2018

@mrmckeb If that's the case I can understand it. I must have missed such a declaration in the big stream of comments contained in these issues then. Didn't mean to offend anyone, just trying to provoke a clear answer from mantainers in order to, as you said, keep the conversation on the right track.

@mrmckeb

This comment has been minimized.

mrmckeb commented Dec 13, 2018

It's OK, it's frustrating that the support for this doesn't exist - as I said, I feel it, and we're asked for some kind of solution for CSS modules all the time at CRA now that we have TypeScript support. I hope it'll come soon, but I also understand that it opens up a huge area of risk for the team.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment