Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow "Compiler Plugins" #16607

Open
mohsen1 opened this issue Jun 18, 2017 · 31 comments
Open

Allow "Compiler Plugins" #16607

mohsen1 opened this issue Jun 18, 2017 · 31 comments

Comments

@mohsen1
Copy link
Contributor

@mohsen1 mohsen1 commented Jun 18, 2017

From wiki:

TypeScript Language Service Plugins ("plugins") are for changing the editing experience only

TypeScript plugins are very limited. Plugins should be able to:

  • Apply Transformers
  • Provide type definitions
  • Override module resolver (LanguageServiceHost#resolveModuleNames)
  • Emit code beside TypeScript standard output
@DanielRosenwasser
Copy link
Member

@DanielRosenwasser DanielRosenwasser commented Jun 19, 2017

@rbuckton and I have some offhand thoughts about this

  1. The scope of this is fairly broad - does this include generating new nodes for type-checking? If so, that makes this a much larger item.
  2. When it comes to module resolution, there are quite a few subtleties.
    1. Custom module resolution also means providing some special module resolution "host" which can provide this behavior in the editing scenarios.
    2. It's not clear how custom module resolution works between dependencies & dependents.

In general this isn't simple but we're open to at least hearing ideas.

@mohsen1
Copy link
Contributor Author

@mohsen1 mohsen1 commented Jun 20, 2017

I'm not familiar with TypeScript well enough to write a proposal. Instead I can list a few plugins that can be useful and exist in the wild in other forms (Webpack plugin, Babel transforms) to make a case for having such extensibility:

Custom module resolution

Code transformers

Emitting other code

  • A Swagger code generator embedded in TypeScript plugin
  • Functionality similar to Webpack: emitting image and other asset files

Providing types

  • Typed GraphQL query responses
    • This applies to pretty much any database query language. Result of a query can be typed based on query

There are so many other use-cases for compiler plugins that I'm not aware of but I'm sure compiler plugins will make TypeScript ecosystem thrive.

@phra
Copy link

@phra phra commented Jul 3, 2017

i think that this can be a very powerful feature.
in particular i'm interested in points 1, 2 and 4 at the moment.

@ryanelian
Copy link

@ryanelian ryanelian commented Jul 5, 2017

Big yes to this feature being supported by TypeScript.

I propose that the plugin system should be implemented using streaming pattern.

// NodeJS stuffs
import * as stream from 'stream';
import * as path from 'path';

export enum TypeScriptModuleEnum {
    CommonJS, AMD, System, UMD, ES6
}

export interface ITypeScriptTransform {
    (filename: string, module: TypeScriptModuleEnum): stream.Transform
}

////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
// As sample plugin to transform file content into string if matches extensions. Handy for templates.
////////////////////////////////////////////////////////////////////////////////////////////////////////////////////

// Plugin creators can use this to extend TypeScript!
import * as through2 from 'through2';

export interface IStringifyOptions {
    extensions: string[]
}

function doStringify(filename: string, extensions: string[]) {
    return extensions.includes(path.extname(filename));
}

export function StringTransform(options: IStringifyOptions): ITypeScriptTransform {
    return function (filename, module) {
        return through2(function (file, encoding, next) {

            // Determines whether we should stringify the file.
            // For example, the file name = 'test.txt' and extensions list = ['.txt', '.html']
            if (!doStringify(filename, options.extensions)) {
                return next(null, file);
            }

            let s = JSON.stringify(file);

            if (module === TypeScriptModuleEnum.CommonJS) {
                s = 'module.exports = ' + s + ';\n';
                return next(null, s);
            } else {
                return next({
                    message: 'Module not supported!'
                });
            }
        });
    };
}

////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
// Later...
////////////////////////////////////////////////////////////////////////////////////////////////////////////////////

import * as ts from 'typescript';
import { StringTransform } from 'StringTransform';

let tsString = StringTransform({
    extensions: ['.html', '.txt']
});

ts.useTransforms([tsString]);

////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
// Later in application source code...
////////////////////////////////////////////////////////////////////////////////////////////////////////////////////

import s = require('./test.txt');

This system has the following benefits:

  • The vast majority of browserify and gulp plugins (which are mostly written using through2) can be ported to TypeScript easily!
  • Transforms can be executed sequentially by TypeScript during compilation, in-memory, using the order defined in the passed transform array.
  • Allows file manipulation freedom by plugin developers.
@cameron-martin
Copy link

@cameron-martin cameron-martin commented Aug 8, 2017

Something similar to browserify transforms or webpack loaders would be very powerful and cover most of these use cases.

@phra
Copy link

@phra phra commented Aug 8, 2017

looking forward to see this feature implemented! 🙌

@data-ux
Copy link

@data-ux data-ux commented Aug 21, 2017

Currently program.emit() already has customTransformers as possible parameter, but it isn't exposed to consumers of the tsc command line program. It would be great to be able to give transformers in compilerOptions as proposed in #14419. Currently to use customTransformers you have to use the Compiler API and re-implement all the functionality in tsc like watching files etc.

@mohsen1
Copy link
Contributor Author

@mohsen1 mohsen1 commented Aug 23, 2017

Transformers do not allow custom module resolution or extra file emit

@data-ux
Copy link

@data-ux data-ux commented Aug 25, 2017

@mohsen1 Yes, you're right. I was suggesting it as an approach for your first point "Apply transformers". For a plugin to do all the things suggested in your issue description is too broad of a scope, as @DanielRosenwasser noted. I think exposing custom transformers is the highest value feature of the suggested and it's also the most simple to implement taking into account the way the TS compiler currently works.

@WiseBird
Copy link

@WiseBird WiseBird commented Sep 27, 2017

@Jack-Works Isn't Language Service doing it already?

@geovanisouza92
Copy link

@geovanisouza92 geovanisouza92 commented Dec 22, 2017

Any news on this? I'm writing a plugin, and would like to just plug it on my current setup (tsc & webpack + awesome-typescript-loader).

@fis-cz
Copy link

@fis-cz fis-cz commented May 31, 2018

I am sorry. I didn't know it is possible to write language service plugins... I should read before I write. So, for the compiler it can be implemented in the similar way as for ls.

Regarding this topic I have some tips.

It would be great if loading of plugins is configurable through the tsconfig.json file. Thats because i.e. VS code syntax highlighter / lens / intellisense will use the same plugins as the compiler during the regular build. Plugin can be standard Node module and can be resolved in the standard CommonJS way.

When the plugin for the typescript compiler will be defined in the tsconfig.json file it should be loaded during the tsc startup and tsc should provide access to all currently available tsc APIs (it would be also great if extended tsconfig can be read through API too as when I was playing around the API about year ago i had to write custom config reader / extender, what is not good as with next release of ts you can remove or add some options and its hard to maintain the code afterwards ;).

During the init phase, the plugin can replace various stages of the compiler API with custom implementations (such as file reader) or bind event listeners to events occurring during the compilation process (file load, resolve, parse, transform, emit...). Events with "pre" and "post" would be also great in order to pre-process or post-process the stage data while original components are still in use. I.e. preParse is great time to run text preprocessor which can implement #ifdefs and replace them with empty lines to keep it possible to generate source maps properly, or postParse when AST can be searched for dead code and the dead code can be removed from furthermore processing.

From my perspective, it would be much easier to implement call puigin.init(...) with references to all available tsc components and let plugin developer to choose if he will replace it or not what would be specified in the return object.
I'll update this later once I'll check tsc sources.

If this would be possible we can simply use various plugins for code preprocessing, death code elimination, output minification or whatever else we can imagine directly under the hood of the compiler "executable" but without touching the compiler code itself. Currently, we have to write everything as a new compiler using the tsc API. Unfortunately, this later means we have to implement the "new" compiler to our development tools (such as VS code or full VS, what is almost impossible ;).

@pcan
Copy link
Contributor

@pcan pcan commented May 31, 2018

Events with "pre" and "post" would be also great in order to pre-process or post-process stage data (i.e. preParse is great time to run text preprocessor which can implement #ifdefs and replace them with empty lines to keep it possible to generate source maps properly, or postParse when AST can be searched for dead code and remove it from the furthermore processing.)

I would really need this one. The ideal spot for synthetic code injection is after the parsing phase: here you have the AST ready, you can do some enhancements, and they are already available to language service!

@fis-cz
Copy link

@fis-cz fis-cz commented May 31, 2018

I would really need this one. The ideal spot for synthetic code injection is after the parsing phase: here you have the AST ready, you can do some enhancements, and they are already available to language service!

I think it is enough if you can replace the parser with custom one and inside of it you would do your pre, call original parser with modified input and your post where you would modify the AST.

@professional-human-being

Please make it happen.
There is just too much need in this.
Whenever i find myself in need of extending the typescript compiler with some feature (typed css modules, #10866) i always think "This is definetly can/must be done with a plugin". Babel has a plugin system, and people have an opportunity to do anything they need to do to get a work done, without bothering the core team with requests of new features. Besides, all burden of support lies on the plugin author. I mean that the community can add a ton of requested features to the typescript without bloating the core codebase (which will make it hard to support).

@xtuc
Copy link

@xtuc xtuc commented Oct 27, 2018

Now that Babel 7 has the support for TypeScript, would it be possible to achieve that trough Babel?

@krzkaczor
Copy link

@krzkaczor krzkaczor commented Oct 27, 2018

@xtuc i think this would break all existing tooling etc.

Personally i belive that type providers should be a way to extend typescript (only type system).

@mrmckeb
Copy link

@mrmckeb mrmckeb commented Nov 22, 2018

I've been working on this plugin, and I would love that it would work during build - and not just when working with files in the IDE.

https://github.com/mrmckeb/typescript-plugin-css-modules

@fis-cz
Copy link

@fis-cz fis-cz commented Dec 24, 2019

Just for those who need it I wrote simple AST preprocessor for conditional compilation. Before plugins are implemented it must be run as a complete separate compiler.

See #33800 for details.

@axetroy
Copy link

@axetroy axetroy commented Feb 22, 2020

Hi.

All operations of the plugin are now synchronized

I want to know if the Typescript team plans to support asynchronous operations

Eg.

interface LanguageServiceHost extends ModuleSpecifierResolutionHost {
-  getScriptFileNames(): string[]
+. getScriptFileNames(): string[] | Promise<string[]>
}

All operations of the plugin are synchronized

It is possible that some plugins will block the entire Typescript Service until make it crash

@nadavwix
Copy link

@nadavwix nadavwix commented May 13, 2020

would love to be able to create plugins that transform non typescript files into d.ts files.
these can be used while type checking and will provide a better work experience with non js/ts files.

let all our css files be typed

@glen-84
Copy link

@glen-84 glen-84 commented May 15, 2020

@DanielRosenwasser,

The TypeScript 3.8 Iteration Plan included "Plugin Investigation" under the Expected Work Items – was any progress made in that regard?

Will this issue be considered for the next roadmap (July - December 2020)?

@oychao
Copy link

@oychao oychao commented Jan 13, 2021

Here I need to transform some IDL files into type definition files, by creating a custom module loader (if available maybe)?
It's January 2021 now, any progress?

@blaumeise20
Copy link

@blaumeise20 blaumeise20 commented Apr 15, 2021

I really love TypeScript but I think that this is a feature that is really missing. I have a few thoughts about it:

Naming - How would the naming structure be like?

How would we publish plugins for others to use them? Of course everyone could use its own name but I think that is not good. Then we will see names like typescript-plugin-css or svelte-for-typescript-plugin (I'm using Svelte) on npm and that is not beautiful. There has to be something like a naming convention like the BabelJS plugins have. How would that look like? A few options:

  • typescript-plugin-xyz
  • plugin-typescript-xyz
  • xyz-plugin-typescript
  • @typescript/plugin-xyz (like BabelJS)
  • @typescript/xyz-plugin
    We would have to choose one of them or just let people choose one for their plugin, but that doesn't seem to be so great to me.

Another think also included below is local plugins (no npm modules). I don't think we should add a naming convention to them, because people have to choose themselves what they do in their projects.

Loading - How do we tell the compiler to load a specific plugin?

There is currently a plugins compiler option but it is only for IDEs. To not get confused with compiler plugins I would recommend to add a new config option called compilerPlugins. It would take an array of plugin configurations. I imagine it like this:

{
	"compilerOptions": {
		// some other options...

		// array of plugins
		"compilerPlugins": [
			// version 1 - string including the path or module name
			"@typescript/plugin-cool-features", // npm modules
			"./typescript-cool-plugin.ts", // for local ones

			// version 2 - configuration with an object
			{
				"name": "@typescript/plugin-cool-features-with-options",
				"options": { // add type checking for this
					"coolOption": "cool value"
				}
			}
		]
	}
}

Implementing - How do we write the actual plugin implementation?

Plugins should always be written in TypeScript. All functionallity should be exported in one single file, the file registered in compilerPlugins. I don't know how that should be done best, if the file should export an init function or a class. I would prefer a class. I'm unsure about the actual structure, but something like this would be cool:

import { Plugin, FileType } from "typescript/plugins";
// notice this, there should be a base class and some helpers in "typescript/plugins"

// enough to allow resolves of ".svelte" files like `import App from "./App"` with file "App.svelte"
class SvelteFileType extends FileType {
	name = ".svelte";
}

interface Config {}
export default class CoolPlugin extends Plugin {
	static info = {
		// don't know if this is needed
	}

	init(config: Config) { // can request config
		this.context // current context (can add things like providers and listeners and is used to remove all added things when disabling this plugin)

		this.context.parser.on("preparse", () => {});
		// i don't know what things should be possible here

		this.context.registerImportResolver(data => {
			return "resolvedFile.ts";
		});

		this.context.registerFileType(SvelteFileType);
	}

	remove() {
		// optional, if you have any sub-objects to destroy do this, all listeners are removed automatically
	}
}

What do you think about these ideas? If you have other ideas please tell me!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet