Support the esbuild plug-in system？

[ycjcl868]

esbuild is small but complete in every detail, do you have any plans to support the plugin-in system to extend the web development workflows?

[evanw]

Not right now. I may figure out an extensibility story within esbuild after the project matures, but I may also keep esbuild as a relatively lean bundler with only a certain set of built-in features depending on how the project evolves.

The use case of using esbuild as a library was one I hadn't originally considered. It's interesting to see it start to take off and I want to see where it goes. It could be that most users end up using other bundlers and esbuild is just an implementation detail that brings better performance to those bundlers.

I'm still thinking about how I might add extensibility to esbuild in the back of my mind. Obviously it's made more complicated by the fact that it's written in Go. It would be possible to "shell out" to other processes to delegate the task of transforming input files, but that would almost surely be a huge slowdown because JavaScript process startup overhead costs are really high.

One idea I've been thinking about is to have esbuild start up a set number of JavaScript processes (possibly just one) and then stream commands to it over stdin/stdout. That could potentially amortize some of the JavaScript startup cost (loading and JITing packages from disk). It would be more of a pain to debug and might still be surprisingly slow due to all of the serialization overhead and the single-threaded nature of the JavaScript event loop.

Another idea is to turn the esbuild repository into a Go-based "build your own bundler" kit. Then you could write plugins in Go to keep your extensions high-performance. The drawback is that you'd have to build your own bundler, but luckily Go compiles quickly and makes cross-platform builds trivial. That would likely require me to freeze a lot of the Go APIs that are now internal-only, which would prevent me from making major improvements. So this definitely isn't going to happen in the short term since esbuild is still early and under heavy development.

[rsms]

Finding this interesting!

TL;DR: The subprocess approach sounds like a solid idea.

Some thoughts in regards to the conversation above

It would be possible to "shell out" to other processes to delegate the task of transforming input files, but that would almost surely be a huge slowdown because JavaScript process startup overhead costs are really high.

This is a really interesting approach I think, although not the most portable (i.e. running systems like iOS that do not support subprocess creation.) I'd think about the quality-performance problem here in the same way as with Figma plugins:

• Making it clear to the user that they are using "extra things"
• Warnings when things seems to be slow or hung (like how a web browser running a locked-up javascript might ask the user "Foo is taking a long time to run. Do you want to wait for Foo or kill it?".
• Allowing async operation (not just execution) so that things that don't need to be serialized isn't. For example: a process that rsync's changes to a remote server. This program would want to be triggered on a build but does not directly depend on the build nor the build depending on it.
• Processes are widely understood, supported and have a robust execution model.

One idea I've been thinking about is to have esbuild start up a set number of JavaScript processes (possibly just one) and then stream commands to it over stdin/stdout.

In my experience what makes nodejs programs slow is I/O rather than CPU. For example, loading gatsby.js causes an incredible amount of files and directories to be read etc. TypeScript is an example of a good player making the best they can, avoiding runtime imports, but starting tsc is still painfully slow (thus their daemon/server model, which is a solution just like the "subprocess" idea you have!)

Another idea is to turn the esbuild repository into a Go-based "build your own bundler" kit.

Perhaps a nice option for people who are comfortable with Go. Might "pair well" with a subprocess approach for "putting lego blocks together" vs "build your own lego blocks".
Ideas about stuff to consider:

• Maintenance cost of an API that can't change much. I.e. the API provided for "build your own...". Perhaps making it extremely minimal with just two-three functions for pre- and post-processing with a string file list would get you most of the upsides with a low API maintenance cost?
• How to manage a future fragmented landscape of "many different esbuild versions out there." For example, imagine someone effectively forking esbuild and adding a substantial and desired addition like OCaml transpilation. Ideally they'd just gomod import esbuild but for some reason they had to fork it. Now you end up with users of that version filing issues and so on.

Some thoughts on Go plugins

I've worked with plugins in go in the past and it's a little bit complicated (for good reasons.) Since Go is statically compiled and doesn't have a fully dynamic runtime like for example JavaScript it is a little tricky to load code at runtime and really hard to unload code (replace/update.)

Some things to keep in mind:

• Go's plugin package provides this functionality
• Plugins must be built by the same Go compiler as the program that loads them
• Plugins are compiled with copies of code that it makes use of (i.e. imports.) I think these must be the same versions of any packages linked in the main program and in other plugins, but I could be wrong.
• Plugins can't be unloaded in any meaningful way (since Go uses fairly complex memory management.) However I've had success with adding some structure to what plugins must do, with things like an unload function it must provide and use to zero any references to code outside itself.
• Loading plugins is fast since you just load the code into memory and then "call it".

Some example code from GHP:

Loading plugins:
https://github.com/rsms/ghp/blob/8ab5e52dded3ad7443849bf7a51a82e8e7ee2de2/ghp/servlet.go#L65-L71

Add structure to plugins to allow unloading them (without actually unloading their code.)
https://github.com/rsms/ghp/blob/8ab5e52dded3ad7443849bf7a51a82e8e7ee2de2/ghp/servlet.go#L107-L119

A plugin: (called "servlet" in this project)
https://github.com/rsms/ghp/blob/8ab5e52dded3ad7443849bf7a51a82e8e7ee2de2/example/pub/servlet/servlet.go

[zmitry]

What about something like this https://github.com/hashicorp/go-plugin. Use RPC based plugin system so anyone could write plugin in any language.

[evanw]

@rsms thanks for writing up your thoughts. It was very interesting to read through them, and helpful to learn from your experience. I haven't seen the plugin package used before. I hadn't thought of the Go compiler version problem but that makes them much less appealing.

Maintenance cost of an API that can't change much. I.e. the API provided for "build your own...". Perhaps making it extremely minimal with just two-three functions for pre- and post-processing with a string file list would get you most of the upsides with a low API maintenance cost?

Yes, I was thinking of something extremely minimal. Of course that would come at the cost of performance, which isn't great. I'm not sure if there's a great solution to this.

What about something like this https://github.com/hashicorp/go-plugin. Use RPC based plugin system so anyone could write plugin in any language.

I think something like this is promising. This is basically how esbuild's current API works, except over stdin/stdout. The advantage of this over shelling out is that it lets you amortize the startup overhead of node by keeping it running during the build. You could imagine a more complex API where esbuild has hooks for various points and could call out to node and block that goroutine on the reply. That would let you, say, run the CoffeeScript compiler before esbuild processes the source code of a file.

I plan to explore this direction once esbuild is more feature-complete. My research direction is going to be "assuming you have to use a JavaScript plugin, how fast can you make it". If we can figure that out then that's probably the most helpful form of API for the web development community.

[floydspace]

I would like to vote for this feature.

Currently, I'm migrating some small project from webpack to esbuild and it has graphql schema defined in a file .graphql which is bundled using graphql-tag/loader, of course, I can just change schema definition using the different approach. But It would be nice to have this capability to write esbuild loader without raising a PR in your repository for every single case.

Thank you

[chowey]

If you do implement a plugin system, please consider making it Go-based (or better yet, cross-language as per @zmitry's suggestion).

I think there is a very big opportunity for non-JS-based tooling to transpile JS. As you state in your readme:

I'm hoping that this project serves as an "existence proof" that our JavaScript tooling can be much, much faster.

[zmitry]

I guess for first iteration it would be nice to have just golang api for plugins. I guess if you can run some arbitrary code in golang you could spawn sub process in any language. So even simple golang api would be enough.
I like rpc based approach because it would allow to do plugins hot swap or remote plugins and it's much cleaner approach.