Output multiple files when bundle: false

[jridgewell]

TLDR: When using the JS API and setting bundle: false and write: false, it'd be nice for the result to include multiple output files for each file in the module tree.

This is a bit specific, but one of my biggest goals for my test suite treat each file as an independent module and not bundle them into a large single build file. AMP's current test suite generates a file at about 60mb, and trying to debug that in Chrome is a nuisance (extremely slow to load files in sources pane, and sourcemaps make it even slower). Having each file be independent an module served by the test server would eliminate this pain, and would make incremental building of the test suite extremely fast (just update a relatively small file, instead of concatting the 60mb file together again).

This would be really simple if everything just used ESM. But we still require CJS node modules, and so we still need some sort of transpilation layer to build. Additionally, we use Babel to do a few custom things, so we're definitely stuck with running some transpiler.

Currently, setting bundle: false means that each entry point is considered an independent module, which is good. But it doesn't allow us to generate a metadata for the full dependency tree, and it doesn't continue non-bundling any dependencies of the entry point. What I would like to happen is for esbuild to traverse down the tree and generate output files for each dependency (don't bundle them into a single output). This would allow us to transpile the CJS into ESM using esbuild, and hook into the plugin API to feed into babel for our custom transforms.

[evanw]

Have you considered doing this with an esbuild plugin and bundle: true? The plugin could mark all imported paths as external but have the side effect of kicking off additional builds (as long as the path hasn't been seen before). That should let you experiment with doing this.

However, esbuild currently doesn't convert CommonJS require() calls to ESM because there is no 1:1 mapping. ESM doesn't have synchronous dynamic imports. Sometimes it might make sense to import using an import statement and other times it might make sense to import using an import() expression, but AFAIK it's not generally possible to automatically convert CommonJS into ESM and preserve the semantics of the code. Can you describe what transform you are expecting?

The only exception to this that I can think of is if you converted every CommonJS module into an ES module that exports the CommonJS wrapper closure but without executing it, and then also converted every import of a CommonJS module in an ES module into code that automatically calls that CommonJS wrapper. But this is not a transform that any tool currently does as far as I'm aware, so it would be incompatible with other stuff? It also assumes that you never import ESM from CommonJS which some libraries in the wild actually do, so that still doesn't work.

I think the full solution would involve converting all ESM code into CommonJS to ensure that module instantiation could be code-directed, then convert all code (which is now CommonJS) back into ESM using lazily-evaluated CommonJS wrappers. This is essentially doing most of what it would take to turn esbuild into a single-file-at-a-time dev server, which is basically what you're asking for. Having esbuild do something like this is currently out of scope for now, and is not a priority. I think it makes the most sense for esbuild to focus on being a good bundler at the moment.

However, I think what you're looking for already exists! I believe tools like Vite and Snowpack do this, and from what I hear they do this very well. People seem generally satisfied with them. And they also implement hot-module reloading which you might also be looking for. Since these already exist and have a lot of effort going into them, I expect people looking for features like this to use those tools instead of using esbuild directly. These tools actually both still use esbuild under the hood and are supposed to be quite fast.

[jridgewell]

Have you considered doing this with an esbuild plugin and bundle: true? ... That should let you experiment with doing this.

What I think you're saying here is that I would have an onResolve that returns { ... external: true } (for the non-entry points). This will call the resolver for each import, but the external will prevent them from being added to the bundle. Because it's external, it'll won't process that file anymore, and I'll need to start another build call to process that file as the new entry-point, and do the same.

That's workable, but a bit more involved. It requires direct access to the build process, which means it'll need to be specifically designed to handle this recursive transforming. The testing plugin is meant to allow the user to configure esbuild (including their own plugins, but there'll always be a root file which imports their entrypoints). I was hoping that we could just detect that the result contains X output files (and they already contain the path!), so we could immediately know that the user intends for a un-bundled output.

However, esbuild currently doesn't convert CommonJS require() calls to ESM because there is no 1:1 mapping. ... Can you describe what transform you are expecting?
The only exception to this that I can think of is if you converted every CommonJS module into an ES module that exports the CommonJS wrapper closure but without executing it, and then also converted every import of a CommonJS module in an ES module into code that automatically calls that CommonJS wrapper.

Sorry, I'm jumbled a few points. I don't expect esbuild to transform ESM to CJS.

We currently patch the files during compilation to convert simple exports.foo = ... into export const foo = ..., or just wrap the entire contents with a export default function() { /* contents */ } to delay the evaluation. This is very specific to my project, and not something that esbuild would have to handle. So your assumption was exactly right.

But this was just a smaller detail of the initial feature request, I mainly wanted to focus on the ability to traverse the entire tree and generate some output.

I believe tools like Vite and Snowpack do this, and from what I hear they do this very well. People seem generally satisfied with them. And they also implement hot-module reloading which you might also be looking for. Since these already exist and have a lot of effort going into them, I expect people looking for features like this to use those tools instead of using esbuild directly. These tools actually both still use esbuild under the hood and are supposed to be quite fast.

I agree, but both of these require you build for their serving framework, so they're higher level designs and the cost of switching to them is much higher. The bundler is lower level, and mostly interchangeable in our build process (we've already gone browserify to esbuild, and we've sampled rollup/webpack, too). So I was looking to see if esbuild could support something similar to this, without having to fully adopt a different serving framework.

[jridgewell]

Another thought I've had since opening this: Supporting a 1 for 1 input->output file module in build mode, instead of just transform mode, could allow for tree-shaking. Because we'd be aware of the full module graph, we could apply tree-shaking to each file to strip out unnecessary bits.

[Venryx]

A similar request was made for Webpack (>50 upvotes): webpack/webpack#5866

One of the webpack developers responded later with "Yes, it will be great feature, we will consider it after the stable webpack 5 release", but so far no progress has been made on it in webpack itself.

That said, apparently someone made a plugin which approximates the desired behavior (each input file is processed by the webpack plugin chain, but then is written to its own output file rather than placed into a bundle): https://github.com/DrewML/webpack-emit-all-plugin

I'm considering attempting to write a similar plugin for esbuild, but I'm not sure if that's even possible, due to less of the build process being accessible from JavaScript plugins.

@evanw Do you know if the approach used by the plugin above is possible for esbuild's plugin system as well?

The plugin's full source-code is here, for reference (56 lines): https://github.com/DrewML/webpack-emit-all-plugin/blob/master/index.js

The key question, I guess, is whether there's something equivalent to the afterCompile hook in Webpack, as that is how the plugin "intercepts" the processed contents of each module prior to bundling.

[graup]

I think this is a duplicate of #708