[Feature] Add 'run' and 'watch' modes?

[zaydek]

tl;dr: esbuild run main.ts and esbuild watch main.ts.

---

This may definitely be beyond the scope of what you’re trying to achieve, but I noticed several new tools that seem to all address the same problem space that use esbuild internally.

The idea is to add 'run' and 'watch' modes to the CLI, somewhat similar to go run in Go, where you can feed esbuild a filename and esbuild runs it as if it were a Node.js program. This would be incredibly valuable because it would make esbuild not just a bundler but also a runner / watcher, and you could write real programs that execute using Node.js as a backend (without the user being concerned about build steps / transformations, etc.).

Furthermore, you could leverage support for plugins and your watch mode implementation so that the system is inherently extensible and performant.

In some ways this addresses some of the same problems as Deno in a more backwards-compatible approach. AFAIK Deno made several breaking changes whereas esbuild seems to be more concerned with maintaining the survivability of the existing ecosystem and improving it by making JSX and TypeScript first-class primitives.

Unanswered questions in my mind are:

• How would this integrate with tools like Jest?
  • For example:
    • https://github.com/aelbore/esbuild-jest
    • https://github.com/AkifumiSato/esbuild-jest-transform
• If the interface is only the CLI, can this interact with plugins?
  • Related to Specifying plugins via the CLI #884, maybe more

Here are some examples of programs that seem to do the same thing (all using esbuild):

• https://github.com/a7ul/esbuild-node-tsc
• https://github.com/egoist/esbuild-register
• https://github.com/mhart/ts-eager
• https://github.com/folke/esbuild-runner
• https://github.com/digital-loukoum/esrun

I’m sure there’s more.

I found these lines of code quite interesting as a mechanism to invoke Node.js:

const evaluator = new Function('process', 'require', code)
process.argv = [process.argv[0], inputFile, ...args]
return evaluator(process, require)

https://github.com/digital-loukoum/esrun/blob/main/src/main.js#L63

Anyway, I’d love if I could just do esbuild run ... or esbuild watch .... I know it’s not that hard to implement in user land, but seeing as they’re are going to be dozens of tools that all try to solve the same problem with their own constraints / limitations, it may be appropriate to just add this functionality to esbuild. This would make working in Node.js so much easier because it offsets so many of the burdens that slow developers down (from being productive).

I totally understand if you don’t want to do this but I do think it’s worth proposing and gathering your feedback as well as the community generally.

[evanw]

This is interesting, but I think it's best for a project like this to be separate from esbuild. It sounds simple when you first think about it but something like this can easily get really complicated:

• People probably want it to check types too. How does it integrate with the TypeScript type checker? How does that work with type definitions from plugins?

• People coming from Deno will also want to be able to import https:// URLs. How does that work? Where are they cached, and how does the cache work?

• What about supporting other compile-to-JS languages? What about Rust? How does that integrate with the Rust toolchain?

Finally, esbuild may or may not be the best tool to build upon depending on your requirements. Unfortunately node's use of CommonJS means that require() is synchronous and node's worker_thread bugs mean that currently the only way to use esbuild in a synchronous fashion is to create a new child process every time. This is much slower than other approaches that use a native node module instead. I'm not interested in changing esbuild over to a native node module because a) this use case is already outside of the scope of being a bundler, b) I want esbuild to be independent of node instead of tied to it, and c) I still have hope that node will some day fix the bugs blocking the use of worker_thread to turn asynchronous APIs into synchronous APIs. Other tools that use native node modules may be a faster primitive to build upon, at least in the short term.

TL;DR a tool like this is likely to get pretty complicated so it doesn't seem like a simple addition to esbuild. And given that esbuild's scope as a bundler is already really large, I think it's best for a project like this to live outside of the esbuild project.

[zaydek]

This is interesting, but I think it's best for a project like this to be separate from esbuild.

I’m with you. I also had the knee-jerk reaction myself, but despite that I wanted to ask because the value proposition is quite high for me. Something like this would flip the equation and I’d be able to take Node / JavaScript more seriously as a programming environment.

I learned Go first so I have a strong inclination to types, but you can only ignore Node.js for so long. Eventually you realize it’s so pragmatic to a) be able to code server-side JavaScript and b) extend frontend knowledge base to the backend and vice-versa. I guess my point is that knowing Go is powerful, knowing Node.js is powerful, but knowing both is best. And I’ve always had a hard time with Node.js because JavaScript has never felt like a mature development environment.

My point is that it’s starting to feel like JavaScript is becoming a ‘real’ programming environment that can be taken seriously because of the advent of TypeScript and esbuild. TypeScript solves the type problem and esbuild solves the toolchain problem. With both of those problems solved, TypeScript powered by esbuild is no joke.

• People probably want it to check types too. How does it integrate with the TypeScript type checker? How does that work with type definitions from plugins?
• People coming from Deno will also want to be able to import https:// URLs. How does that work? Where are they cached, and how does the cache work?

I really appreciate you enumerating your concerns. These are my personal expectations:

• I’d like to be able to run / watch JS / JSX / TS / TSX without having to think about Babel / JSX transformations or TypeScript type-checking
• I don’t need esbuild to do any type checking for me. I see type checking purely as a DX concept seeing as JavaScript is not suited for static type checking (I know ts-node exists / works but I can tell you it feels like it’s doing ‘too much’ work and startup times are slower than expected, even with the -T transpilation-only flag)
• ESM feels like a solvable problem by specifying --esm. Then you can enforce ESM or Node-based module resolution throughout
• If there’s a caching step involved, I see no problem with creating and deleting temporary hidden directories on demand, seeing as that’s a cross-platform idiom

• What about supporting other compile-to-JS languages? What about Rust? How does that integrate with the Rust toolchain?

That’s a fair point but for me is beyond the scope of what makes sense / interests me. The idea here is to make JavaScript / TypeScript run and watch without build steps / build artifacts / etc., which are in turn destroying developer productivity. That may be an exaggeration but not a misplaced one.

Finally, esbuild may or may not be the best tool to build upon depending on your requirements. Unfortunately node's use of CommonJS means that require() is synchronous and node's worker_thread bugs mean that currently the only way to use esbuild in a synchronous fashion is to create a new child process every time. …

I’ve actually never worked with worker threads before (though I’ve heard of them) and this is a fair criticism of why this idea may not be a valid candidate, but I also think most devs in user land haven’t actually programmed worker threads before. I could be wrong, but I think the 90% case actually doesn’t involve worker threads.

TL;DR a tool like this is likely to get pretty complicated so it doesn't seem like a simple addition to esbuild. And given that esbuild's scope as a bundler is already really large, I think it's best for a project like this to live outside of the esbuild project.

I have no doubt about that but it is encouraging to see people bootstrap this idea to better or worse effect in several dozen lines of code (some of the OP linked repos are quite small). It would be nice if esbuild supported this workflow because it’s not a stretch of the imagination that your users want to run their bundled / unbundled code without build steps.

---

I absolutely understand the reluctancy but I want to at least plant the idea as a seed (for you and or myself) because I’m really excited about programming in rapid-development JavaScript / TypeScript environment that has almost as superior tooling as Go. esbuild is yours and I don’t want to push you to take it some direction you don’t want to go, but as a user I’d love to provide perspectives that interest and may be of value to you.

Edit: Btw I noticed you pushed 0.8.54 but I don’t see any release notes just yet. You may very well be typing them now. 🙂

[evanw]

Edit: Btw I noticed you pushed 0.8.54 but I don’t see any release notes just yet. You may very well be typing them now. 🙂

The source of truth for release notes is the change log, so you can read the release notes here: https://github.com/evanw/esbuild/blob/v0.8.54/CHANGELOG.md#0854. Someone added a GitHub action that normally copies the release notes to a GitHub release, but it seems like GitHub actions are broken at the moment.

[zaydek]

@evanw I wanted to follow-up; I have this little experiment called esnode based on this idea. So far it’s working for me and it makes working in Node.js very refreshing. Cheers!

https://github.com/zaydek/esnode

Some implementation details you may find interesting:

I simply configure esbuild and write .outfile.esbuild.js to disk in the current working directory and always unsync the build artifact whether and error occurred or the run was successful. All package.json dependencies are passed to the external field so that only user files are bundled. This also means that if I write to a temporary directory outside of the current working directory then things with node_modules get weird, which is why I simply emit a build artifact locally and clean up after. I tried doing everything in memory but I had trouble getting source-map-support to work this fine, but that’s not a big deal.

Anyway, this is very exciting for me. Getting esbuild to work for this use-case was easier than I expected and your V8 parsing code makes debugging more approachable. I haven’t been able to fully reproduce all of the subtleties of how esnode logging works, so I added support for STACK_TRACE=true to simply concatenate the whole stack trace when desired, but this is off by default.

Anyway, thanks for all of your work. I’m finding esbuild to be an incredibly pluggable and flexible system.

@heyheyhello you may find this interesting so tagging you. 🙂