Support node-like module resolution in dev mode

[zhuhaow]

Add support for a node-like module resolution in workerd dev mode would make workerd play nicely with existing solutions.

Problem

Currently, when people use meta-frameworks such as qwik, remix, etc, they all expect to use the built-in dev server with all kinds of DX features. And most of these frameworks depend on Vite's dev server to provide most DX features.

However, Vite's dev server SSR support is not compatible with workerd (or any other edge runtimes that I'm aware of). Vite dev server expects the SSR will happen in node runtime so it loads the code "in-place" and does SSR in node runtime instead of the edge runtime. To make things faster, Vite will resolve and transform the requested modules on the fly as node imports them so things are always lazily processed. I created an issue here vitejs/vite#10770 proposing if it's possible to create a bundled script on request in the dev server.

On the other side, even if Vite does provide a bundled script, some frameworks, e.g., qwik, may still not work with workerd. This is due to in dev mode, qwik may load the component code with import. I created an issue here QwikDev/qwik#1984.

Root cause

All these incompatibilities happen because all these tools are designed without the edge runtime limitation in mind, so they do optimization by assuming everything will happen in node runtime with fs access and in the same process.

Solution

I'm wondering if it's possible on workerd side, that we can provide an interface for a special dev mode that is more compatible with node module resolution behavior, so we can hook workerd into existing tools and frameworks more easily. Because none of the issues above are trivial to fix, some parts of them even contradict the optimization they promise.

If, for example, in node we can programmatically have workerd load a script with a provided import/require, then all of these issues can be more easily fixed and we won't have to fix all existing and (probably most) new tools and frameworks by changing their design.