Feature: Workerized JS Module Script Loading

[DerekNonGeneric]

The future (and hopefully very near future) way to achieve performative module script loading will likely come in the form of implementing “worklet infrastructure” in core since it does not seem like a userland solution should be reasonably expected.

Worklets are a piece of specification infrastructure which can be used for running scripts independent of the main JavaScript execution environment, while not requiring any particular implementation model.
— https://html.spec.whatwg.org/multipage/worklets.html

This issue should help us track progress in this area, which I find to be of great interest.

[guybedford]

This might also be beneficial to syncify some loader hooks. For example it is seeming more and more likely Chromium wants to ship a synchronous import.meta.resolve while Node.js has an experimental async version to support the async resolver. Syncifying resolve would permit alignment on these specs, which is currently at risk.

[JakobJingleheimer]

Synchronous import.meta.resolve() would be great, as it being async is a bit of a pain/gotcha.

[cspotcode]

Thinking out loud about all the potential use-cases for loaders being discussed:

Do any of the http loader use-cases currently assume the ability to make http requests within resolve()?

It's no problem for resolve() behaviors that synchronously query the local filesystem, just wondering about other use-cases that may be assuming access to async operations.

[GeoffreyBooth]

Do any of the http loader use-cases currently assume the ability to make http requests within resolve()?

None of the examples in https://github.com/nodejs/loaders/blob/main/doc/design/proposal-chaining-recursive.md currently do, but I think it’s easily conceivable that a loader might want an async resolve. For example in the unpkg loader example, that example is trivial but a more realistic one might include something like a network call to an unpkg API to check if a package is hosted by unpkg before we rewrite the specifier to load it from there. I feel like it wouldn’t take much to think of other reasonable use cases for an async resolve hook. Maybe that logic can always be moved into load (?) but I’m not sure.

[cspotcode]

Oh right, something like the unpkg example is what I keep mentally returning to: a loader with resolution logic that requires looking up files / directories / URLs first, such as consulting a package.json or other manifest.

If the resolved URL is dependent on async logic, then I don't think it can be moved into load, since load is unable to modify the URL. If module A imports B imports C, then proper resolution of C requires B to have the correct URL.

Is the goal to implement synchronous import.meta.resolve() but to keep the underlying resolve() hook async? So that the main thread is blocked while the loader thread performs resolution? (once loaders are moved to a separate thread)

[DerekNonGeneric]

For example, it is seeming more and more likely Chromium wants to ship a synchronous import.meta.resolve while Node.js has an experimental async version to support the async resolver.

So, all I have for Web stuff is:

https://github.com/DerekNonGeneric/web-context-js

The "specifier" part is "opaque" (defined by "host"), until "import maps" finally ships (did it?), that would be a no.

We cannot use "http" at all currently, (nor do we to my knowledge, that is a Deno-only thing currently).

[JakobJingleheimer]

@DerekNonGeneric looks like no, import maps have not yet landed 😢 nodejs/node#49443

For remote resolution (which would suggest needing to check outside knowledge), I would expect a prefix like what @bmeck does in his module mock PR. An import map could suffice, but puts a wee bit more onus on the user who then needs to be aware of unpkg or whoever's conventions—hardly a showstopper, but could maybe be improved with a hook:

Continuing with unpkg, unpkg:react. The unpkg resolve hook would be aware of its own specifier and URL conventions, so it would pick that up, build the URL to whatever it should be, and then a (probably generic https) load hook would check whether it actually exists, is available (eg if authorisation is a factor). This seems like a proper separation of concerns too—resolution is only concerned with producing the URL, and loading is concerned with getting the resource on the other end of that URL.

This (as well as import-map) also works in scenarios where packages/modules are created during the app's lifetime (such as what Framer.com do constantly): resolve doesn't need to be aware of the new package/module, only how to find it.

If the other side of the URL has a package.json specifying subpaths via exports or something similar, that's still load()'s job (which we already know needs to support async) to follow; resolve() still got us to the front door.

If I understand it correctly, import.meta.resolve() is only async because the resolve() hook is. It seems to me neither resolve()s inherently need to be async. I could see a resolve hook wanting (or maybe needing?) to interact with an API that has only an asynchronous version though, which would be a bummer.

If it's possible to keep import.meta.resolve() and loaders resolve() separate and sync and async respectively, I think that could make sense and allow us time to see the resolve() hook in the wild. But if resolve hooks need to be aware that import.meta.resolve() is synchronous, that would be a mighty pitfall IMO.

[giltayar]

@guybedford what would be the reason browsers want a sync import.meta.resolve? Seems not to make sense given the inherent asynchronicity of HTTP, and of ESM in general.

[ljharb]

@giltayar with import maps, the initial URL in a browser is synchronously available. The final URL, of course, would be both variable and also async, but I can certainly see the need for a sync API to determine the first network request URL that will be contacted. (no idea if browsers actually want this or not, ofc)

[giltayar]

@ljharb I can see that initially they don't really need an async api, but they would be closing the door on a browser loader api that should be (IMO) async.

[guybedford]

Right, I was always personally hopeful for an async resolve hook because of the possibility of dynamic resolution, which could seem important not to rule out. It's at a little bit of an impasse between Node.js and the browser right now (and there is an HTML PR up for a sync import.meta.resolve). Hopefully putting thought to dynamic import maps will shed some light on this question up again for browsers, ideally when firefox implement import maps.

[ljharb]

Node and browsers are different; there's really nothing wrong with making a different name - if browsers make a sync resolve, node could provide an async one named something else, and folks who want to write code universally can feature-detect.

[giltayar]

@ljharb having two resolves (a sync and an async one) doesn't solve the problem. Because once there is a sync resolve, then the "resolve" loader hook MUST be sync, which will preclude async resolutions (e.g. HTTP).

(and, yes, I know we can now do sync HTTP using workers and atomics, but that means that the loader needs to create a worker for this, which isn't a lightweight option.)

Just putting it here. It's not like there's something that can be done in this context. 😊

[DerekNonGeneric]

I apologize for allowing this issue to get completely derailed from the original goal. We seem to have one global object missing needed to support some sort of “Workerized JS Module Script Loading” according to an unofficial draft specification in the WICG.

I have linked to the issue below — it looks interesting…

nodejs/node#40844

This is somewhat relevant and necessary for interop (with who is not the important part, but perhaps to achieve the goal of having “Worklet Infrastructure” outlined in that specification being the important part).

[DerekNonGeneric]

This issue should help us track progress in this area […]

There seems to be a good amount of momentum behind URLPattern (Chromium is either almost done implementing or needs to write more tests for them to pass) according to the web-platform-tests dashboard. Still, I'd give it some more time before we jump on the bandwagon, though (that's just one implementer).

This API may still not be up to snuff for us as it's currently specced as mentioned in the original issue (need more info). I guess I wouldn't get my hopes up too soon — it looks like something we should keep our eyes on (maybe even potentially review their spec and participate if we know of any ways to optimize to make it more usable on the backend).

Seeing as how non-browser environments would sometimes have a filesystem path rather than an actual URL (as in the case of CommonJS specifiers), I wonder how optimal this may be for us? I almost wish it could handle filesystem paths, too; otherwise, we'd have to normalize some specifiers using pathToFileURL before using it.

• WICG link: https://wicg.github.io/urlpattern/#urlpattern
• MDN links:
  • https://developer.mozilla.org/en-US/docs/Web/API/URLPattern
  • https://developer.mozilla.org/en-US/docs/Web/API/URL_Pattern_API
• WPT links: https://wpt.fyi/results/urlpattern?label=experimental

I still haven't heard too much interest from the loaders team about this yet…

[bmeck]

@DerekNonGeneric I at least in the Loaders side of things don't see the need for URLPattern or even the advantage to using it. For example, if we scope things using URLPattern that means a cost prior to doing any operation. Additionally it uses string based semantics and really is best suited for HTTP/HTTPS and not mixing with other URL schemes. Im especially confused on how this relates to putting stuff in workers.