Future extension to permit asynchronously loaded built-in modules

[littledan]

A previous version of import maps supported fallback lists for the mapping of modules. This could be used to remap a built-in module just in the case where the JS engine did not have the module. The network load to would only happen in that case.

It would be great if JS built-in modules supported this kind of conditional polyfill loading. But this would require that, somehow, modules which used this polyfilled built-in module waited until that module was loaded from the network. Due to top-level await, we have the infrastructure in JS to have modules in the graph which take time to load. So I'm wondering if we should permit this pattern in BuiltinModule.export.

I'd suggest an alternate form of BuiltinModule.export, called BuiltinModule.exportAsync, which expects a Promise (or thenable) as its second parameter. Importing this module with an import statement or dynamic import would act like the built-in module was defined with a top-level await. (I'm not sure what BuiltinModule.import should do--maybe return undefined, or maybe return the Promise.)

I think BuiltinModule.exportAsync separates out well as a follow-on proposal, so I think this proposal is good to go as-is for Stage 2, but I wanted to share the idea.

[ljharb]

How would accessing this value this work in Scripts?

[littledan]

@ljharb Scripts could use import() and await on it, or maybe BuiltinModule.import and await on it. The idea is, we're only fetching this code from the network if needed, so the script will have to wait for the fetch to come down. It's the page itself that would be opting in to this asynchronous polyfilling strategy. Scripts which want to use a synchronous polyfilling strategy could still do so--this proposal would not change the return value of BuiltinModule.import when something was exported normally.

[ljharb]

A constraint that is important to me is that every builtin module - including the polyfilled version - is synchronously available in Scripts, so that Scripts are not second-class to Modules. This entire proposal should not proceed without that capability.

[littledan]

It sounds like your constraint implies, it's important to you that all polyfills are served to all browsers (unless it's possible to skip that using UA sniffing), even if they don't need it. Am I misunderstanding?

[ljharb]

Yes, because it’s not possible to know if they need it until runtime.

[bakkot]

is synchronously available in Scripts, so that Scripts are not second-class to Modules

I don't understand what you mean by this. All modules are asynchronous. It is impossible for anything to be synchronously available in a module (in the sense in which things are synchronous in scripts) because it is impossible for a module to be synchronous at all. The reason you can use import declarations in modules is because they are, essentially, implicitly wrapped in an async IIFE with a header which awaits all the dependencies.

Making built-in modules synchronously available in scripts would be granting scripts substantially more power than any module has, not making them equivalent.

[devsnek]

technically, the way the spec is currently written, you can resolve and link a module graph synchronously.

[bakkot]

Yeah, I suppose I ought to have specified "ES source text modules in web browsers".

[ljharb]

Right - builtin modules are special in this sense, by design.

[littledan]

I can definitely see the pragmatic case for making built-in modules synchronously available in scripts by default, and making it possible to polyfill them in a way that leaves them synchronously available. However, I don't understand what it would break to also expose an asynchronous polyfill feature, for those applications that are just targeting modules.

I believe this opt-in asynchronous loading capability could really help improve load performance. For example, the Temporal polyfill is rather large, and it would be great to exclude it from bundles when the built-in version will be used.

[bakkot]

@ljharb I don't understand that reply at all. Can you say it another way?

[ljharb]

@bakkot sure! It is indeed impossible to synchronously load userland ES Modules, and this is by design. However, language-provided builtin constructs need to be identically (ie, synchronously) available in both Scripts and Modules, in this case so that first-run code can delete/mutate/polyfill/etc them and be guaranteed that later-run code can't get to the originals. Absent builtin modules, "globals" provides all this; the synchronous loading capability is an attempt to ensure this isn't lost with builtin modules.

[bakkot]

@ljharb I'm with you on the "it is important for early-run code to be able to virtualize stuff" point. Where I'm failing to follow is the "language-provided builtin constructs need to be identically (ie, synchronously) available in both Scripts and Modules" bit. I don't understand what you're pointing at there. Nothing is synchronously available in any user-authored module in browsers, because no user-authored module can be synchronous at all. If scripts could only load built-in modules asynchronously, that would be identical across scripts and modules.

[ljharb]

@bakkot code in userland ES Modules appear to run synchronously; in particular, within the same module graph, i don't believe you can observe that the loading is asynchronous (modulo TLA, perhaps)?

[bakkot]

Not from within that module graph, but it's trivially asynchronous from the outside. Other scripts are not blocked from running while the graph loads, just as if you await an import in a script.

[guybedford]

It seems like with the constraints mentioned above this pushes the solution space back into the resolver (as fallbacks did).

Array fallbacks in import maps were deemed a necessary feature to be able to ship builtins at all due to the backwards compatibility / cross browser compatibility constraints which are certainly non-trivial. Fallbacks did seem to accomplish "solving" this problem though, while not placing any async constraint on builtin execution.

@littledan this wider use case does seem pretty important, would you consider discussing fallbacks-style resolution-level schemes here, or do you think that is off-topic for this proposal itself?

[littledan]

@guybedford This proposal overall seems to be trending towards a much more imperative approach, which I think could also work and has benefits of being more flexible/expressive. I made my suggestion in the OP within that pattern. I don't quite understand where a declarative fallback list would fit in, and I would be curious to hear more of your thoughts (here or in another thread). I fear fallbacks would run afoul of @ljharb 's concerns as well, but if they are a way through, then great.

I do want to reiterate that I think this question is one we do not need to resolve before Stage 2. I think we can iterate on these API details before Stage 3 if needed, and also, I don't see an async API as necessary for the built-in modules MVP.