Built-in Modules

[bterlson]

Built-in modules come up repeatedly around the various proposals. I am making this issue to centralize the discussion such that champions of the eventual proposal have a good central location for information. I will keep this up-to-date if there is further discussion in this issue.

Existing Discussion

• Can this API be provided as a standard module rather than globals? ecmascript_simd#323

Syntax Options

Naming convention inside module specifier

This option entails establishing a naming convention for built-in modules. Has to be compatible with existing ecosystems, eg. we cannot clobber an npm or standard node package name.

Strawman: *-sigil

import "*SIMD";
import {float32x4} from "*SIMD";
import SIMD from "*SIMD";

Strawman: URL scheme

import { float32x4 } SIMD from "std:SIMD";

Distinct syntax for built-in module imports

This option necessitates additional reflective capabilities in the Loader API to request built-in modules by name as opposed to going through the normal module resolution process.

Strawman: IdentifierName instead of StringLiteral

import SIMD;
import {float32x4} from SIMD;
import SIMD from SIMD;

Semantics Requirements

Must defer to loader to resolve built-in modules (important for polyfillability). Loader may see either a special string or possibly an options record indicating that a built-in module is requested.

[dherman]

Just to gather a few constraints:

• The aesthetics matter a lot, since importing standard libraries is something we expect to happen in an extremely large number of modules.
• We have a choice between just establishing a naming convention vs. reserving syntax (whether it's different surface syntax or distinguished module identifiers in the string literal).
  • If we choose a naming convention, it has to be compatible with existing ecosystems; in particular we have to watch out for clobbering an npm or standard node package name.
  • If we choose the latter, it will require additional surface area to the reflective layer in the loader API.
  • In particular, anything we do has to allow standard modules to be polyfillable/prollyfillable, which means the registry and/or (probably and) the loader pipeline have to be able to expose distinguished standard modules.

Edit: I see @bterlson already mentioned the exposing-to-the-loader constraint, apologies for the dup.

[zenparsing]

@bterlson Thanks for starting this discussion. I think it's very important for TC39 to set some kind of precedent here since platforms (e.g. DOM) will also likely want to start putting things into their own "standard" modules.

To throw another strawman out there, could we not also use a URI scheme for this purpose?

import { float32x4 } SIMD from "std:SIMD";

I'm concerned about this form:

import SIMD;

since it would presumably have non-local effects (by adding properties to the global object). If the bindings are local instead, I would reject to it on the same grounds as import * from.

[bterlson]

(Updated OP with new information)

I'm concerned about this form: import SIMD;

The way I see it this form is simply a shortcut for import SIMD from SIMD. There are no properties added to the global, SIMD is bound in the module environment record as normal imports are.

[ljharb]

To clarify the polyfill scenario: I must be able to write code that can mutate, overwrite, create, or freeze, a built-in module, just like I can do right now with a built-in global. Mutation/overwriting is for when engines inevitably ship bugs, and shims want to fix them - freezing is for things like SES that want guarantees that nobody can maliciously mutate/overwrite builtin modules later - creating is to provide new modules in older environments.

I also agree that whatever precedent we set should pave a cowpath for engines to add non-language-builtin builtin modules in a non-colliding way, but that ensure the same capabilities I mentioned in the previous paragraph.

[zenparsing]

Also, the IdentifierName variant

import {float32x4} from SIMD;

would be future-hostile to lexical modules, FWIW. And in general, I think it's "lexically surprising" to see an identifier in that position referring to something that's not in scope.

The way I see it this form is simply a shortcut for import SIMD from SIMD

For consistency, users need to write that as:

import * as SIMD from "<whatever>";

We should try not to special-case or give special meaning to forms which import built-ins.

[zenparsing]

Sorry, or

import SIMD from "<whatever>";

if it exports an default.

[domenic]

And in general, I think it's "lexically surprising" to see an identifier in that position referring to something that's not in scope.

I agree with this.

Of the positions so far I like a "std:" prefix the most. It reuses an existing part of the module resolution space (absolute URLs) in a way that can't conflict (std: is not a valid URL scheme today).

[caridy]

@ljharb

To clarify the polyfill scenario: I must be able to write code that can mutate, overwrite, or freeze, a built-in module, just like I can do right now with a built-in global. Mutation/overwriting is for when engines inevitably ship bugs, and shims want to fix them - freezing is for things like SES that want guarantees that nobody can maliciously mutate/overwrite builtin modules later.

You can't really mutate a module or a named export you are importing, in the case of modules, it is likely to be just a shimming process via export and export-from syntax. We have all the pieces in place to support this use-case today.

[dherman]

@caridy Maybe @ljharb just means replace the registry entry, which definitely is a requirement for polyfilling. (This is why it was important that even though modules cannot themselves be mutated from the outside, we still made it possible to mutate the registry.)

[ljharb]

Exactly that, yes ^ sorry that wasn't clear.

[allenwb]

Another problem with the IdentifierName syntax for module specifiers is that it is hostile to existing tools that already parse import statements. Keeping the string syntax means that tools that don't care about the actual semantics of the import don't need to change.

[allenwb]

@ljharb
Regarding "freezing", any actual object values exported by a module can certainly be frozen using the usual techniques.

[allenwb]

From a core language perspective, I think very little would have to be said about the semantics of supporting built-module shimming. Basically, an import of a module identified using a built-in module designator that is recognized by an implementation uses the built-in implementation unless the active module loader has explicitly registered an interest in handling that module. Unrecognized built-in modules and those that the loader has registered for over-rides are simply passed on the the loader.

Loader APIs of course have to provide for registering built-in over-rides. But a simple implementation that only supports a single built-in loader doesn't need to even worry about that case.

[allenwb]

BTW, I also like: "std:SIMD" as long as we are confident that we can safely use "std:" without tripping over any other URI protocol.

I only threw out "*SIMD" (with an escapable *) as a strawman in anticipation that there might be concerns about conflicts with things like "std:".

[ljharb]

@allenwb i meant, freezing it in the registry so further changes to "what gets imported" are impossible, ie, not just Object.freeze on the export, but Module.freeze on the registry entry, or similar.

[tracker1]

I would only hope that whatever the implementation, that consumption/publication of CJS/Node and ES6 modules can interoperate... perhaps following browserify and webpack's logic in this regard.

Ir at least some awareness...

[allenwb]

@ljharb

i meant, freezing it in the registry

ok, then it seems like an orthogonal issue in the design of the module loader API and really doesn't impact the idea of specifying a way to designate standard built-in modules.

[ljharb]

I would think that for SES purposes, the ability to lock down a builtin module from being replaced in the registry is a blocker - @erights?

[allenwb]

Let me try to be clearer. In the absence of a standardized or implementation provided module loader that exposes a module registry there is no way to replace (or lock down) a built-in module. So, from the perspective of the core semantics of import this is a non-issue and shouldn't stand in the way of specifying built-in modules.

Certainly browsers and most other significant implementation will provide such capabilities so the module loader specifications needs to address it. But it shouldn't be a blocker that forces us to avoid defining built-in modules.

[bterlson]

I also like the module specifier "std:SIMD". IANA doesn't have any "std" scheme registered which is a good sign but of course there could be unregistered usage out there.

[nbdd0121]

What about using :SIMD instead, then we don't have to worry about "std" being a valid scheme.

[erights]

Just esthetically, the leading colon looks pleasing to me. And as @nbdd0121 observes, we don't need to worry about the empty string becoming a valid scheme name.

[rossberg]

Of course, the obvious scheme would be "js:" or "es:".

[allenwb]

Do any known file systems assign meaning to a leading ":" in file/path/device names?

Over course, if we are worried about running into that, we could do "::" escaping like I suggested for "*".

Regardless, I like the explicitness of intent we would get with "std:".

[paultyng]

Perhaps the scheme, whatever it was, could be optional for use in the case of resolving ambiguity?

[ljharb]

Please no - optional things cause ambiguity and would present a refactoring hazard if you suddenly added another import that made it ambiguous. Whatever format is decided, it should be always required.

[lars-t-hansen]

Do any known file systems assign meaning to a leading ":" in file/path/device names?

Over course, if we are worried about running into that, we could do "::" escaping like I suggested for "*".

The old Mac OS uses ":" as the path name separator. Names are absolute by default, a leading ":" makes them relative, multiple leading ":" walks up the file system tree, eg, ":foo" is in the cwd, "::foo" is in the parent, etc.

More here, under Classic Mac OS

[tomzhang]

There seems to generally be an assumption in this thread that built-in modules would have to be identified using a string literal module descriptor. But that's not really the case, TC39 gets to define new syntax. To avoid any clashes with host/platform use of string module specifiers, I suggest the following change to the ModuleSpecifier syntax:

ModuleSpecifier :
    StringLiteral
    BuiltinSpecifier

BuiltinSpecifier :
    IdentifierName
    BuiltinSpecifier . IdentifierName

With the static semantic restriction that std as the first IdentifierName of a BuiltinSpecifier is reserved for use in naming built-in modules defined by TC39.

So we might have:

import {doc, freeze} from std.decorators;
import streams from web.streams;
import procs from node.processes;
import gcControl from v8.gc;

It isn't clear if there is a requirement to use dynamic import for built-in modules. Because built-ins are built in and presumably already present as part of the implementation it isn't obvious that there would be any advantage to conditionally loading them. But if that functionality is needed, it can be accommodated without any syntactic ambiguity by:

ImportCall :
    import ( AssignmentExpression )
    import BuiltinSpecifier

I do think this is the best one for naming URL which clear enough for human understanding