Vote on ES module detection

[bmeck]

In order for the EPS for ES modules to progress we need a definitive vote to choose between:

• Using a new file extension for ES modules such as .jsm
• Using a few new package.json fields
  • main -> module for ES module entry point
  • modules to list all the .js files in the package that should be treated as ES and not CJS.

The full discussion is in nodejs/node-eps#3

A sidecar discussion specific to module detection is in nodejs/node-eps#13 (comment)

[jasnell]

/cc @trevnorris @nodejs/ctc

[mikeal]

I'm a little concerned about this.

There are a few too many externalities for this to be voted on strictly by Node.js contributors.

[bmeck]

@mikeal the determination is not solely based upon this, but I am having trouble since people continue to ask for a head count of votes. There also needs to be a clearer way to make decisions on highly debated topics.

[mikeal]

There also needs to be a clearer way to make decisions on highly debated topics.

This is definitely true. But this is specifically within the Node.js project, we can't be expected to get an accurate vote or representation from outside of self-identified Node.js developers.

This has a much broader effect than what I feel we have the capacity to represent. I understand the concern, and the complaints, and I also know that there is a lot of tension because of the feeling that TC39 is not representative of our interests, but I don't necessarily think that the solution is push forward with our own unrepresentative decisions.

[bmeck]

@mikeal the proposal has input from TC39 both ways, as well as input from those who are front end. What are you looking for in order to make this decision? If there is not a definitive requirement my request for a vote will still stand. Do we need to form a separate standards body prior to this decision?

[bmeck]

To note:

• TC39 as a body is not concerned with how mode is detected ( Add application/javascript+module mime to remove ambiguity tc39/ecma262#322 )
• Browsers solely use content-type and type=module which is independent of this decision ( add +module to mime type for script type=module whatwg/html#558 ).

[mikeal]

@bmeck yes, it has input from a broad base, but I would assume those voting here would have voting rights associating with their committer or TC membership status in Node.js Core, which is not as representative.

[trevnorris]

@mikeal There's a misunderstanding of what a vote from the CTC represents in this case. We are voting on whether or not the implementation presented in the EP is an implementation node core is willing to adopt. Our vote does not directly reflect the ultimate decision of the TC39 spec, and if ultimately the spec were to deviate from the EP then we would work with it.

We have received external requests on how node would like this implemented, and it is our responsibility to make sure our voice is heard. Especially in a change so substantial it will alter a basic component of the node ecosystem.

These external requests have asked "what is an acceptable implementation for node?" This EP is the result of substantial discussion and debate. As such, we will not please everyone, but irregardless the external requests have a deadline and that is now.

Again this is not authoritative or even irreversible but it is necessary.

[ChALkeR]

@bmeck (also /cc @ljharb), I remember that we tried to sum the proposals up at https://docs.google.com/spreadsheets/d/1aC2kPwpahPnTjFmfb9EL2hFoNl7NhRGbdh6UG3VG3XU. I hadn't been keeping an eye on this for some time, are there any new proposals that are not listed there, or is anything there out of date?

[ljharb]

An ideal outcome, in my opinion, is one where someone could write up a few straightforward instructions to follow that would encompass the "best practice" for using CJS and ES6 seamlessly, across browsers, node, build tools, and editors.

In other words, I don't think it's important to please everyone - I just think it's important to the entire ecosystem that every corner of it shares a cohesive story, even if it's a bit less convenient for some.

@ChALkeR I don't agree with some of the degree of the colors on there, but i think the list is complete and accurate otherwise.

[trevnorris]

@ljharb Many of those have already been discussed (i.e. pragma or content-sniffing) in the linked EP or associated issues. The presented solution in the EP was reached after much debate.

@bmeck Mind chiming in on these?

[ljharb]

@trevnorris Definitely! i currently believe that a .jsm approach in node is the best one to achieve the ideal outcome I described.

[JacksonTian]

I prefer .es.

[bmeck]

@trevnorris I piped in when that spreadsheet was created. Same with @ljharb I have disagreements on coloring, but those colors are vague and affected by subjective views sometimes.

Note: All methods need a flag to state the mode of things coming in via -e, -i and stdin. We will refer to it as --module for now. The proposal keeps CJS as the default mode for unknown source in order to impact node as little as possible. Affects of changing to ES would most likely be minor for -e, but are breaking by forcing strict mode and they could break in silent ways after we did some tests against top 100 npm modules.

Why is it down to these two options?

Parsing mostly.

One thing to note is that parsing is a big effect on the ecosystem mechanically, both JS and JSON parsing, and this kind of parsing is not typical of asset pipelines. NVM, rails asset pipelines, redirect rules, and various tools w/o parsers would struggle with these. JSON parsing is much more available and less of an impact in this regard, but it would still require a change in mentality for all tools currently based off file extensions.

• Some people have noted that some transpilers do parser based detection, in the EP we note that parser based detection has ambiguity which is a no go.
• It should also be noted that parsing is an expensive operation that we will have to perform double parsing as well should we choose anything that can switch modes as the grammar of Modules and Scripts in the spec differs.

Why keep package.json if it concerns parsing?

File extension impact.

Any .htaccess files or scripts that use rules regarding *.js will need to include the new extension. This means existing configuration may have to change for those servers.

• Some people have noted that the package.json approach would require package.json to run anything in the module goal. This is mitigated for the lone file use case via --module, however, this does not cover any cases where the file is not the top level entry point to the program (config.js files, ~/.npm-init.js, etc.).

[jmm]

• Some people have noted that some transpilers do parser based detection, in the EP we note that parser based detection has ambiguity which is a no go.
• It should also be noted that parsing is an expensive operation that we will have to perform double parsing as well should we choose anything that can switch modes as the grammar of Modules and Scripts in the spec differs.

The emphasis should be on the second bullet point (EDIT: and considerations stated in the preceding paragraph). If those considerations make it unworkable, there you go. But the ambiguity argument is less compelling.

Probably most ES6 modules will already organically include at least one import|export that unambiguously establishes its type. That'd leave just things like polyfills, side effect "modules", and single-module packages that would require authors to opt-in with some explicit indicator of type, which could just be no-op module syntax: export {}.

Any of these solutions (file extension, package.json metadata, CLI flags, or any kind of pragma or in-source marker) will be somewhat of a nodeism. So if potential ambiguity were the blocker for parser based detection, I think that a solution that only requires "opting in" a fraction of files that would be ambiguous otherwise, with something that's standard ES6 syntax anyway, would have an appeal vs opting in every file (via extension or pragma) or package (via metadata).

The performance and toolchain issues should be the main reasons given against that idea. There's also an argument about ambiguity for humans about what type a file is, but that's also an issue with package.json metadata.

[trevnorris]

@JacksonTian The argument could boil down to "file extension" vs "package.json". If file extension wins then exactly what that file extension is can be bikeshed later. That aspect only affects the implementation details on our end.

[bnoordhuis]

I probably won't be able to participate in the vote on today's CTC meeting but for the record, I think I prefer the file extension approach. If nothing else, it's easy to find -name \*.jsm for.

[jasnell]

@nodejs/ctc ... TBH I do not yet fully grok enough of the details of the ES6 module system to feel comfortable voting on this issue so I'm going to abstain from voting. I'll have to spend a bit more time digging into the details on this.

[jasnell]

Closing given that this was discussed by the ctc and voted on. The result is that the CTC preferred the file extension approach.

[yuu2lee4]

why not use .es as a ES modules file extension?