transparent-or-not interop

[jdalton]

Seeing as #81 and #82 hinge on a call on transparent or non-transparent interop I think we should have an agenda item to discuss this (and possibly come to some decisions) at next week's meeting.

[weswigham]

More than just those - there's a bundle of other use cases in the use case doc that also hinge on it.

[ljharb]

To clarify, we need to make a decision on whether the defaults support transparent interop or not (as separate from “named exports from CJS”, which is a discussion worth having if we decide that transparent interop is a default)

[ljharb]

I’ll also request that we defer to the meeting after next - next week’s meeting conflicts with TC39, and a number of members will likely be unavailable.

[jdalton]

To clarify, we need to make a decision on whether the defaults support transparent interop or not

Correct.

I’ll also request that we defer to the meeting after next - next week’s meeting conflicts with TC39, and a number of members will likely be unavailable.

That's cool.

[GeoffreyBooth]

Rather than skip a meeting, where the next meeting would be June 6, could we just push next week’s meeting by one week? So the next meeting would be May 30.

[ljharb]

That sounds great to me; we’d have to get consensus from the group tho.

[jdalton]

@ljharb Geoffrey opened #92 to seek consensus to move next week's meeting to May 30th.

[GeoffreyBooth]

Could someone please explain exactly what the decision is that needs to be made? And the consequences of deciding one way or the other?

Or put another way: why wouldn’t we try to support transparent interoperability whenever we can and it doesn’t conflict with other priorities (spec compliance, performance) and via loaders for all other reasonable use cases?

And if the explanation could be written plainly enough to be understood by someone who has no familiarity with the inner workings of Node’s module system, that would be great. Asking for a friend.

[giltayar]

@GeoffreyBooth

Let's start with parse-goals: in JS (from ES2015 onward), a JS file is parsed and evaluated differently if it's a 'script' (this is all parsing pre-ESM) or a 'module' (i.e. an ES module, not a CJS). Note that NodeJS parses CJS modules as 'script', and this is unchangeable, as it would be a major breaking change.

Unfortunately, TC39 did not specify a way of saying what the parse goal of a file is. The browser world decided on two simple rule: 1. if the <script> has a type=module attribute, then it's a module, otherwise it's a script, and 2. if a file was import-ed by another module, then it is a module too.

This means that, in browsers, it is the "importer" that decides the parse goal of the file.

NodeJS decided differently: the parse goal of a file is decided by its extension: .js for script (i.e. a CJS module) and .mjs for ES modules. An interesting proposal to make it out of band in another way (i.e. a mode=esm in package.json is also floating around). All proposals of pre-parsing the file to determine what the parse goal is were rejected (e.g. "use module", or assuming it's a module if there's an export or import statement). Funnily enough, some babel contributors, it seem, agree that the file extension should be the way to determine parse-goals: https://twitter.com/loganfsmyth/status/996088384182411264.

What has this got to do with transparent interop? If we keep the transparent interop, then the runtime, when import-ing the file, needs to determine whether the parse goal is script or module, and thus needs the current mechanism, i.e. the mjs/cjs split. This necessarily precludes working the browser way, which is having the importer determine the parse goal.

You can't have it both ways. Either have transparent interop and force the mjs/js file extension split (which many people don't like), or don't have transparent interop, and have the ability for the importer to determine whether the file is a module or script.

That's one reason for people to dislike transparent interop. The other reason is this: why would we need transparent interop? If we had a meta.import.require that enables ESM to require CJS, we wouldn't really need transparent interop. Except that the babel world works this way. And if we want existing babel code to work transparently, we need transparent interop. This everyone agrees.

But, unfortunately, if we stick to the letter of the spec, we can only default import from a CJS (because determining named exports MUST occur before evaluating the code, and figuring out the named exports can only occur after code evaluation in CJS). And if we have only default imports from CJS, we're anyway not babel compatible.

So if we don't have babel compatibility, why would we need to have transparent interop?

Those are the two reasons I know of.

[GeoffreyBooth]

@giltayar Thanks for this, your explanation is clear and straightforward. So please correct me if I’m wrong, but it seems to me from your post that if we decide we want transparent interop, which it seems like is our default position unless the compromises that need to be made to achieve it are too unpalatable, then solutions must be found for two issues:

• How should Node know whether the file to be imported is ESM or CommonJS?

• How should Node handle named imports for CommonJS, such as import { shuffle } from 'underscore' as opposed to import _ from 'underscore'; const shuffle = _.shuffle?

With regard to how Node decides whether a file to be imported should be treated as a module or a script, I thought that options other than .mjs were back on the table. Besides the ones you mentioned (package.json, flag, etc.), I’m intrigued by the methods implemented in the npm proposal.

With regard to the named imports, it seems like a binary choice between either supporting named exports for CommonJS or maintaining spec compliance. But I read a lot of discussion among people trying to somehow achieve both; if someone can figure it out, is that a third option? I suppose we could always decide something like “if a spec-compliant way to do named imports for CommonJS can be found, sure, we do that; but if not . . .” and then decide on the backup option. The other backup option seems to be allowing it via a loader that can let users opt in to non-spec-compliant behavior (and by implication, building support for loaders that can allow such things).

And I guess the last decision to be made is what to do if we find a solution to one problem but not the other: ship one solution or neither.

Does this sum up all our decisions and options, or am I missing any?

[benjamingr]

or don't have transparent interop,

One thing that can be done is transparent interop with a loader rather than by default like Myles suggested.

[WebReflection]

One thing that can be done is transparent interop with a loader rather than by default

that would make any ESM (loaded via flags, package.json, etc.) a default parsing goal from the importer with meta.import.require used to load old CJS, which would be the cleanest choice for future code, and quite possibly favorite from the JS community + easy as migration path.

Babel, as well as bundlers, can pre-parse all of that too, so I do hope that'll ship regardless .mjs

[ljharb]

While I think most anything should be possible via a loader, the defaults are most critical - so I think that (regardless of what the defaults are for named imports from CJS and/or JSON), transparent interop by default is critical - alongside the ability for loaders to customize behavior.

[giltayar]

@GeoffreyBooth

which it seems like is our default position

It's already implemented in NodeJS, so it is default in that sense, but we can still change it.

then solutions must be found for two issues

must is a strong word. I would say these are branching points in our decision.

The main problem, IMHO, the modules question in NodeJS has, is that there are so many branching points, and so many branches on each point. The number of dimensions is big, and the implications for each decisions are not fully understand, and are difficult to understand globally.

if someone can figure it out, is that a third option?

Yes, definitely. Otherwise, if nobody takes the time to figure it out, then it's (IMHO) NOT on the table.

via a loader that can let users opt in to non-spec-compliant behavior

A lot of people are leaning to this option (they would still have to answer how they differentiate between parse goals, but I dare say it will be some kind of pre-parse thing). I would dare suggest that this option is like saying there is no transparent interop, because I believe people will gravitate towards the no-loader solution, and only the diehards that refuse to accept the behavior, and the people that are slowly migrating from a babel codebase, will use it. Which is fine by me! :-)

[WebReflection]

only the diehards that refuse to accept the behavior, and the people that are slowly migrating from a babel codebase, will use it

solving via loaders is a migration path as much as using babel itself. babel is a migration old to future code too so I think the migrating crowd, as soon as there is a clear direction, will be much bigger and much sooner than "we" think.

Which is fine by me! :-)

FWIW if ESM becomes the standard (initially enabled by flags or whatever) and loaders can solve interoperability with, or without, the meta, I think that'd be fine for a lot of developers, at least most of those I know.

Regards

[benjamingr]

I think that's definitely something to figure out in #85

[benjamingr]

Also @WebReflection - kind reminder that you are (still) invited to participate in meetings and in the meeting today in particular.

[WebReflection]

thanks for the reminder @benjamingr but like I've mentioned before I feel already well represented by various in this group that having me there would probably result into redundant/unnecessary noise for the whole group to move forward.

But I am following pretty much every discussion, so eventually I might participate if I think something hasn't been considered / discussed.

Regards

[benjamingr]

@WebReflection whatever works for you - just making sure you are reminded that you're welcome to should you choose to participate.

[MylesBorins]

I'm going to remove this from the agenda until we can reach consensus on a minimal kernel / approach for the fork. Please feel free to re add or let me know if this should be on today's agenda

[bmeck]

Can we close this and do it on a per phase basis?

[ljharb]

We definitely can decide the defaults later (prior to unflagging, of course), but I'm not sure we should close the issue until it's decided.

[MylesBorins]

Closing this as there has been no movement in a while, please feel free to re-open or ask me to do so if you are unable to.