"Transparent" Interop is a confusing term

[bmeck]

We should try and find a better set of terms as this usage of the word "transparent" is proving to be different across multiple discussions / viewpoints. I'd suggest we split up different points and avoid using the word "transparent" when talking about things. How do other people feel about deprecating using the term "transparent" when talking about interoperability?

[MylesBorins]

+1 to deprecation

Do you have a suggestion for alternative?

Mixed Mode Interoperability?
Overloaded Import?

[jkrems]

How about just "interop"? In the context of the module discussion that seems sufficient. Potentially with "backward interop" and "forward interop" to describe the two directions?

[guybedford]

I don’t really understand where the confusion comes here - it seems an incredibly well defined term to me: Given a parent module written in a module format, the ability to load a child module of a different module format from the parent using the natural native module syntax of the parent module. The term interior covers the fact we are bridging module systems. The term transparent covers that we are using the existing module format machinery. What are the confusions we’re seeing here with this definition? Can we clear them up somehow?

…

On Mon, 25 Jun 2018 at 19:35, Myles Borins ***@***.***> wrote: +1 to deprecation Do you have a suggestion for alternative? Mixed Mode Interoperability? Overloaded Import? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#138 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAkiytk1yNXD3DHbYR6ptlIqQC_GCwZEks5uAR9ZgaJpZM4U2ira> .

[GeoffreyBooth]

I interpret “transparent” to mean, “as a user, I don’t need to know the module type.”

I agree it can be confusing, but I think maybe it really is this simple? The core of it is that I can import a package without needing to know whether that package is ESM or CommonJS. The rest of transparent interop flows from that: I need to be able to export from a package such that this “blind” import works, etc. Ditto for other ways of importing and exporting modules, such as importing or running single files or strings.

One especially confusing area is the “drop the file extension” part of the .mjs proposal, where it lets me do e.g. import './index' and in the code it’s transparent whether I’m importing index.js or index.mjs, but it’s obviously not transparent in the filenames on disk.

This assumes that interoperability is limited to importing and exporting, and perhaps there are other forms of interop that I’m not considering, in which case maybe those other parts should get another term? And I’m sure there are other things that people have lumped under this rubric that perhaps should have other terms, but I think “transparent interop” is definable.

[bmeck]

I don't think we can have a single term.

The problem I think is that it covers too many things, we need to classify different aspects of interoperability. Per my thoughts on how to classify things:

• require interoperability : the ability to use require() to load ESM
• import interoperability : the ability to use import to load non-ESM (JSON/C++/CJS/etc.)

Part of the problem also is the usage of interoperability when paired with transparent also brings up things that are not about interoperability but specifics about how isomorphism could work inside of an interoperability story:

• module isomorphism : the ability for a module to present the same API if loaded via import or require()

  This isomorphism has several topics of discussion since there are limits and constraints to discuss for each. Only having a limited story for cross module system Isomorphism seems fine:

  • named export isomorphism : the ability for a non-ESM module to present the same API as an ESM containing arbitrarily named exports
  • platform isomorphism : the ability for a module to be loaded and run properly into all platforms without alteration of source code
  • timing isomorphism : the ability for a module to execute in the same timing regardless of being ESM or not

• specifier isomorphism : the ability for a specifier to resolve in the same manner in all platforms

We often treat the idea of isomorphism as all or nothing when using the word "transparent" which makes it hard to discuss transition paths and compatibility for things that only wish to have a path, even if not universal:

• forwards compatibility : the transition path/overlap created that allows isomorphism under all the types above. This does not necessarily apply to all forms of modules CJS/ESM/C++/etc.

In addition we are also using the term to discuss specific, often configurable, implementations of loading steps:

• resolution mechanisms : mechanisms used to locate a resource
• translation mechanisms : mechanisms used to parse, produce a module record for, and evaluate a resource. These must use facades for non-ESM.
  • this topic generally seems to only have real debate around how to decide on which translator to use
• loaders : configurable mechanisms used to alter the module records being loaded via hooks

[ljharb]

@GeoffreyBooth “as a user, I don’t need to know the module type" is why .mjs is transparent (because only the author needs to know that, not the consumer).

I think we should differentiate between, to name a few:

• consumers need to know the module type, or not
• authors need to know the module type, or not (i can't conceive of how this can be avoided, ofc)
• require('esm')
• import 'cjs'
  • named imports from CJS

All of these are facets of "transparent", but not its entirety.

[robpalme]

Some people think transparent interop is obvious/well-defined and others interpret it as some combination of independent attributes. I believe the simplicity of the term is unhelpful and is preventing us from communicating clearly in discussions. Maybe if we define the terms precisely we can better debate the problem space.

I've made an attempt at defining these terms loosely based on @bmeck and @ljharb 's breakdown using @SMotaal 's latest glossary document as a place to hold the definitions. I would invite people to comment/edit the document so we can iterate on the terms one place, as opposed to building ever-longer issues.

https://docs.google.com/document/d/150lG_qm7GdylZ08akrk8gdIUgZC8HtnTGAfgD9R5QEI/edit#heading=h.mn114lmmcllh

To be clear: my opinion is that the term transparent interop has now become so over-used for different things that we should (a) deprecate/suspend using the term until we can agree that it has one written definition, and (b) encourage the usage of more precise terms where possible.

I'm not attached to the new terms in the doc or their definitions so feel free to refactor vigorously until they make sense.

[bmeck]

@robpalme that is a good direction to take this I think. I personally would like it to be split up more, but think adding terms separately might be enough.

[zenparsing]

@robpalme Thanks. I think it's interesting that we are converging around the definitions of "require interop" and "import interop".

In your definitions, you specifically call out some challenges with each interop direction:

• For "require interop", the challenge of returning a Promise from require, because of async ESM modules.
  • Comment 1: I would say that returning a Promise from require is an interop failure. Async is infectious. If I have to wait on a dependency, then I become async and can no longer satisfy the same module API.
  • Comment 2: I think it's important to note that the only essential asynchronicity is the one introduced by top-level await. In that sense, I would say that dependency graphs containing top-level await are not "require interoperable", even through transpilers or transpiling loaders.
• For "import interop" the challenge of returning named imports.
  • Comment: Given that we don't want to mess with module evaluation order, I don't see a way to support named imports "out of the box". Perhaps it is enough for CJS authors to "opt-in" to named exports by publishing a "stub" ESM entry point which does require("./") and re-exports the correct names?

[robpalme]

@zenparsing thanks for the comments.

• For require interop, I agree and am inclined to think that the existence of dynamic import() negates the desire for a version of require() that returns promise-wrapped modules. So I have redefined require interop to mean returning the namespace directly as a value. I'll add the promise-wrapping variant back if anyone says they want it.
• For import interop, I think we still need both explicit qualifiers (import interop without named exports & import interop with named exports) because some desire exists even if we believe ultimately it is not feasible.

[bmeck]

@robpalme I'd add the Promise wrapping variant because I don't think that the ability for APIs interoperate deals with isomorphism of the APIs.

[robpalme]

@bmeck Sure. We can go for completeness if you think it's important. But is there anyone that would ever want a variant of require() that returns promise-wrapped ESM namespaces? I'd prefer succinct terms for things that we all agree only have one desired meaning in practice.

[bmeck]

@robpalme people shipping libraries to versions of Node that don't support ESM and want to create an API that works the same as in a version of Node that supports ESM. They could be compiling ESM down to Promises using something like babel, or they could be directly writing CJS that is isomorphic to people on versions of Node supporting ESM. Basically, anyone who wishes to continue support consumers using require() (most likely due to shipping to older versions of Node).

[zenparsing]

@bmeck I agree that "require interop" does not necessitate isomophism of the module's API. On the other hand, I'm not sure we should call Promise-returning require "interop".

Let's assume for the sake of argument that we have a version of "require interop" that returns promises for module namespace objects. If I have a module C which takes such a dependency, then C's exports will also need to be made asynchronous. And then any module that requires C will also need to be made asynchronous. Even if I keep the shape of my module the same, consumers of that module are forced into an async mode.

In such a case, package authors and consumers are forced to agree on the module "mode", which is something I think we are trying to avoid with interop.

[bmeck]

@zenparsing in my description above this is exactly why I had timing isomorphism. However, I will note that I feel very strongly that this is a form of interop since you can still use require() in an isomorphic manner, albeit for a subset of all possible modules. I don't think isomorphism means that all ESM/CJS is capable of running in the the opposite mode, just that there is some form of isomorphism that can be achieved that may require specific coding patterns such as producing CJS of the form:

module.exports = async () => {
  await null;
  // ...
};
module.exports = module.exports();

The above example does produce a situation with timing isomorphism with an ESM if it were to be returned from require() as a Promise. It also shows interoperability of using require() to interact with ESM. You would need to persuade me somehow that neither of those statements are true for me to consider it to not be a form of interoperability between require() and ESM that has isomorphism.

I agree in your example that asynchrony is viral in nature, but that is unrelated to interoperability. Interoperability on its own is related to the ability to use something from a different thing (in this case ESM from require()). Interoperability strictly is not about requiring no code changes, that is the realm of isomorphism.

[zenparsing]

@bmeck

Interoperability on its own is related to the ability to use something from a different thing (in this case ESM from require()).

I agree, but it appears to me that if I consume a promise-returning require, then based on the examples above, I'm no longer really CJS, at least in the way that CJS is typically written and consumed. I've now become something slightly different: maybe we could call it asyncCJS. And all of my consumers are forced to become asyncCJS.

From that point of view, require(esm): Promise<ESM> is interoperable with asyncCJS, but can we say that it is interoperable with CJS?

If not, then I would say that any essential asynchronicity (like top-level await) breaks "require interop", regardless of whether node or a custom loader is providing that interop support.

[bmeck]

@zenparsing in my example above it works perfectly well to put that in a CJS file today, what makes the distinction that it is not CJS but something else? How is a CJS file that exports a Promise<T> not CJS?

[ljharb]

@zenparsing
This is an example of how you might consume things asynchronously today, using live bindings, with CJS.

const promise = require('some-async-thing');

module.exports = { foo: null };
promise.then(x => { module.exports.foo = x; });
// or
let cache;
module.exports = function getFoo() { return cache; }
promise.then(x => { cache = x; });

It's certainly not idiomatic, but there's all sorts of cases where requiring a promise need not force the export to be a promise in current CJS.

[zenparsing]

@ljharb I'm not sure I would call that "live bindings". I'd call that zalgo! 😵

[bmeck]

I don't think we can give the list of names on the export to form that module namespace synchronously, so I don't think we could preserve that exact code since it doesn't wrap the whole export in a Promise.

[ljharb]

@zenparsing it's identically as z̲̗̼͙̥͚͛͑̏a̦̟̳͋̄̅ͬ̌͒͟ļ̟̉͌ͪ͌̃̚g͔͇̯̜ͬ̒́o̢̹ͧͥͪͬ as ESM live bindings :-p

[zenparsing]

I was totally trying to figure out how to put that graphic in there and settled for dizzy face. You win!

[demurgos]

@bmeck
Your example works well if we have a CJS consumer. You can keep the same API for the lib (CJS or ESM lib).
What if the consumer moves to ESM? The exposed interface is no longer the same then.

[bmeck]

@demurgos the consumer could still get access to require() somehow and still load that file? Moving to ESM means they most likely would import the CJS dependency though, and that is no longer a topic about require() interoperating with ESM.

As a related topic, if the library were to move to ESM and the consumer were to remain CJS/use require() to access the library it would continue to work in the same manner as my example above if the library were to match whatever isomorphism is allowed.

[demurgos]

See #139 for a more detailed explanation of my issue with this solution.

The goal of the pattern you provided is to enable the lib to move from CJS to ESM without breaking a CJS consumer (as you stated in your second paragraph). But if the consumer switches to ESM before the lib, then the lib migration becomes a breaking change in this case.

It means that the library needs to both update its API and switch to ESM in a single step to avoid this kind of situation.

[zenparsing]

@bmeck

How is a CJS file that exports a Promise not CJS?

Rather than pursuing that line of argument, I think I'd just like to say that require(x): Promise<T> is not a desirable or compelling interop story. Even leaving the viral async argument aside, I think (hope!) I can safely say that no one wants to consume an ESM module from CJS by doing this:

module.exports = (async () {
  const foo = await require('foo');
})();

(Also, if I am using Babel to transpile my ESM to CJS, is Babel supposed to generate the above?)

[robpalme]

In the spirit of permitting CJS to be written for today's Node (pre-ESM) that can cope with importing future ESM module graphs containing Top-Level Await, I find @bmeck 's arguments in favour of require(esm) : Promise<ESM> reasonable.

This can be achieved today by creating a CJS module that just exports a promise containing an object that meets the constraints of ESM namespace (using restricted identifiers, not callable, etc). Then in future Node with ESM, require() would operate such that it wraps ESM namespaces in the promise.

It may not be to everyone's tastes, but it does solve a problem. In terms of the tradeoffs, and how CJS-ey it feels to use, I can understand @zenparsing 's view that this is not the interop many people would want, i.e. users would prefer to forfeit compatibility with future module graphs that contain Top-Level Await because (maybe) that will be a niche feature.

So I think at least we can give these two types of require interop names:

• Full require interop: Compatibility with the full set of future ESM module graphs, which necessitates require() returning a promise.
• Synchronous require interop: Compatibility with the set of future ESM modules graphs that does not use top-level await, which means values can be return directly.

What do you think?

[demurgos]

I think that promise-wrapping on the CJS side is an anti-pattern because it complicates the migration of consumers to ESM.

If you can use import() in CJS, then the main use case for require("esm") I see is to use it in libraries dealing with lots of dynamic imports and having to work with Node versions where import(...) is not supported. You can then use Promise.resolve(require(...)) with some logic on top of it to backport import(...) (mocha for example?).
Trying to promote async require("esm") as the default consumer-agnostic way to do imports instead of asking the consumer to use import(...) will lead to more issues. It's behavior may be defined but it will be surprising and bite a lot of people. Unless I see a better pattern for API compat that does not require the consumer to keep using CJS, I'd be careful with require("esm").

[bmeck]

@demurgos you cannot use import() in CJS in versions of Node that don't support ESM. I'm not sure I understand since migration path will be occuring for people on versions without ESM as well. Your suggestion with Promise.resolve(require(...)) requires consumers to manually implement what require("esm") would do, at which point we should question why we don't support that feature, and why library authors wouldn't just export Promises to prepare consumers needing to swap to import(). I'm unclear on how mocha is involved here.

Trying to promote async require("esm") as the default consumer-agnostic way to do imports instead of asking the consumer to use import(...) will lead to more issues. It's behavior may be defined but it will be surprising and bite a lot of people. Unless I see a better pattern for API compat that does not require the consumer to keep using CJS, I'd be careful with require("esm").

Can you clarify what is surprising / what is wrong with the API?

Also people may ship their own form of dual builds that ship a synchronous CJS form and ESM form which doesn't seem to be taken into account; the recommendation for that migration is about when they are seeking to transition entirely to ESM, not around multi-goal support.

[demurgos]

@bmeck
I discussed it more in the "Promise Wrapped Plain Object" section of my post about interop.

you cannot use import() in CJS in versions of Node that don't support ESM.
Your suggestion with Promise.resolve(require(...)) requires consumers to manually implement what require("esm") would do

My assumption is that if you have require("esm") then you already have import(...).
It means that if the users cannot use import(...) then they can't use require("esm") either. Is it true?
This is why I assume that you either can use import(...) directly or backport its behavior anyway because you cannot rely on Node's implementation.

My issue is that if the goal of require("esm") is to enable libraries to migrate from CJS to ESM without breaking their consumers (is it its goal?), then they need to use the Promise-Wrapped Plain Object pattern for their API. I consider this pattern harmful because it only enables a safe migration for the lib if the consumers keep using CJS. If the consumer switches to ESM, he gets surprising results with this pattern.
If there exists another way to use require("esm") to let libraries provide an API allowing to migrate without breaking their consumers, I'd like to see it.

Agnostic CJS consumer (enabled by require("esm") and PWPO):

// main.js
require("./lib")
  .then((lib) => {
    console.log(lib.default());
  })

Here is what would happen if the consumer from the last example moves to ESM:

// main.mjs
import("./lib")
  .then((lib) => {
    console.log(lib);
    // If the lib uses CJS:
    // { default: Promise { { default: [Function: default] } } }
    // If the lib uses ESM:
    // { default: [Function: default] }
  });

If the consumer moves to ESM before the lib, two bad things happen:

• The value of the default property on the result changes from Function to Promise<{default: Function}>. This is highly unexpected and very confusing.
• Later on, when the library migrates to ESM thinking that it is safe, it will break the consumer: the returned value changes back.

The drawbacks of the Promise-Wrapped Plain Object method require the consumer to be aware of the implementation of the lib. This defeats the agnostic consumer goal. Once this goal is out of the table, import(...) is more predicatable.

I mentioned dual-builds as an alternative in my post. require("esm") is not relevant for them. I think dual-mode builds provide the best experience for consumers but may also require some tooling for the libs.

[zenparsing]

@robpalme That summary looks reasonable, and I really want to agree with it, but I'm having a hard time.

Unfortunately, I simply can't imagine a package author asking users to call "then" on their library. Nor do I understand how Babel is supposed to generate the call to "then" if the package consumer is transpiling. Does Babel convert all static imports into require(x).then? Does that force the consumer's API to return a promise when it is required?

If there were no top-level await, then the promise-wrapping issue with "require interop" would dissipate. It may turn out that top-level await creates significant interop hazards; perhaps we should not be assuming it?

[ljharb]

There is not yet a TLA, so i think we should be discussing things primarily without assuming it (not that we should ignore it).

[bmeck]

Even without TLA, import() is going to act asynchronous, there are zebra striping issues with cycles and synchronous ESM and CJS for modules without well known shapes, and loaders are likely to use asynchronous behaviors to allow for things like off thread processing. I don't think TLA needs to be assumed, but asynchrony should be.

[zenparsing]

@bmeck Let's try to unpack some of that.

import() is going to act asynchronous

True, but import() alone does not affect the ability to deliver the module binding graph synchronously; it is not an interop hazard.

there are zebra striping issues with cycles and synchronous ESM and CJS for modules without well known shapes

Can you expand on this a bit? For the sake of argument, let's assume that "import interop" does not support named exports.

loaders are likely to use asynchronous behaviors to allow for things like off thread processing

I don't have a very clear picture of what this scenario looks like. Are we talking about a custom loader that loads non-JS files, or a custom loader that loads JS files in a different way, or both?

[bmeck]

Can you expand on this a bit? For the sake of argument, let's assume that "import interop" does not support named exports.

Zebra striping is a term from https://github.com/whatwg/loader/blob/0093dc874b9739c8cbf96a5994de2b923778e4e5/rationale.md#dynamic-modules-do-not-have-tracked-dependencies and conversations that led up to that point. It is the concept of having multiple "colors" of the module graph while evaluating code and generating bindings. In particular the problem arises from when the list of bindings is dynamic in some fashion. You must evaluate things with dynamic shapes (such as CJS under babel's interop) prior to binding them in the graph. This is doable if the CJS subgraph is a leaf of the entire ESM graph. However, once you have ESM importing CJS which cycles with ESM you have a problem. The ESM in a cycle with CJS must evaluate prior to the CJS that does not yet have a shape (because it is still evaluating). CJS -> ESM -> CJS suffers the same issue. I have some old slides on this topic that may better illustrate.

We have a few scenarios here and I'll try to describe 3 that come to mind:

1. import returns a namespace with a well defined set of names regardless of source text

This is what non-ESM only generating a default export looks like. There is no zebra striping.

2. import returns a namespace with a statically defined set of names (most likely from source text, but not necessarily such as by out of order evaluation).

There is no zebra striping.

3. import returns a namespace that depends on evaluation of source text (be it WASM, CJS, C++, or anything else).

There is zebra striping going on here. This is not able to be achieved in a way that is possible in the current specification. Adding a late binding mechanism would allow this to be achieved at the cost of not being able to generate your binding graph prior to evaluation.

I don't have a very clear picture of what this scenario looks like. Are we talking about a custom loader that loads non-JS files, or a custom loader that loads JS files in a different way, or both?

Loaders can return any supported format, so probably can output CJS, WASM, and ESM at least. They don't necessarily resolve to URLs that point to JS and they don't even necessarily return the same format as that which they resolve to (could compile ESM to CJS or vice versa for example).

[jkrems]

Loaders can return any supported format, so probably can output CJS, WASM, and ESM at least.

One example: If we allow resolving a module graph synchronously, we will lock ourselves out of WebAssembly.instantiateStreaming, which is the recommended way of handling WASM according to MDN. If we are in an async world, we can stream the .wasm content from disk, (potentially) compile it on another thread, and finally link it into the graph.

Forcing import to work synchronously will tie our hands behind our back, even if we could make it work.

[zenparsing]

@bmeck

Thanks for expanding on that. I agree both that we want to avoid any zebra-striping and that we want to support a wide range of async custom loading scenarios.

@jkrems

Forcing import to work synchronously will tie our hands behind our back, even if we could make it work.

Thanks for the concrete example. I agree that graph loading should be async to support all of these use cases.

[weswigham]

One example: If we allow resolving a module graph synchronously, we will lock ourselves out of WebAssembly.instantiateStreaming, which is the recommended way of handling WASM according to MDN. If we are in an async world, we can stream the .wasm content from disk, (potentially) compile it on another thread, and finally link it into the graph.

AFAIK It's recommended because in chrome v8 limits synchronous wasm evaluation to something like 500kb to prevent thread hangs from large wasm blobs. It's less of a problem in node, where a thread "hang" on a dependent resource is sometimes the desirable outcome (or at least always is for a require).

[devsnek]

its recommended in every platform that supports wasm. the format is designed to be streamed. its simply good design.

[weswigham]

Streaming the parse result only matters if you process the stream as it comes in, otherwise all it does is affect the API required to do the same thing (and prevent a thread lock). No esm consumer is going to do something on a partially parsed wasm file, or anything, for that matter, while imports are being resolved. Afaik, the only observable difference would be weather the just thread is locked or not when you use dynamic import. For import statements, weather the resolver you use streams or not is irrelevant; the resulting API is the same.

[jkrems]

No esm consumer is going to do something on a partially parsed wasm file, or anything, for that matter, while imports are being resolved.

That's a weird assumption. import('file') is a thing. So - yes, a consumer might absolutely try to do things while a bunch of web assembly is resolving and compiling. And if it's a lot of code, loading it might have an observable impact on performance if it happens on the main thread.

[weswigham]

@jkrems

For import statements, weather the resolver you use streams or not is irrelevant; the resulting API is the same.

And you could probably use the synchronous API for a statement based request and the async one for a dynamic import if need be.

[robpalme]

In an effort to progress the original issue, I have made a PR to define terms for interop across public package boundaries (Agnostic Package Consumers) vs the opposite (Agnostic Module Consumers). Please use that PR as a place for discussing these specific terms.

[MylesBorins]

Can this be closed or merged into another issue?

[bmeck]

To my knowledge we still don't have a clear definition for "transparent" but are still using the term.

[bmeck]

this term is not being as heavily used these days and i think confusion has been alleviated.