discuss: asynchronous module.import

[WebReflection]

I am not fully sure this is the right place to discuss CommonJS patterns but I'd like to understand if there's the possibility to discuss a possible new Module.prototype method like the following one:

// Module as module.constructor
Module.prototype.import = function (path) {
  return new Promise(res => res(this.require(path)));
};

This would enable asynchronous import and asynchronous exports out of the box.

// modules as we know: converter.js
const crypto = require('crypto');
module.exports = {
  sha256: (str, secret = '') =>
    crypto.createHmac('sha256', secret)
          .update(str)
          .digest('hex')
};

// same good old synchronous flavor
const converter = require('./converter');
console.log(converter.sha256('yolo'));

// module.import asynchronous flavor
module.import('./converter')
.then((converter) => {
  console.log(converter.sha256('yolo'));
});

To export asynchronously the module has to simply export a Promise

// generic async export
module.exports = new Promise(define => {
  // some asynchronous operation ...
  define({my:'module'});
});

// asynchronous converter.js
module.exports = module.import('./crypto')
.then((crypto) => {
  // return the module to export
  return {
    sha256:(str, secret = '') =>
      crypto.createHmac('sha256', secret)
            .update(str)
            .digest('hex')
  };
});

Multiple dependencies can be easily handled via Promise.all.

module.exports = Promise.all([
  './awe',
  './some'
].map(m => module.import(m))).then(modules => {
  const [awe, some] = modules;
  return {awe, some};
});

I'm testing already these patterns on both server and client [1], and there's nothing I can spot that could cause troubles, in terms of backward compatibility, but it could enable new kind of possibilities for both client and server.

Thank you for your consideration.

[1] https://medium.com/@WebReflection/asynchronous-module-import-path-b9f56675e109#.ehnofaj2q

[mscdex]

Have you considered joining the discussion on ES modules? I think discussing features like async module loading is best had there IMHO.

[WebReflection]

@mscdex I've already mentioned above playground there and they believed I was just sponsoring my library, while I'm actually proposing a CommonJS pattern, which is also something they're not interested in (if they were, the current ES2015 import/export status wouldn't be where it is now).

The idea is to use the only two things agreed by TC39, the Promise based solution, which is future friendly with async and await, and the import name, which cannot be polyfilled on the global scope anyway since it's a reserved word.

Would NodeJS community consider implementing this feature, maybe they will have a precedent to consider and work with, instead of the other way around.

Does this answer your question?

[mscdex]

What I meant was why not voice your support for async ES module loading instead of adding it on to CommonJS modules?

[WebReflection]

more than bringing their own proposal into a well known and used env as NodeJS via 3 lines of code and a little utility library that demonstrates all problems mentioned in there are solved, I am not sure what to do.

I am active in the ML and I've mentioned this approach already but we all know it takes ages for all vendors to agree and implement something so here there's hopefully more freedom, being CommonJS a de-facto standard?

Since AFAIK the agreement on the Promise is there already, this idea is future friendly.

Or you are saying that any new feature in NodeJS should be agreed as ECMAScript standard first?
This one seems like a cheap addiction full of wins, doesn't it?

I've opened a thread here to discuss what NodeJS people think (so yes, I might keep bothering TC39 people too with exact same idea, but they already have it there)

[joyeecheung]

At this point creating another defacto standard on our own instead of collaborating with other vendors and make it a ECMAScript standard(even though that would be slow) would only make things worse IMHO. What are the benefits of this proposal over dynamic import(except it would be faster to roll out)? What will happen to this API when dynamic import becomes fully usable?

[WebReflection]

What are the benefits of this proposal over dynamic import?

Dynamic import is unpolyfillable (reserved word, unless you pass through global.import() each time) while module.import is potentially already available and polyfilled for every NodeJS user since Promise have landed in code.

My lil' utility makes this experimentation easily portable to any browser too so that developers could start using this right away.

Every modern API is also shipping Promise based, see customElements.whenDefined('com-name') or document.whenReady, all things that compose well with this module.import but also on NodeJS it happened more than once that I return a module synchronously while I am initiating a DB connection asynchronously (talking about DBs, MQs, and friends).

This pattern would benefit both worlds and it could do it sooner than dynamic import.

except it would be faster to roll out

Well, that's not something to ignore neither, and quite possibly one of the major points.

Having an already adopted solution also could speed-up rolling out on standards since those work counting implementors too. (it's in ES stages process).

They won't adopt CommonJS, it was never the plan, but they can verify what they have proposed is already working at scale.

What will happen to this API when dynamic import becomes fully usable?

What is happening, or will happen, to require once static synchronous import will be widely available?

Transpiled code will fallback to synchronous require or to dynamic module.import until every vendor shipping an ES engine will be onboard with the dynamic loader.

In other words, this 3 LOC feature can move both client and server forward with, as far as I can tell, full backward compatibility and zero risks in shipping it.

Eventually, like every feature in every language, we'll see low adoption but if it doesn't go out we're somehow in a Catch-22 like situation, hence my proposal to roll it on the most used CommonJS based environment.

[joyeecheung]

I see, so this proposal is like another bridge between require and import filling in the gaps of async loading?

I think this discussion should go to https://github.com/nodejs/node-eps/? (Not sure if something like this has been raised there before).

EDIT: And yes, this is irrelevant to import() because it's a pure NCJS thing. I am not sure if this counts as an implementation of import() for TC39 to consider and speed up the process of import() though, because they are not the same thing.

[WebReflection]

I see, so this proposal is like another bridge between require and import filling in the gaps of async loading?

hopefully it's the only needed bridge, accordingly with current dynamic import() proposal, and yes, it could fill the gaps of async loading.

I am not sure if this counts as an implementation of import() for TC39 to consider and speed up the process of import() though, because they are not the same thing.

Like I've said, this idea is indeed strictly a CommonJS related one but it's fully based on this standard prototype/proposal without any of those issues described in there.

Thanks for the hint on node-esp, I wasn't aware of the special repo for these kind of discussions.

I'l try to search for similar proposals and, if none, I'll PR this one. Cheers

[joyeecheung]

Glad to help. I think this issue over here can be closed now.

[joyeecheung]

Also I think nodejs/node-eps#39 (diff) is discussing the same thing as the proposal?