This repository has been archived by the owner. It is now read-only.

NG Modules #5

Open
AlexYursha opened this Issue Mar 2, 2015 · 33 comments

Comments

Projects
None yet
@AlexYursha

AlexYursha commented Mar 2, 2015

As module subsystem API is locked in io.js this is a place to discuss possible non-backwards compatible improvements to it.

As @brendanashworth wrote in nodejs/node#848

all modules have been brought down to "stable" status besides the timers and modules modules, which were locked because we can't risk a change that would end up breaking part of the massive userland system

This issue was inspired by nodejs/node#1015 and nodejs/node#848.

@AlexYursha AlexYursha changed the title from NG Module Subsystem to NG Modules Mar 2, 2015

@AlexYursha AlexYursha referenced this issue Mar 2, 2015

Open

NG Timers #6

@mikeal

This comment has been minimized.

Member

mikeal commented Mar 2, 2015

The locked nature of the module system should be viewed as pertaining to that module system. If we were to introduce a new or next module system it could be entire new and even potentially have a new standard library so long as the prior module system remained unchanged.

@chrisdickinson

This comment has been minimized.

chrisdickinson commented Mar 2, 2015

Re: "potentially have a new standard library", Of note, this means we have to leverage the ES2015 module loader to pull this off, and then we have two things named "fs" (for example) that expose different APIs (which is a trick we only get to pull off once!) The not-too-distant alternative is to expose an "fs2" API.

Re: changing the behavior of the module loader: from the other thread you mention removing the ability to "require into" packages. It's unlikely that the new module loader will break with this behavior, because for it to be embraced it should be a (nearly) drop-in replacement for require.

@vkurchatkin

This comment has been minimized.

Member

vkurchatkin commented Mar 2, 2015

It's unlikely that the new module loader will break with this behavior, because for it to be embraced it should be a (nearly) drop-in replacement for require.

This is sad, but I agree. We need to offer an clean path to migrate to NG

@mikeal

This comment has been minimized.

Member

mikeal commented Mar 2, 2015

This is sad, but I agree. We need to offer an clean path to migrate to NG

"Migration" shouldn't be necessary. Old style modules would "just work" and new style modules would be loaded through the new loader.

@Qard

This comment has been minimized.

Member

Qard commented Mar 2, 2015

The existing pattern of transpiling import declarations into require statements may be problematic. That's something to think about.

@mikeal

This comment has been minimized.

Member

mikeal commented Mar 3, 2015

@Qard ya, I've been pondering that problem. I think it's solvable though because existing transpilers only compile down to require statements because the target for the compilation implements require. Even with a new module system my guess is that people using transpilers today will still use them as they will come to rely on new language features before they are available in the platform. It is then just a matter of transpiling to a new target, one that isn't implementing require but instead a new module format.

@chrisdickinson

This comment has been minimized.

chrisdickinson commented Mar 3, 2015

"Migration" shouldn't be necessary. Old style modules would "just work" and new style modules would be loaded through the new loader.

Migration will be necessary in the sense that one will want to be able to switch our one's require statements for import statements in a straightforward manner.

@mikeal

This comment has been minimized.

Member

mikeal commented Mar 3, 2015

Migration will be necessary in the sense that one will want to be able to switch our one's require statements for import statements in a straightforward manner.

True, but I don't think we're going to be able to offer a lot of value to existing functional modules that will get them to use the new format. We can create a lot of incentives for using new features in new modules and applications but are pretty limited in what we can do for modules that are already fine. The biggest incentive may end up just being left behind, as the growth of new modules quickly eclipses the number of old existing ones.

@chrisdickinson

This comment has been minimized.

chrisdickinson commented Mar 3, 2015

I'm not too worried about incentive – folks will want to update, I'm sure. The point is that we should make updating straightforward and easy, and part of that is that we probably shouldn't break existing capabilities of require, like the ability to require into packages (which was the feature request that originally generated this thread.)

@mikeal

This comment has been minimized.

Member

mikeal commented Mar 3, 2015

folks will want to update, I'm sure.

I wish I shared this optimism :) Regardless of how I feel about it personally I think that any strategy that assumes thousands of people will do manual work in order to "upgrade" is doomed to failure. Especially with our deps of deps of deps of deps issues in npm where not only do people need to update but all their dependents need to update, as do their dependents, and so on.

We should build this to work in a world where if nobody updated it would all still work fine, but also one where people who do make the effort to update don't find the process unnecessarily difficult.

@chrisdickinson

This comment has been minimized.

chrisdickinson commented Mar 3, 2015

We should build this to work in a world where if nobody updated it would all still work fine, but also one where people who do make the effort to update don't find the process unnecessarily difficult.

I think we're arguing the same point, here. Things that work with require should still work with new-style import syntax, yes?

@mikeal

This comment has been minimized.

Member

mikeal commented Mar 3, 2015

@chrisdickinson nah, I'm not saying that. I mean, I'm not excluding it, I'm just open to changing it. require should continue to work as require does now, I think we're all agreed there.

Think about this for a second: what if import had no stdlib? What if the stdlib for import was entirely from npm modules? Might not even be possible, or worth it, but I think the possibility is worth considering. It's also worth considering the possibility of a stdlib that is bundled but can be updated independently.

Remember that if people want to update to the new module system but retain the old stdlib all they need to do is change their require statements to import old-fs or whatever and add old-fs to their npm deps. If we want to dramatically alter the presentation of stdlib we should feel free to experiment with anything short of breaking the existing module system.

@Qard

This comment has been minimized.

Member

Qard commented Mar 3, 2015

The biggest problem I see with the current transpiling pattern is that I've seen many modules that ask you to include register('babel/register') at the start of your app, which will transpile everything that gets required. When import suddenly has its own meaning in io.js, that could very easily break things.

If we want to do something with the import syntax, I think we should work closely with the transpiler writers to make sure potentially dangerous practices are discouraged away before the functionality lands.

@chrisdickinson

This comment has been minimized.

chrisdickinson commented Mar 3, 2015

@chrisdickinson nah, I'm not saying that. I mean, I'm not excluding it, I'm just open to changing it. require should continue to work as require does now, I think we're all agreed there.

Sorry, should have clarified "things" a bit: should import be able to do something like import name from "package/internal-file.js";, as require can?

@mikeal

This comment has been minimized.

Member

mikeal commented Mar 3, 2015

@chrisdickinson oh yes, absolutely :) although you should never use grab bag modules that export more than one property. BAD!

@piranna

This comment has been minimized.

piranna commented Apr 7, 2015

Maybe nodejs/node#1278 would be of interest...

@Fishrock123

This comment has been minimized.

Member

Fishrock123 commented May 14, 2015

It may be prudent to think about NODE_PATH (& related) if es6 modules will be handled differently than commonjs: nodejs/node#1627 (comment)

@trevnorris

This comment has been minimized.

trevnorris commented Jul 9, 2015

While reviewing the wasm module loading system I realized that the Loader spec for import contains issues wrt being used on the server. Some of these have been outlined here: WebAssembly/design#256

There are aspects of import in general that need to be taken into consideration. For example, devs can no longer conditionally set the path of a native module depending on the build type.

@balupton

This comment has been minimized.

balupton commented Dec 1, 2015

Wondering if any progress has been made since for adding support for ECMAScript Modules into Node? What are the considerations and tasks that need to be done to land this?

@mikeal

This comment has been minimized.

Member

mikeal commented Dec 1, 2015

@balupton modules haven't landed in v8 and the loader spec is in a state where we could never support it. So there's significant spec and v8 work before this even gets considered in node and there's an argument that leaving it out of Node is actually the best way to support it since compilers are already cross-compiling to/from node's module format and throwing in native ES6 modules might break all of that.

@balupton

This comment has been minimized.

balupton commented Dec 1, 2015

@mikeal perfect thanks!

@mgol

This comment has been minimized.

mgol commented Dec 1, 2015

there's an argument that leaving it out of Node is actually the best way to support it since compilers are already cross-compiling to/from node's module format and throwing in native ES6 modules might break all of that.

If you're transpiling ES6 modules then Node native support won't break that as transpilation will still happen. Future transpiler versions might break compat but that's why you use semver.

I think this would be a harmful position to not support something "because you can transpile". Having the final syntax+semantics work without transpilation should be a goal, transpilation always introduces an additional barrier of entry, barrier to debug etc. Nothing beats native.

@mikeal

This comment has been minimized.

Member

mikeal commented Dec 1, 2015

@mzgol the problem is that not all the compilers use the same module resolution semantics as Node. This isn't an issue while the formats are different and can easily be distinguished but becomes problematic if you add native to the language.

I think this would be a harmful position to not support something "because you can transpile"

That's not the justification. The justification is that it might actually be better for users of this feature not to support them natively.

Promises are a good example where native support wasn't actually necessary since Promise implementations are widely adopted in npm, and are actually faster than native right now, but they landed and we're improving the experience with them in node because native offers us a few things you can't get with userland promises.

Modules are a bit different, the module standard is the connective tissue we use in order to connect all of the code written in JavaScript. The primary value of the current standard is its compatibility with 200K+ modules in npm. ES6 modules standard doesn't do that, it offers capabilities to users that they might prefer when building applications. Compilers are what currently connect the world of published modules to applications built using ES6 module features. For everyone involved, adding native ES6 modules causes a lot of problems and I haven't seen a compelling story about what they actually add in terms of value being native.

The primary argument seems to be that native support is some kind of stamp of approval, which really isn't what core does. Node is best when the ecosystem and larger community own solutions rather than core.

@calebmer

This comment has been minimized.

calebmer commented Dec 1, 2015

@mikeal as you said, modules are the connective tissue of node, therefore different solutions by the ecosystem is not tenable. Now a couple of responses:

“Native support is some kind of stamp of approval.”

That's a cynical way to put it, the feature has already been specced into ECMA. It's already been approved. The primary argument is more about coherence to the already approved and well thought out standard.

“I haven't seen a compelling story about what they actually add in terms of value being native.”

So first, there is undeniable subjective value to developers, even if they can't articulate it. This can be seen in the existence of this issue.

Second, for objective benefits I always refer to this article. To sum it up, ES6 modules make a few critical guarantees by being static for better developer tooling/performance. But saying these benefits should only be on a project by project basis is short sited, if all 20,000 node modules supported this static syntax the creative possibilities are unimaginable (well actually imaginable in a Browserify/Webpack context). Native support gives this guarantee.

Third, for a more technical reason, statically defined modules solves a problem every developer faces at one point or another (anti pattern or not). Circular dependencies. Even with a compiler, this intuitivly simple problem is not solved.


However, I have to agree with every single scale point made. JavaScript development with ES2015/2016 is way different from the current version with CommonJS. I don't think it's a question on promises, or classes, or es6 modules, or any new feature if node should implement them (it seems to me this has been answered by the ECMA spec committee and all the developers using Babel), but rather a question of how and when.

@mikeal

This comment has been minimized.

Member

mikeal commented Dec 1, 2015

So first, there is undeniable subjective value to developers, even if they can't articulate it. This can be seen in the existence of this issue.

I should have been more specific. When I mentioned the features for application developers I was acknowledging this, but that value is there today for developers. What I struggle to find is the value we add by supporting it natively, I only see the pain we cause those using it already.

@mikeal

This comment has been minimized.

Member

mikeal commented Dec 1, 2015

Circular dependencies.

Circular dependencies work in our current module system so this isn't a new feature.

@mikeal

This comment has been minimized.

Member

mikeal commented Dec 1, 2015

Second, for objective benefits I always refer to this article. To sum it up, ES6 modules make a few critical guarantees by being static for better developer tooling/performance. But saying these benefits should only be on a project by project basis is short sited, if all 20,000 node modules supported this static syntax the creative possibilities are unimaginable (well actually imaginable in a Browserify/Webpack context). Native support gives this guarantee.

This works without native support, tools that are ES6 Modules aware already do this even though the code is running through a compile step. In fact, all 200K+ module do support this when you import them into an ES6 module through this tooling.

It's actually quite impressive how many things developer tools support through compile-to languages and toolchains, just check out the TypeScript features in Visual Studio.

@calebmer

This comment has been minimized.

calebmer commented Dec 1, 2015

@mikeal

I only see the pain we cause those using it already.

I see that pain too, but I also see the pain in compiling to UMD. I also see the pain of full stack software engineers trying to push the language and the stack forward to great new heights limited by legacy code (how did that turn out for IE?).

Circular dependencies work in our current module system.

Forgive me, the last time I tried it didn't work (I was using a compiler, isn't that ironic).

It's actually quite impressive how many things developer tools support through compile-to languages and toolchains, just check out the TypeScript features in Visual Studio.

I have, and this is why I fight 😊
I come from a Java background where this is the norm. So it's been proven to be effective by other technologies. So you yourself have seen the benefits.

These tools are not easy they require some level of obfuscation. At the end of the day, with all the polyfills I want I'm still requireimg a CommonJS module. Most modules written in ES6 don't even ship that source code to NPM.

We see benefits in the status quo to the elite few who can afford to get their hands dirty. By taking modules native we give this power to everyone. To dark matter developers, to designers, to teenagers. ES6 modules are intuitive and they are the decided future of JavaScript (even if they are not yet the decided future of node).


Again, I want to be clear that I understand all the reasons you disagree with me. I think they are all valid and should be obsessed over. But I don't think they should hold node back. I don't think the 200K legacy modules are an impenetrable barrier, especially when so many of them are using Babel already.

To quote the developers working on the new Wordpress project Calypso (a project you @mikeal actually introduced me to in your tweet):

One of the hardest things to do in technology is disrupt yourself. Most open source projects fade away rather than make evolutionary jumps.

ES6/7 is a huge evolutionary jump, I think node should be jumping with it.

@mikeal

This comment has been minimized.

Member

mikeal commented Dec 1, 2015

ES6/7 is a huge evolutionary jump, I think node should be jumping with it.

Agreed, but as JavaScript has evolved it has done so with one critical rule in place: don't break the web. Don't break the mass of code already out there, add things in an iterative and backward compatible manor.

Node.js should follow the same rule and sometimes that's a bit harder for us because we have this mass of code that sits between the existing module ecosystem and the language itself. It's not always obvious what we might break or trouble we might cause adopting new patterns and altering the platform.

As far as what this means for ES6 Modules, the investigation we've done so far says that the loader spec is not doable in Node.js and we need to get more involved in the spec process to create something that might work or simply kill off the loader spec and get v8 specific hooks for implementing our resolution semantics (it's not clear that having a mutable loader is a good idea).

@calebmer

This comment has been minimized.

calebmer commented Dec 1, 2015

As JavaScript has evolved it has done so with one critical rule in place: don't break the web.

Yes, but node is semvered. The web is not.

Node is software, node has one implementation. The web is a set of specifications and has many different implementations.

I agree that node should strive for backwards compatibility, but it shouldn't be a hard requirement.

@mikeal

This comment has been minimized.

Member

mikeal commented Dec 1, 2015

@calebmer that's not really how we approach it. We can't do anything that breaks a significant portion of the ecosystem, semver or not. If we want to change anything significant, even mild deprecations, we have to get most of the ecosystem to migrate off of it before we introduce a breaking change. Much of the value Node.js provides is now in the size and scope of the ecosystem, not in core, and so it's core's responsibility not to break all of that value.

@navstev0

This comment has been minimized.

navstev0 commented Oct 6, 2016

It has been awhile since anyone has said anything in here beyond that last comment in 2015. However there has been a obvious voice through other tickets, and yet no information here or any where on plans. I understand that we are still waiting on v8 to support this; v8 modules. Has the stance on this changed, is it still that node might not ever support this based on the loader spec? Not to be divisive but wasn't the io.js fiasco because node was not keeping up with v8 and therefor new es6 features that v8 was implementing?

@jasnell

This comment has been minimized.

Member

jasnell commented Oct 6, 2016

@navstev0 ... See https://hackernoon.com/node-js-tc-39-and-modules-a1118aecf95e#.kptb8i719 for an insight on the current state of discussion.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.