4f

[martinheidegger]

I really liked that you stand up to the TC "decision" (hopefully) since I am also not quite happy with the current state.

That being said given that I call the current proposal 4d I would like to add an few concerns I have with 4d and would like to offer a 4f (jumping over e). Maybe it could be useful?!

Concerns

• It adds three important properties (modules, module and modules.root) to package.json which have to be learned.
• The difference between module and main is not clear by looking at the code. (It is easy to mistake module for main when skimming the package)
• When looking at the files in a folder it is not easy to distinguish between CommonJS and ES6 files since the rules might be complex depending on the modules property definition.
• For a package to be supporting both CommonJS and ES6 it is required to pack both variants into the same .tar.gz. Which means that when downloading the package that supports both it will be twice as big.

---

Proposal 4f

Based on those concerns I thought if there might be a way to incorporate that into your work and well this is what I have come up with:

One new property to package.json

In order to figure out if .js files in a package are CommonJS or ES6 we add a new property "syntax": "es6" to the package.json. All files in this package will automatically be treated as ES6 while "syntax": "commonjs" would specify that the files are treated as CommonJS.

Backwards compatibilty through syntax variants

Old Node.js versions do not support ES6. Loading a package that contains es6 would thusly fail. In order to support this case we add support for syntax variants into npm! A new version of npm (say npm@4) supports a new "publish-time-only" property of the package.json: "variants": {"commonjs": "./common_js"}.

Every key in this property points to a folder. On publish npm stores two variants of the package in the package storage: One with all folders except the "./commonjs" folder and one where the "./commonjs" folder content is treated as root.

This way we have two packages stored in the npm registry. The old packages would all be treated as commonjs.

Old npm versions would automatically install the CommonJS variant of the package. The new version of npm would by-default install the es6 variant of the package a new --syntax=CommonJS flag would allow to request the CommonJS variant of it in case npm@4 runs on a old version of node.

(Of course npm could simply assume --syntax=CommonJS if npm runs in an old version of node. Yet I still think a flag is good-to-have)

Npm could even go as far and install the latest fitting CommonJS version of a package in case --syntax=CommonJS.

Transistion support through Babel

Packages that want to port their code from CommonJS to babel but have a hard time to are hard to up-date all at once could for-the-intermediate-time (until all modules are ported) simply use Babel (and "syntax": "commonjs"). This would add a processing penalty and more devDependencies to this case but it would work.

Benefits of this approach

• (like in 4e) deep linking require statements (require("lodash/array.js) would still work.
• (like in 4e) we only deal with the .js file ending.
• syntax is easier to understand and differentiate from the other properties in the package.json.
• Since npm only downloads one variant the downloaded amount would not grow.
• The penalty on transistionary packages would motivate package maintainers to transist completly.
• Variants of a package could open the door to frontend specific packages.
• When someone chooses to learn about supporting older versions of Node.js she has no chance of missing out complex configuration that could end up bloating the registry.
• Backwards compatibility could be added to ES6 package with a comparably simple babel script.

One last concern

With the simplicity of this comes the obvious lack of the possibility that we can't properly mix CommonJS and ES6 modules. The reason as to why this would be required is because ES6 doesn't support all the features of CommonJS (namely: dynamic imports). I see this as task of TC39 to improve the ES6 specification to support dynamic imports.

---

I would be very happy to hear your comments.

[jokeyrhyme]

• (like in 4e) deep linking require statements (require("lodash/array.js) would still work.

My current understanding is that deep linking require statements will still be allowed with native import in Node.js:

• WRT PR #3 - ES module detection nodejs/node-eps#13 (comment)

[martinheidegger]

@jokeyrhyme Yes, deeplinking would still work but packages that port from .js need to port to .mjs in order to be ES6-only. In order to keep API compatibility to former versions they would need to contain .js variants in the package.

[Fishrock123]

"decision"

You may not want to mock us if you actually want buy in from the CTC. (Which whatever is accepted in the end will require, obviously.)

since I am also not quite happy with the current state.

I should probably make it clear that the CTC doesn't really like any of the solutions but that the extension is the most favorable of the bunch. Ideally we wouldn't need anything special but that isn't possible when you have two possible filemodes for a file and need to avoid double-parsing.

A new version of npm (say npm@4) supports a new "publish-time-only" property of the package.json:

It is probably favorable if npm does not need to do anything. I think we've probably caused even more of a mess if that is necessary?

The penalty on transistionary packages would motivate package maintainers to transist completly.

Not exactly positive imo

[iarna]

It's pretty important that npm not have to select different tarballs based on syntax mode and here's why:

Updating the cli to support new registry semantics and updating the npm registry to support them is, relatively, "easy". (I say as the person who only has to be concerned with the former. =p)

But how do you select variants for git dependencies? What about tarball dependencies? What do you do when talking to a third party registry that hasn't implemented this yet?

You might say "well, don't publish things to the third party registry using this feature" except that many of them act as transparent caches for the npm registry.

Making changes to package selection semantics is, I think, best kept outside the scope of any es6 module proposal.

[martinheidegger]

You may not want to mock us if you actually want buy in from the CTC. (Which whatever is accepted in the end will require, obviously.)

@Fishrock123 I did not mean to mock you in any way. I just hoped it is not a final decision - it can be changed. That is why I posted it as comment here, in this repo, that seems to be of same sentiment.
I am sorry if it came across as insulting. Thank you for taking the time to answer this. (I have not been in contact with the CTC and I feel pretty as concerned outsider here.)

The penalty on transitionary packages would motivate package maintainers to transist completely.
Not exactly positive imo

I would be curious about why not?

But how do you select variants for git dependencies? What about tarball dependencies? What do you do when talking to a third party registry that hasn't implemented this yet?
You might say "well, don't publish things to the third party registry using this feature" except that many of them act as transparent caches for the npm registry.

Thank you for the insight @iarna . I think though this also could be easily taken care of. Let me extend the the Proposal a bit:

---

Git/Tarball dependencies

The package.json will have a syntax field to dedicate a package as es6 or commonjs. If the git or tarball dependency has no syntax field commonjs is assumed. Obviously a --syntax=... to download variants would not work with git but the user would need to choose a different branch or a different tarball.

Support for old registries/proxies

The current registry and proxies should have no problem to support the "syntax" property do not support package variants. When npm@4 tries to publish a package with multiple variants to a server that doesn't support (different endpoint) multiple variants the package owner will be informed of this and he needs to specify npm publish --only-syntax=es6 in that case he can only publish es6 packages.

If the proxies or registries don't upgrade they will by-default get the commonjs variant (if given) of a package.

---

It is probably favorable if npm does not need to do anything. I think we've probably caused even more of a mess if that is necessary?

Making changes to package selection semantics is, I think, best kept outside the scope of any es6 module proposal.

npm could help make this situation better. I think it is worth to explore this path. After all this decision will affect pretty much everyone.

[Fishrock123]

@Fishrock123 I did not mean to mock you in any way. I just hoped it is not a final decision - it can be changed. That is why I posted it as comment here, in this repo, that seems to be of same sentiment.
I am sorry if it came across as insulting. Thank you for taking the time to answer this. (I have not been in contact with the CTC and I feel pretty as concerned outsider here.)

Ah, alright. Apology accepted.

Generally we don't do specs and so anything that isn't existing functionality is up for change, to a degree of course.

The penalty on transitionary packages would motivate package maintainers to transist completely.

Not exactly positive imo

I would be curious about why not?

This is a very, very big misinterpretation of the size of the ecosystem and how much time people have to transition, and generally the availability of downstream users to adapt on a dime. We'd rather be a bit "wrong" than hurt everyone hugely.

npm could help make this situation better. I think it is worth to explore this path. After all this decision will affect pretty much everyone.

Perhaps, but why should they have to? If they must, node core has failed in a sense. Plus npm is already super busy so I'd rather we don't hold them up on this too.

[martinheidegger]

This is a very, very big misinterpretation of the size of the ecosystem and how much time people have to transition, and generally the availability of downstream users to adapt on a dime. We'd rather be a bit "wrong" than hurt everyone hugely.

Transitionary packages are packages that contain both es6 and commonjs modules. there is no penalty for commonjs only packages. Also it needs to be clear that this "penalty" consists of a compile script that compiles mixed sources to commonjs. (not at all a huge penalty imho)

Perhaps, but why should they have to? If they must, node core has failed in a sense. Plus npm is already super busy so I'd rather we don't hold them up on this too.

This could also make way for implementing frontend packages which is on the agenda of npm. Also I think that implementing es6 modules will affect them quite a bit (i guess) in their work. So: @iarna 's opinion surely matters a lot.

[iarna]

Any time I think about es6 modules I get uncomfortable feelings around forward compatibility for published modules. The current status quo, which is that you write your modules in whatever and compile to es5 at publish time, pretty much guarantees forward compatibility.

That is, not only do newer versions of node & npm continue to work with older modules (backward compatibility) but older versions of node & npm will work with newer modules.

This is critical because our usage data shows that users have been very slow to move forward. In fact given the community growth rate, I my gut feeling is that much of the migration to v4+ has been mostly new users.

I would very much like any solution to work with npm@2 and Node 0.10 (or maaaybe 0.12). That is, the package.json should be setup such that if I want to make a multi-mode module w/ both es6 and es5, that an older npm can install the module and an older node load it.

[iarna]

As an aside: npm's support guarantees are broader than Node.js'. Currently npm only supports two major releases of npm itself. But it supports using them on versions of node going all the way back to 0.8. We're very much hoping to drop 0.8 support this year but inevitably I suspect we will always be supporting more versions of Node.js than Node.js does itself.

[martinheidegger]

@iarna I totally agree with that sentiment and I see this proposal as best proposal yet to actually support this:

1. CommonJS would not be a second-class citizen. It would just be one flavour.
2. I see this proposal as simplest, cleanest way to make ES6 modules also support CommonJS.
3. I do not see any backwards-incompatibility. New ES6 modules would, in-any-case, be incompatible but system-wise there is no incompatibility?!
4. By allowing variants of packages the amount downloaded by npm would be significantly less than by the proposed .mjs solution.

[martinheidegger]

(Note: I see this proposal 4f a little bit like the 32bit and 64bit packages that are distributed in linux package managers)

[iarna]

32bit and 64bit packages that are distributed in linux package managers

There's nothing intrinsically wrong with that, but pushing it into an already established ecosystem feels untenable to me.

I must be missing something… I'm gonna write through this... so you say:

This way we have two packages stored in the npm registry. The old packages would all be treated as commonjs.

So that implies, I guess, that the registry would return package metadata including something like this:
(Currently it includes only shasum and tarball entries under dist. This is how npm knows where your actual package data is)

      "dist": {
        "shasum": "8efbd89e4cab125eef5f47360f55c5bf141aa415",
        "tarball": "https://registry.npmjs.org/abraxas/-/abraxas-0.1.0.tgz",
        "shasum-es6": "some other sha",
        "tarball-es6": "https://registry.npmjs.org/abraxas/-/abraxas-es6-0.1.0.tgz",
      },

(Or maybe "dist": { "variants": { "es6": { "shasum":…, "tarball":…} } }, the specifics aren't important, I don't think.)

Or if the module ONLY supported es6 then it'd return:

      "dist": {
        "shasum-es6": "some other sha",
        "tarball-es6": "https://registry.npmjs.org/abraxas/-/abraxas-es6-0.1.0.tgz",
      },

Except I'm concerned that would set older versions of npm on fire, as they make lots of assumptions about shasum and tarball being set, so it wouldn't have an opportunity to give a reasonable error message. Sure it wasn't gonna work for them, but I don't want to introduce modules to the ecosystem that just by existing are guaranteed to break older npms.

And of course, if you swap that around and make es6 the default, then even multi-mode packages make npm catch fire.

[iarna]

(None of this is to even speak about all-in-one-file packages, which are just JS files with the package.json embedded in the top. But I would be very happy to call those cjs ONLY.)

[martinheidegger]

@iarna I see two different implementations but neither of them would touch the response of the npm server:

One would be using HTTP headers: The npm client requests it either with an additional header NPM-SYNTAX: es6. An older service would return a common.js package which can be checked by the client after unpacking (if the service returns es6). A newer service would return the es6 package (same structure as given now).

Another implementation would be using a different endpoint. If the endpoint doesn't exist there would be an error from old servers. And npm would fallback to commonjs and the old endpoint.

Edit: Of course the old endpoint would still response with the old data. In order for old versions of npm to work as expected: In short this is just an extension of the current server.

[martinheidegger]

@iarna Wouldn't that work?