When to make --default-type=module the Node.js default

[GeoffreyBooth]

Now that nodejs/node#49869 has landed, and we need to figure out what to say about it for the 21.0.0 announcement, we should agree on a tentative plan for when the default value of --default-type flips from commonjs to module. That would be a semver-major change.

Obviously we can’t announce a certain date or version, since we don’t know what issues may arise during 21.x as people test out --experimental-default-type=module and when we might consider it stable, but assuming things go fairly smoothly then we could conceivably make the flip in 22.0.0 in April 2024. I think that would probably be preferable to 23.0.0 in October 2024, which would mean that the first LTS line with ESM by default wouldn’t begin until 24.x in April 2025, which is a long way off.

I think for the health of the project and the greater JavaScript ecosystem it’s important for Node.js to shift to an ESM-first mindset. We will continue to support CommonJS as a first-class module system, but it becomes opt-in rather than opt-out, conforming Node with other runtimes and modern tools.

This is primarily a change for new users. Most users learning JavaScript today are learning it with import/export syntax and get frustrated when Node doesn’t run their code right away without messing with configuration settings. I think it’s important that we prioritize the UX of these users. The users still writing CommonJS today are among our most sophisticated and experienced, and will be able to figure out how to opt into the old mode via --default-type or the "type" field.

[GeoffreyBooth]

For those wondering, this is what the migration paths look like:

• For users already writing ESM, either via package.json "type": "module" or .mjs, nothing needs to be done.

• For users writing CommonJS apps or libraries, where their code is in a folder that has a package.json file, they would need to add "type": "commonjs" to that package.json file if they haven’t already; or run Node with --default-type=commonjs.

• For users writing “loose” .js or extensionless JavaScript files, that have no package.json file in the folder alongside them or in any parent folder, the user would need to do one of the following:

  • Add a package.json file with "type": "commonjs".
  • Rename the file to end in .cjs.
  • Make the path of the file into a symlink to a file that either ends in .cjs or is under a folder with a package.json file with "type": "commonjs".
  • If the file is an executable script with a shebang line and their system supports env with the -S option, change the shebang to #!/usr/bin/env -S node --default-type=commonjs.
  • Use some shell scripting to have the script passed to Node via STDIN with --input-type=commonjs.

Or, of course, they could refactor code from CommonJS to ESM, but I’m not sure that needs to be presented as an option. This is nothing like Python 2 to 3.

The other significant migration is of packages that rely on monkey-patching the CommonJS loader. The authors of major instrumentation libraries and of Yarn Plug-n-Play have been very active engaging with the loaders team in ensuring that the module customization hooks can be used to support their use cases. Those APIs are already in “release candidate” stage, just baking to see if anyone reports any bugs before we declare them stable. Regardless of the --default-type flip, though, it’s important for the maintainability of the modules subsystem that anyone who currently monkey-patches moves away from that pattern and onto public APIs. The need to support monkey-patching use cases has severely hampered our ability to do significant refactoring or improvements of the modules code. Substantial performance achievements, such as could be attained by moving parts of the modules code into C++, can only be done if we can break the already unsupported monkey-patching use cases. Having an official announcement that the ESM loader will become the default module loader in a future Node version will hopefully be what corporate teams need to prioritize the work to move away from monkey-patching.

[ljharb]

Doesn’t this mean that users writing a library would have different default behavior for themselves than for when they’re installed in someone else’s node_modules?

[mcollina]

Doesn’t this mean that users writing a library would have different default behavior for themselves than for when they’re installed in someone else’s node_modules?

Yes. I think changing that is going to a be a massive breaking change that would break everybody. Most modules would need to keep working with the flag.

Other alternatives are

1. deprecate typeless packages outside of node_modules
2. ask npm init and the like to include a type property by default

I would personally prefer to ship this in 23, having v24 the first LTS with the change, but I'm not opposed to an accelerated timeline.

If we decide not to flip the switch in v22, we could ship some form of warning in v22 for typeless packages outside of node_modules, taking a similar "iteration" approach that we used in the past.

Something to consider is the coordination with the TS team, as this will impact them and their resolution algorithms.

Of course, Loaders should be stable by then and no critical bugs should be there (or in ESM) to do the switch.

[GeoffreyBooth]

Yes. I think changing that is going to a be a massive breaking change that would break everybody. Most modules would need to keep working with the flag.

Let’s investigate this, shall we? Let’s go through some scenarios:

1. ESM-only packages, like node-fetch: The flip has no effect, on either the package author or its consumers.

2. CommonJS-only packages, like express, where the author is developing the package using a pre-flip version of Node and the consumer is running post-flip Node: The flip has no effect, on either the package author or its consumers.

3. CommonJS-only packages, like express, where the author upgrades to the post-flip version of Node. Assume that the package uses .js extensions and the package.json file has no type field: The first time the author tries to develop their package after upgrading, it will fail to load because .js is ESM now. The error message will prompt the author to add "type": "commonjs" to their package.json. The author does so and is back to developing. No effect on consumers of the package.

4. A new ESM-only package, where the author starts writing the library in .js files with a package.json file that has no "type" field: (Assume for the sake of evaluating worst case scenarios that we get no help from npm or build tools; they function as they do today.) The author publishes their package to npm, thinking all is fine, but any consumers of the package—both using pre- and post-flip versions of Node—see errors when they try to consume it, because in all versions of Node a type-less package under node_modules is interpreted as CommonJS. This published version of the package is effectively broken for all users, which would surely trickle back to the author, who would add "type": "module" to the package.json and publish a new version that would work for everyone. Because this is a new package, it wouldn’t have millions of consumers; the mistake would be rectified very early on, before it has a chance to grow its popularity.

So basically, package authors would need to learn to add the type field to their package.json files, and hopefully we get some help from npm or other tools reminding them to do so. But the consequences of them forgetting to do so are low: a package either works for all users or is broken for all users, which is really what we want, because broken versions won’t find any adoption. I don’t see any scenarios where either authors or consumers would use --default-type=commonjs; it’s easier and more permanent to just add "type": "commonjs". Are there any other plausible scenarios I haven’t thought of?

[benjamingr]

Excellent summary @GeoffreyBooth that really clarified how migration would look and it sounds very reasonable.

We should also contact npm and ask for:

• Adding "type": "module" by default to npm init (as suggested by Matteo)
• Check that type is defined in the package.json during npm publish

@nodejs/npm does that sound reasonable?

Edit: opened an issue with npm: npm/cli#6855

[GeoffreyBooth]

We should also contact npm

For reference, adding type to npm init was proposed and rejected in 2019, then proposed and rejected again in 2021, then proposed again in 2022 (still open) and again in 2023. Of course, all of those prior discussions happened before the proposal of Node changing its default, so I am optimistic that a new outcome might be possible once we make an announcement on our side.

[wraithgar]

Am I following this correctly, that every package that's currently published will stop working once this goes into effect, and only newer versions of those packages that have this field defined will work?

[benjamingr]

@wraithgar no, the issue is with the default changing outside node_modules, the issue is when people publish new versions of packages without specifying type and relying on the default being esm - which is why we want the small npm enhancement to let people know their published code requires "type" to work as it did before publishing.

[joyeecheung]

I think this means if you have a simple script like this:

// test.js
require('fs')

This would stop working once the flag is flipped, unless you change the file name or add a package.json. I believe there is an enormous amount of utility scripts and tutorial examples that would be affected, now the question is do we want to create this havoc in all those scripts already out there and create a python2-like situation...

[joyeecheung]

Also, wouldn't this break most of our tests unless we do something? (and to that extent, it could probably break a lot of other tests in other projects that are run in a similar "loose" fashion). It could also force all projects not yet migrated to do something, while we already know that there are a lot of simple projects that are not actively maintained but are still heavily depended upon in the ecosystem - or the Node.js ecosystem once favored very simple packages like this, so if an author has tons of very simple packages that do not need much maintenance since Node.js v0.x days (because who would've thought that Node.js is going to break such simple use cases), they will now be forced to update all these small packages. And many of the old reproductions of confirmed bugs would also need updating...

[GeoffreyBooth]

Also, wouldn’t this break most of our tests unless we do something? (and to that extent, it could probably break a lot of other tests in other projects that are run in a similar “loose” fashion). It could also force all projects not yet migrated to do something,

The “do something” is what I wrote above as the migration path for CommonJS projects with no type field: add "type": "commonjs" to package.json, or create a package.json containing only { "type": "commonjs" }. That’s it.

People seem to keep pointing at Python 2-3 as the only example of a migration, but this is nothing like that: that required everyone to refactor their code, whereas this requires no one to refactor their code. They can if they want to, but this is a configuration change not a syntax change.

Yes the “shell scripts” case is the most awkward, because people often lack package.json files for those. But as I’ve written elsewhere, those scripts aren’t our primary use case and they shouldn’t be prioritized over application developers. Per https://nodejs.org/en/about, “Node.js is designed to build scalable network applications.” Just about every application framework has instructions in ESM syntax. TypeScript has exclusively supported ESM syntax since its inception, if I’m remembering correctly. We’re more out of sync with the ecosystem by requiring ESM to be opt-in than opt-out; while there will be some old tutorials and examples that become outdated after this change, there is far more that’s the reverse. Having that code work by default in Node.js will improve the new user experience and bring us into conformity with the broader ecosystem.

[joyeecheung]

that required everyone to refactor their code, whereas this requires no one to refactor their code

Requiring everyone to add a new file or change file name is no less a problem, I would say, especially considering:

1. A lot of tools that run scripts would need to take this into account
2. git history problems and 404-like problems caused by bulk file renames
3. It really doesn't matter what needs to be changed. A lot of the times people run into issues in their dependencies and they just have no idea how to deal with them and do not want to invest the time. They only want simple things to keep working without breakages i.e. stability. Any change that affects what the hello world everywhere tells you to do is disruptive.

Per https://nodejs.org/en/about, “Node.js is designed to build scalable network applications.”

First of all I don't think whatever we put down on that web page is regarded as our principal. There are also a lot of things on that web page is out of sync with how Node.js is used in the wild. Also on that note, the hello world on that page is still using ComomnJS.

Second of all, claiming that "Node.js is designed to build scalable network applications." does not mean Node.js is only designed for that and we should disregard other use cases, for example, tooling. Also, can one lose the ability to build a scalable network application just because the default entry type stays the way it is since >10 years ago? I doubt how much difference the default entry type makes if that's the only thing we ever care about.

But as I’ve written elsewhere, those scripts aren’t our primary use case and they shouldn’t be prioritized over application developers.

I think above all Node.js prioritizes stability and not breaking existing code unless absolutely necessary.

while there will be some old tutorials and examples that become outdated after this change

Running a .js file that allows you to use require() to load builtins is something that Node.js supports since >10 years ago, it's everywhere in the tutorials and blog posts and books, and it has also been in our hello world examples since the initial days of Node.js. There are very few things that can be even more stable than this in the Node.js API contract. I don't think they are just "some old tutorials", personally I'd be convinced that > 80% of the Node.js tutorials out there still teach you to do this and maybe only 10% of them can ever be updated. There would be countless people counting on this to continue to work. I don't think this should be simply dismissed as "not something worth to be prioritized".

[joyeecheung]

Out of curiosity, I googled "get started with Node.js", and the top results all rely on being able to run a .js file that require():

1. https://nodejs.org/en/docs/guides/getting-started-guide
2. https://www.w3schools.com/nodejs/nodejs_get_started.asp
3. https://www.freecodecamp.org/news/introduction-to-nodejs/
4. https://www.simplilearn.com/tutorials/nodejs-tutorial/getting-started-with-nodejs
5. https://nodejs.dev/en/learn/
6. https://www.pluralsight.com/guides/getting-started-with-nodejs
7. https://code.visualstudio.com/docs/nodejs/nodejs-tutorial (assumed in screenshots)
8. https://levelup.gitconnected.com/the-ultimate-guide-to-get-started-with-node-js-4ce54579ceb7
9. https://cloud.google.com/nodejs/getting-started (it's actually started by google cloud, I think cloud providers need to go through the hurdle of upgrading their infra and documentations for that change too, it would be even messier factoring in support for multiple Node.js versions)
10. https://developer.ibm.com/learningpaths/get-started-nodejs/explore-nodejs-concepts/
11. https://developer.mozilla.org/en-US/docs/Learn/Server-side/Express_Nodejs/development_environment

I've stopped looking after glancing over the top few and none of them is teaching people to write the entry type as ESM (I didn't scroll to bottom or click around, but at least CommonJS is always the first thing being taught). And this is just the English version...

[GeoffreyBooth]

Yes, many guides to Node.js were written a long time ago. Most people don’t develop “Node apps” per se anymore, they develop in frameworks that build on Node, or with things like TypeScript that use Node. If you look at those docs, most use ESM syntax. @mhdawson do we have any survey data to put some numbers on how people use Node, or what they use it with, or what their preferences are regarding modules?

Here’s what happens if you use ESM in a CommonJS context today:

mkdir test && cd test
echo 'import "fs"' > entry.js
node entry.js
(node:51228) Warning: To load an ES module, set "type": "module" in the package.json or use the .mjs extension.

This would be the inverse experience when we flip the default, in that the message would say something like “Warning: To load a CommonJS module, set "type": "commonjs" in the package.json or use the .cjs extension.” It’s to the point and tells you how to fix your problem. It’s not like the code runs but there are silent errors or unexpected behaviors. We should probably improve this by adding a link to a nodejs.org page explaining in more detail, but even the brief version tells you what to do.

A lot of the times people run into issues in their dependencies and they just have no idea how to deal with them and do not want to invest the time. They only want simple things to keep working without breakages i.e. stability. Any change that affects what the hello world everywhere tells you to do is disruptive.

Because the new mode doesn’t affect anything under node_modules, any dependencies installed through common methods like npm install would be unaffected by the flip.

[joyeecheung]

Most people don’t develop “Node apps” per se anymore, they develop in frameworks that build on Node, or with things like TypeScript that use Node.

I don't think we should base our decision of such a wide-spread breakage on pure impressions. We need evidence that this is not going to break, say, more than 5% of Node.js users (no matter they are writing code in a framework, or just using Node.js as a runner/tool, or using Node.js inside another environment e.g. cloud provider, desktop environment, or using Node.js to run mod/plugin script), before we can break an API contract that has been by design for >10 years.

Because the new mode doesn’t affect anything under node_modules, any dependencies installed through common methods like npm install would be unaffected by the flip.

By dependencies, I mean that the Node.js application is used as a runner/build tool/a component as part of the project, while developers/users do not necessarily know how to deal with it (e.g. our configure.py kind of use case).

[GeoffreyBooth]

I found the data: https://github.com/nodejs/next-10/tree/main/surveys/2023-04

And in https://github.com/nodejs/next-10/blob/main/surveys/2023-04/Node.js%20Survey%20open%20ended%20answers.pdf, question 18 is “If you wish to use ESM in an existing application, what have been the pain points or blockers preventing you from doing so (if
any)?” and there are hundreds of responses so it’s not really summarizable, but you can get a sense from the responses of what people are trying to do.

[benjamingr]

I don't think the comparison to Python is fair at all. This is like Python 2/3 if Python 2 code kept working.

Namely:

• In Python 2 -> 3, old Python code and libraries had to be migrated - this is not the case here. Node.js wiil keep running CJS code it just won't be the default. Packages and dependences won't break etc.
• Users had outdated resources in tutorials - this is not the case here both because the language isn't changing only the module system is and because the fix (literally adding a line to a file or passing a CLI flag) will be helpfully shown to the user if they forget it.
• It caused ecosystem fragmentation - which this change won't do by design.

Python 2->3 would have been if we removed CJS entirely, removed all the non-promise callback APIs, removed all deprecated APIs and made breaking changes to promise ones and then created an incompatible Node.js binary effectively making "two Nodes". This on the other hand is just changing the default outside node_modules

[joyeecheung]

I found the data: https://github.com/nodejs/next-10/tree/main/surveys/2023-04

I do not think the survey is created in a way that gathers data about how a widespread breakage in the existing code matter for users. It is true that people may want and do work on transitions when they have time, but it is also true that widespread breakages are disruptive and must be done with care. The right questions should be:

1. Are you still maintaining CommonJS code?
2. Would you accept a change that breaks running node hello.js when hello.js is a CommonJS entry point that require() without a package.json, in order to speed up adoption of ESM?
3. Would you accept other breakages in existing CommonJS applications in order to speed up adoption of ESM?
4. When do you think is the right time to make ESM the default entry type by breaking the existing way of running CommonJS applications?

Even in the survey, you can see that more than 30% of the respondents do not plan to migrate to ESM in the near future. I doubt how many of our users would be happy to trade a breakage in a widespread way of running CommonJS in order to speed up migration to ESM. It's not impossible to do forever, but I think we are still very far away from that point. If this only breaks running loose CommonJS files, and if that was really just a niche use case as you described, how much value would this actually bring to the speedup of ESM migration and why is it so urgent that it need to be done in a year? Either this is very widespread and moving on would be an important milestone, or this is just a niche use case so this is one of the last thing that needs to be taken care of. I don't think it can be both niche and an important milestone.

[joyeecheung]

Yes, many guides to Node.js were written a long time ago.

If you check the search results, many of them are either written or updated within the last two years.

Most people don’t develop “Node apps” per se anymore, they develop in frameworks that build on Node, or with things like TypeScript that use Node.

If that's really the case, why would it matter how we run the loose scripts by default? The frameworks should take care of this for them and how the entry point is interpreted by default is the last thing they need to care about. This is also contradicting what the OP says:

This is primarily a change for new users. Most users learning JavaScript today are learning it with import/export syntax and get frustrated when Node doesn’t run their code right away without messing with configuration settings.

Either most users get started by running a loose script and need to configure something to make it run as ESM by default, or they get started by writing code using some framework and the configuration should be done by the framework. I very much doubt how many JavaScript tutorials would teach readers to run ESM code in Node.js without any extra configuration and let them assume that it would just work, but we have evidence that most top Node.js tutorials are still explicitly teaching people to write a CommonJS entry point and run it without any additional configuration. I don't think we should invalidate existing working tutorials to prioritize non-working tutorials that may or may not even exist. Following a tutorial and finding out that what it says can run does not actually run is much more frustrating than trying out code in an environmnet that the tutorials do not teach (because it's in fact not runnable) and finding out that it doesn't work.

[GeoffreyBooth]

If you want to, you can make any conclusions you want from that survey data. You could look at the 65-35 split of users already using ESM and say that more than two-thirds of our users are already using ESM, why shouldn’t it be the default? Or you could look at the 35% and say but this is still a large group, too large to inconvenience, etc.

I think one fair conclusion that can be drawn is that this data does prove that far and away our users are application developers. They may run some loose shell scripts, but by and large I think it’s safe to assume that the vast majority of our users are probably doing most of their Node development in apps with a package.json file and dependencies in node_modules. These users either won’t notice the default flip (if they’re in the 65-70% of ESM users) or they’ll have to add "type": "commonjs" to package.json.

I do not think the survey is created in a way that gathers data about how a widespread breakage in the existing code matter for users.

I think the framing of “widespread breakage” or “Would you accept a change that breaks” or “Would you accept other breakages” is very leading. Not all breaks are created equal. As @benjamingr wrote, there’s a big difference between adding "type": "commonjs" to package.json and refactoring from Python 2 to 3. And besides, we’re not in a position to run another 1000-plus-person survey again anytime soon. This is the data we have.

Every breaking change, like any change, has costs and benefits. The question for us is whether the benefits of this one outweigh its costs. I think they do: I think it really matters that Node join the rest of the ecosystem in prioritizing the standard module format, that we provide the best experience for users in that mode, and that we make it easier to write standards-compliant JavaScript. I think the requirement of a one-line JSON file change, or adding a flag, is a very minimal cost to impose on a minority of users in order to realize these benefits.

If you check the search results, many of them are either written or updated within the last two years.

Well then that would imply that they’re still maintained, and if we announce that this change is coming, they’ll be updated before April 2024. Regardless, we can add a link to our own tutorial in the error message that someone would get when they run CommonJS syntax in ESM mode. I don’t view this as a blocker.

[joyeecheung]

And besides, we’re not in a position to run another 1000-plus-person survey again anytime soon. This is the data we have.

I don't think "we are not going to do a large-scale survey" is a good response for the lack of data to support an important breakage like this. If we can't run a survey anytime soon, then we should wait until we are ready to run another survey, gather data about how breaking the breakages are, before we make a decision to break something that's been working for more than 10 years and in most top Node.js tutorials out there.

I think the requirement of a one-line JSON file change, or adding a flag, is a very minimal cost to impose on a minority of users in order to realize these benefits.

If the weight of this breakage is enough in the prioritization, then it should be a very commonly used thing that would break a lot of people, hence a huge cost. If it only breaks a minority of users, then it should not have much weight in the prioritization and it's not really worth breaking. I don't think it can only have a minimal cost while also bring a lot of weight in the migration.

[joyeecheung]

Well then that would imply that they’re still maintained, and if we announce that this change is coming, they’ll be updated before April 2024. Regardless, we can add a link to our own tutorial in the error message that someone would get when they run CommonJS syntax in ESM mode. I don’t view this as a blocker.

I think this is a blocker. We need to at least make sure that the top Node.js tutorials out there are teaching people to write ESM by default before we create a sense of emergency, and for example, >90% of our users are ready for the flip. If that doesn't happen, then we should postpone the flip. It's fine to announce that a change is coming, but I think a one-year notice is just too short for breaking something that's been working and assumed everywhere for more than 10 years.

[GeoffreyBooth]

I think what you’re saying is that this is a big deal because it affects so many users; up to a third of our user base. What I’m saying is that it’s a small deal because all it takes is a minute of effort on those users’ part to update their projects to work in the new mode. Both things are true.

If there’s a way to run another survey, sure, that would be great. @mhdawson? But I think we need to write the 21 announcement soon, so I find it hard to imagine we’ll have results in time to inform what we write. Even if we wanted to I don’t think we can promise anything in particular, because no one can predict what PRs get blocked, but I think we can say that we intend to propose flipping the default of --default-type in 22 assuming that the flag is stable by then and issues people report have been resolved. The stronger the statement we make, the more feedback we’ll get; we can even include a survey with the announcement, and that might give us data to make an informed decision by the time 22 is getting ready. And no one will hold us to our intentions; we can always decide that 22 is too early, whatever, and let it slide to 23 or 24.

[joyeecheung]

I think "we'll flip in 22" is a message that shows how little we care about the stability of this software. This is not a niche API that only a fraction of users know about. It's something that has been working and assumed everywhere for more than 10 years, and it's something that almost all Node.js beginner tutorials out there has been teaching. A one-year notice for that is just too hasty. The message we should give is that "we'll flip when there's sufficient evidence that less than X percent of our users would be affected by the breakage" and ideally send a survey gathering data about the breakage (basically, "if you run the Node.js code you publish or maintain with --default-type=module, would it break?" and "how many dependent of your code would you expect to be broken by this flag?" if the answer is yes).

[GeoffreyBooth]

The message we should give is that “we’ll flip when there’s sufficient evidence that less than X percent of our users would be affected by the breakage”

This kind of statement has no acknowledgement that we’re just changing a default configuration setting, and that it’s easy to continue working in the previous mode if desired. I read phrases like “affected by the breakage” as implying that affected users are just irreparably broken, like they can never upgrade. This is not that.

It does imply that we strongly prefer stability and backward compatibility, which we do, and I think that’s one of Node’s strengths. But a statement that says that we need to wait years to catch up with the ecosystem, to embrace a standard that was released in 2015, makes Node seem hopelessly stagnant and stuck in the past. I’d like to think that we’re a more forward-looking and faster-moving project than that.

I think we can draft a statement along the lines of this:

In 21.0.0 we ship a new flag --experimental-default-type that does {explanation here}. We encourage feedback about the new --experimental-default-type=module mode, and whether it should be made the default in a future major version of Node.js. For users not already writing ES module syntax, the migration paths look like {explanation here}. The ESM mode might become the default as early as 22.0.0 if users don’t report too many issues with the flag and it can go stable by then, and if survey responses indicate that the user base would welcome such a change. Please take this survey to let us know what you think: {link}

[joyeecheung]

But a statement that says that we need to wait years to catch up with the ecosystem, to embrace a standard that was released in 2015, makes Node seem hopelessly stagnant and stuck in the past.

Even browsers do not support import in a <script> tag without type="module", I would not blame a JS runtime for following what browsers do by default for stability. I think this is just the unfortunate result of the design of the standard, which should've considered the impact before getting released. This is the just reality that we have to live with. I don't think making Node.js less stable by breaking something that's still been widely assumed everywhere is the right answer.

I think we can draft a statement along the lines of this

I would say this statement only looks appropriate when the flag is no longer experimental. It still looks careless to release a timeline for such a widespread breakage based on an experimental feature that just gets out and is not yet even tested in the wild.