[RRFC] Opt-in solution to support comments in package.json (among other features)

[TheJaredWilcurt]

Edit: I simplified this section and incorporated all comments and concerns up to this point. In hopes that this can finally proceed to be implemented.

Motivation

It is now common for modern frontend/backend applications using Node to have dozens of dependencies and npm scripts, and even configurations for tools like ESLint, Jest, etc. stored in the package.json directly.

This can make the maintenance of a project more difficult when it comes to how and were to document things related to the package.json file. Most commonly things around why dependencies were added, how dependencies are related, why a dependency is pinned to a specific version, and reasons for specific npm scripts. Currently package.json auto-alphabetizes dependencies, removing any intentional grouping of like-items (Example: babel-jest, jest, vue-jest would logically be grouped and related to unit testing, but with a single npm install they are mixed in with the rest).

There is a need for a more permissive format for writing a manifest file.

Opt-in, Backwards compatible, Solution

When the npm executable is called and needs to access the package.json (such as npm i, npm run, npm publish, npm start, npm t, npm audit, etc) it will first check for a package.json5 file. If it exists, it will process it and override the existing package.json file, then proceed as usual.

Benefits:

• Fully opt-in - Projects that need the additional capabilities of json5 have easy access to them, while those that don't need them can continue using package.json as they always have.
• Fully backwards compatible - Since the package.json is always kept in sync with the package.json5, nothing needs to change for Node, or any other tooling. Everything still works.
  • All existing tooling that reads from the package.json continues to work exactly the same.
• People seem to miss that last bullet point, so I'm repeating it here: NO CHANGES TO NODE OR JAVASCRIPT OR BROWSERS ARE REQUIRED. This is a change npm can make unilaterally.
• Features:
  • Complete control over the file
  • Single (//) and Multi-line comments (/**/)
  • Order of dependencies is preserved
  • Allows for empty lines to add visual white space between sections
  • Unquoted keys (scripts: {)
  • Use of single quotes (lint: 'eslint')
  • Trailing commas
  • and more
• A brighter and happier future for all 😃

Edge-cases

• Users of older versions of npm working in repos using this feature would need to either:
  • Use a tool like Volta to automatically ensure their node/npm version matches what the repo expects
  • Manually upgrade to a newer npm version that handles json5 automatically
  • Manually run npx json5 package.json5 to process the file by hand

Downsides (these are all very minor)

• npm would need to ship with an additional dependencies (json5). It already ships with over 400, so this is a non-issue.
• Tooling that writes to the package.json directly would require repo maintainers to manually move those changes over to the package.json5 file. Though updating these tools to support the new format would be very easy in almost all scenarios. Example:

   // Start of file
   const fs = require('fs');
  +const JSON5 = require('json5');
  
  -const manifest = require('package.json');
  +const newManifestFormat = fs.existsSync('./package.json5');
  +const manifest = newManifestFormat ? JSON5.parse(fs.readFileSync('package.json5')) : require('package.json');
  
   // No other code changes until the end of the file
  +const stringify = newManifestFormat ? JSON5.stringify : JSON.stringify;
  -const output = JSON.stringify(manifest, null, 2);  
  +const output = stringify(manifest, null, 2);  
   fs.writeFileSync('./package.json', output);

• If Node ever officially adopts a new format to replace the package.json; they may pick something other than JSON5 (TOML, HJSON, YML, whatever). npm would then need to deprecate JSON5 support as the community transitions over to the same format. I don't forsee Node ever doing this, and if they do npm's use of JSON5 would likely drive them to adopting the same format, avoiding this unlikely downside alltogether.

References

• pnpm has allowed the usage of package.json5 as a replacement for a package.json. However it does not compile a package.json during use, leading to members of the community to decry it as breaking backward compatibility. Thus it exists, but is not recommended by the docs.

[ljharb]

1. JSON was absolutely originally designed for humans, not for machines - it's JavaScript Object Notation, for JavaScript developers.
2. JSON in fact has comments: both the conventional:

{
  "//": "this is a comment about foo",
  "foo": "this is foo's real value"
}

and the actual:

{
  "foo": "this is a comment about foo",
  "foo": "this is foo's real value"
}

3. See Use JSON5 for package.json feedback#56 for all the reasons why an alternative package.json format would be massively disruptive to the ecosystem, not the least of which is, JavaScript itself lacks a parser for anything besides pure JSON.

[TheJaredWilcurt]

My request addresses all concerns mentioned in that discussion.

Arguments to stay with the current format:

1. Problem: Node can only parse JSON
   • Solution: The new system is opt-in and would automatically produce a traditional .json file for Node/backwards compatibility. This completely solves this issue.

Your argument to not allow a more permissive, opt-in, format does not fix any of the problems identified.

1. Comments are strictly not allowed in JSON. There are unofficial hacks, but they are just that hacks.
   • "ManifestComments": ["coment", "comment"] - This hack is accessible via JSON parse, may be desirable in some cases.
   • "//": "comment" - Only the last instance in a section is accessible via JSON parse.
   • "foo": "comment", "foo": "value" - Comment is completely replaced by real value, my be desirable in some cases.
   • My solution would permit these hacks to continue to be used if desired, though it also gives far better options to replace these approaches.
2. JSON was not designed with humans in mind, JavaScript was. JavaScript supports syntax features people would like to use in their manifest, but are unable to because of the syntax limitations put in place to make JSON faster to parse for machines:
   • Trailing commas
   • Single quotes
   • Keys without quotes
   • Comments
3. We know that this is the case, because YML has become the defacto standard in CI configs. YAML, like JSON5 or HJSON, is a meta-language that can be converted directly to JSON. It has an even more permissive syntax:
   • Does not require commas at all for separation
   • Infers strings if text is not wrapped in quotes
   • Allows for # to be used for comments
4. The current package.json format does not permit adding in empty lines as separation between dependencies. (removed on npm install)
5. The current package.json format does not permit the logical (non-alphabetical) grouping of dependencies. (removed on npm install)

My solution fixes all of these problems, and has no negative side effects. Nothing with Node needs to change at all. npm would just need to ship with an additional package built in, and perform 1 extra step before certain tasks.

[ljharb]

None of those limitations are part of json to make it “faster for machines” - that was never part of the motivation for designing json in the first place.

Separately, this isn’t purely opt-in - there are many tools in the ecosystem beyond npm itself that write to package.json, so if a project auto generated that file from an alternative manifest, all those tools would need to be able to write to that alternative manifest instead.

[TheJaredWilcurt]

I see no value in the continued discussion of the original intent of JSON's design. The fact that both of the alternatives I've mentioned have decided to market themselves as "JSON for humans" says enough. At this point it's pedantics.

If third party tools want to support this opt-in feature, they can. If a user uses those tools that do not support the new format, they do not have to opt-in to the new format and continue to use the package.json as their source so that it can be compatible with those tools.

Just because some 3rd party tools would not support this out of the box, that doesn't mean we should prevent everyone from having the benefits this opt-in system gives.

Most 3rd party tools would have a very easy upgrade path to support the new system, only requiring a slight change at the start and end of the code. Something like this:

 // Start of file
 const fs = require('fs');
+const JSON5 = require('json5');

-const manifest = require('package.json');
+const newManifestFormat = fs.existsSync('./package.json5');
+const manifest = newManifestFormat ? JSON5.parse(fs.readFileSync('package.json5')) : require('package.json');

 // No other code changes until the end of the file
+const stringify = newManifestFormat ? JSON5.stringify : JSON.stringify;
-const output = JSON.stringify(manifest, null, 2);  
+const output = stringify(manifest, null, 2);  
 fs.writeFileSync('./package.json', output);

These tools would remain backwards compatible, but also support the new opt-in system. The tools that do not make this simple change will continue to be used on projects that do not opt-in to the JSON5 format.

[ljharb]

json5 doesn’t ship with node, or in browsers - having to add an external dependency to be able to process the contents of a package manifest is a very large barrier to entry.

If node, and/or browsers, were convinced to ship an alternative parser/serializer in core, then I’d say whatever they ship is a strong contender for an opt-in alternative - lacking that, i fail to see how a few niceties are worth the churn caused by allowing more than one format.

[TheJaredWilcurt]

This is a tactic called "Moving the goal posts".

"We should allow a more permissive format to solve these known problems and to eliminate the need for hacks."
We cannot, because it would break Node, use hacks instead, that do not solve most of the problems
"Here is a way that is fully backwards compatible and does not require any changes in Node. It solves all problems, with no side effects"
It would hurt 3rd party tools
"Not really, here's why"
You should personally convince all browsers, Node, and the TC39 to change the built in syntax of the language so that the npm project can have a second opt-in manifest file.
"That is an unreasonable ask for something unrelated to those projects which realistically does not and should not block this from moving forward"

Node and browsers won't add it in unless the TC39 approves. The TC39 will not approve changes to JSON because:

• There are hundreds of thousands of libraries across hundreds of programming languages that support JSON for the purposes of transferring data between systems. These would all need to be updated to support the new format, and to communicate the version they work with.
• JSON being a restrictive and limited subset of ES3 is intentional to make it easier and faster to parse, giving it performance benefits
• There is no benefit in doing this, as almost all usages of JSON are to transmit data from machine to machine, application to application, system to system. Humans creating the data are rarely doing so in the JSON format, and humans consuming the data are rarely consuming it in the raw JSON format, instead seeing the data abstracted through a UI.
• The one exception to this being configuration files like Node.js's package.json, or DevOps/CI tools, which support JSON, but often allow the use of alternative formats like JSON5, HJSON, or YAML since they are using these files as a human interface.

When installing npm it already ships with hundreds of packages in its node_modules. So adding an additional package like hjson or json5 is not new or novel.

The implementation of this feature has no unknowns and would be a small engineering effort.

The documentation of this feature would require minimal effort.

The only thing preventing it would be bike sheds about which technology to use (JSON5, HJSON, or something else).

The negatives of this are very minor and do not outweigh the benefits.

Negatives:

• Additional thing for users to learn
• Additional file in the root of a project
• Requires a new dependency be shipped with npm (which already has nearly 400 it ships with)
• Could cause some minor ecosystem churn as projects begin to opt-in to this feature, though would really only effect 3rd-party tooling
• People on older npm versions using a project with the new manifest would need to run an npx command to process the source, or would need to update npm. This is mostly resolved by the common adoption of nvm by the community.
• 3rd-party tools designed to edit the package.json would need updated to support the feature, though supporting it is optional
• Would be very difficult to remove this feature later (deprecating the feature would require project maintainers to manually migrate back to regular JSON, or to whatever new system would replace it which would presumably be the cause of the feature deprecation)

Benefits:

• Allows for organizing and grouping dependencies logically (not alphabetically)
  • So related dependencies (babel-jest, jest, vue-jest) can be placed next to each other
• Allows adding empty lines to visually separate dependencies or scripts into related groups
• Allows for comments to document:
  • How dependencies relate to each other
  • What dependencies should be updated together
  • Why a dependency is pinned to a specific version, what is blocking it from being updated
  • When to use certain npm scripts, or what they are used for
• Friendlier syntax:
  • single quotes
  • keys without quotes
  • dangling commas in objects/arrays
  • single or multi-line comments
  • Multi-line strings
  • JSON5: Object keys as Identifier names. { cat } becomes { "cat": "cat" },
  • JSON5: Numbers can be hexadecimal, have a leading or trailing decimal point, infinity, -infinity, NaN
  • HJSON: YAML like syntax permitted without the need for commas at all
  • HJSON: # comments
  • HJSON: Quoteless strings
  • and more (see the HJSON or JSON5 specific projects for differences
• Works with all past node/npm versions, fully backwards compatible
• Would drive some people to update to the latest npm to get the feature
• Draws people away from npm alternatives like Yarn

[RobertLowe]

This topic gets a lot of FUD around it, but I support what @TheJaredWilcurt has proposed.

[Chealer]

I suppose completeness commends mentioning Yet Another alternative manifest format, JSON6, which claims a few advantages over JSON5. And also that pnpm already supports both JSON5 and YAML.

Thank you for this request @TheJaredWilcurt

[cwtuan]

As a developer, I often find myself struggling with the limitations of the standard JSON format when it comes to managing my project dependencies.

JSON5 is an extension to the standard JSON format that includes additional features like comments and trailing commas. Despite its growing popularity among developers, NPM does not currently support JSON5 as a valid format for its package.json file.

Adding support for JSON5 would make it easier for developers to manage dependencies and configurations in a more readable way, while still maintaining backward-compatibility with the standard JSON format.

As a community, we must urge NPM to consider supporting the JSON5 format.

json5 NPM Downloads Trend:

[RobertLowe]

Time to grow up npm

[Chealer]

The usage of json5 is rising steadily, and its download times indicate a growing trend. It would be highly beneficial if npm could incorporate support for the package.json5 file format.

What does that graph represent?

[thscott]

As package.json supports more and more features, surely it makes sense to be able to document the configuration alongside it, without resorting to messy hacks?

What brought me here was finding a way to document why I was using a certain override.

I would have liked to be able to do:

"overrides": {
    "fast-json-patch": "^3.1.1" // Fix CVE https://github.com/advisories/GHSA-8gh8-hqwg-xf34
  }

which (even on here) is nicely styled as a comment, but instead had to settle for the somewhat clumsy:

"overrides": {
    "fast-json-patch__": "Fix CVE https://github.com/advisories/GHSA-8gh8-hqwg-xf34",
    "fast-json-patch": "^3.1.1"
  }

---

I'd also like to note that the two comment formats suggested by @ljharb above either don't work:

• Causes error npm ERR! Override without name: //:

  "overrides": {
      "//": "Fix CVE https://github.com/advisories/GHSA-8gh8-hqwg-xf34",
      "fast-json-patch": "^3.1.1"
    }

or are not ideal:

• Linting error (in vscode at least) Duplicate object key json(520):

  "overrides": {
      "fast-json-patch": "Fix CVE https://github.com/advisories/GHSA-8gh8-hqwg-xf34",
      "fast-json-patch": "^3.1.1"
    }

[ljharb]

@thscott good call on the double slash form. The other one is valid, though; duplicate keys in json aren’t an error.

[wesleytodd]

Two points here:

1. Even if npm was to support an alternative form there are many other touch points with package.json which would mean you cannot effectively use a new file format until all the tools you depend on update.
2. npm does not own package.json, neither does any other single group.

I think this is the wrong forum for this discussion. Luckily we are working on a new forum under the OpenJS foundation to discuss topics like this. The initial goal is to document the state of the world, but from there it is to have a place to talk about additions/changes and any related tools/docs/specs which would need to update in response. This is not the final "home" of this work, but here is where the initial research will live. In the meeting earlier this week we decided we will likely create a Collab Space for this, but there will be links and announcements in the repo linked when that happens.

I would say this issue should be closed as "wont fix" and then any future conversations like this be re-directed to the collab space once it is setup.

[TheJaredWilcurt]

You've linked to what looks like a dead repo.

Regardless of what the broader JS community does with package.json, if an improved manifest format (like JSON5) was adopted by some other tooling, npm would still need to make changes in order to be compatible with it. Such as updating the new format during npm install and processing the new format to a regular package.json file for backwards compatibility during npm install or similar commands.

Also regardless of if any other system adopted a similar approach, npm could still approve this RFC and adopt the feature now. The worst case scenario would be if the broader ecosystem adopted a different format (npm goes with JSON5, while Node goes with HJSON for example). Then npm would need to deprecate its JSON5 support and switch to HJSON. A minor annoyance while everyone converges on the same format. Though unless there is a major feature difference, whatever npm chooses will likely be the format everyone else would also adopt, if a working implementation shipped with npm first.

This RFC is the 3rd most upvoted of all issues (open or closed). There is a clear desire for this functionality from the community and there are real problems this gives a practical solution to that are not solvable any other way.

[ljharb]

Until node offers a built-in way to serialize and unserialize JSON5 in all active release lines, it would be a disaster if npm adopted it, since tooling wouldn't be able to easily handle it. npm has an obligation here to minimize disruption, and not to pick format winners.

[MattiasMartens]

It's notable that even the new shiny, bun.js, is held back from adopting a different dependency format by this constraint of NPM: oven-sh/bun#3175

Maybe PNPM is our only hope?

[thscott]

Until node offers a built-in way to serialize and unserialize JSON5 in all active release lines, it would be a disaster if npm adopted it, since tooling wouldn't be able to easily handle it. npm has an obligation here to minimize disruption, and not to pick format winners.

@ljharb This proposal suggests automatically generating a standard package.json file, so I can't really see how it would be a disaster for npm to support it.

As pointed out above, npm not moving towards supporting it is part of the reason that there isn't broader support by other tools.

[TheJaredWilcurt]

@ljharb Node does not need to do anything at all for npm to support this.

This would be completely backwards compatible, because everything related to the package.json5 would be handled by npm and would always produce a package.json update as well. The package.json stays the same, and is just auto-generated from the package.json5. If you are using this file type, you understand that some tooling may automatically update your package.json file and you'll need to manually move that over to your package.json5 before committing, or it will be wiped the next time someone runs an npm command like npm install or npm pkg set. When you see a tool mutating your package.json, you could also inform that tool to update to support the JSON5 format, which, as shown previously is basically just a few extra lines of code.

I think the current system that forces everyone in the entire JS community to invent completely different hacky workaround solutions to deal with not being able to comment their code is the actual disaster we are living in today. Stop holding up progress. People want this.

[wesleytodd]

Node does not need to do anything at all for npm to support this.

His point is not about this feature support when installing or publishing. He is talking about EVERYTHING else. The number of tools which read from package.json and do not support the other formats is massive.

I am not opposed to this proposal, but I do think that this discussion needs more broad discussion (not sure why that comment got downvotes). And if there was a move to support it I honestly am not sure why we wouldn't start with node core itself. Node couples to package.json for a few features and if we got consensus there it would be really easy to get support for it here.

[wesleytodd]

On that note actually, can someone explain how projects using an alternative format with pnpm work with things like node's type, exports or the other package.json coupling points? Does it generate the package.json and you commit both to the project?

[thscott]

The number of tools which read from package.json and do not support the other formats is massive.

@TheJaredWilcurt was directing addressing this point. Tools that read from package.json will still read from the generated package.json. The only stumbling block is tools that write to package.json, which are far less common. This need was also addressed by @TheJaredWilcurt (manually copy modifications over until said tools support json5 themselves).

[rlidwka]

It's notable that even the new shiny, bun.js, is held back from adopting a different dependency format by this constraint of NPM: oven-sh/bun#3175

Maybe PNPM is our only hope?

May I suggest deno where dependencies are kept in js modules (deps.ts file by convention).

Seriously, I've been asking for this change for 10 years now, so maybe it's time to give up on node.js ecosystem and go look for/build a better one.