[BUG] NPM 7.x broke the "--json" CLI parameter

[octogonz]

The problem

NPM's --json option allows scripts to invoke an NPM command and receive machine-readable JSON that can be easily processed. In previous NPM versions, the JSON was written to STDOUT and all other messages were written to STDERR, making it easy for a script to extract the JSON.

With NPM 7.x, everything is mixed together on STDERR. This defeats the point of the --json feature, which was to make parsing easy. Instead, a script has to implement heuristics to guess which STDERR lines are JSON or not. Scripts developed for NPM 6 now fail because they are reading the wrong stream.

Steps To Reproduce:

1. Correct behavior: With NPM 6.x, invoke this command:

   npm search --json
   

   The JSON output is written to STDOUT:

   {
     "error": {
       "code": null,
       "summary": "search must be called with arguments",
       "detail": ""
     }
   }

   The console messages are written to STDERR:

   npm ERR! search must be called with arguments
   npm ERR! A complete log of this run can be found in:
   

2. Broken behavior: With NPM 7.5.4 (the latest version), invoke the same command. Both outputs are mixed together on STDERR. Nothing is written to STDOUT.

This regression appears to have been introduced with NPM 7.0.0.

Environment:

• OS: All OS's
• Node: 12.17.0
• npm: 7.5.4

[RammasEchor]

Hi! I've been looking up this issue, and I would like to work on it.

[bdkjones]

Any progress on this? It's a big regression from 6.x for those of us who integrate npm into tooling. Thanks.

[octogonz]

This was a significant regression for tools that rely on the NPM installation provided by Node.js: Since we can't assume that the developer has the latest Node.js, even if NPM eventually fixes this bug, it may be years before we can safely remove our workaround.

Looks like @RammasEchor made a PR but it was rejected.

If it helps, here's the workaround that we used:

https://github.com/microsoft/rushstack/blob/b2aa54fa2a284e166529cdebbc3918bcd7dc7503/apps/rush-lib/src/logic/setup/SetupPackageRegistry.ts#L486

In brighter news, Node 16 will let you use Yarn without having to install it. yarn info has equivalent functionality, and Yarn's --json parameter is correctly implemented. 👍

[rossng]

This regression is causing problems for atlassian/changesets users. For example: changesets/changesets#397

[bdkjones]

Sounds like switching to yarn is the best approach. "Will get attention when we're freed up" sounds an awful lot like "Will sit in GitHub issues for seven years."

[lukekarrys]

I did some digging and found that this was introduced in #2155 (which was to fix #2150).

It looks like npm ls --json (not sure about other commands yet) would print their own json blob and then the exit handler would print an additional json blob with the error. This caused unparseable json so the second blob was switched to stderr. Obviously, this caused regressions like this one.

A better solution I think would be to switch the main exit handler back to stdout and special case any commands that are already printing their own json output to not print more. I'm more confident (than I am about #2773) that this fix can come before any output refactor work we're doing. So hopefully it's less than seven years 😄

[Andarist]

The OP suggests that everything was previously sent to stderr - that doesn't sound right though. If I understand the issue correctly the change only affected JSON errors - they are now written to stderr where previously they were written to stdout. So the change is of a smaller scope than OP suggests. Can anyone confirm this? Don't take me wrong - this is still super annoying and affecting me directly (I'm a Changesets maintainer and this has caused problems in our publishing scripts), I just want to make sure that I understand the issue correctly so I can fix it properly on our end.