Wiki instructions for updating npm's bundled copy of node-gyp need updating

[DeeDeeG]

See this comment: #2354 (comment)

To restate the main changes needed:

• The Windows instructions are incorrect, as they use the -g flag (meaning, to modify a standalone, global install of node-gyp), whereas the instructions are supposed to be telling you how to cd into the npm folder and locally update the copy/copies of node-gyp installed there. The use of the -g flag is inappropriate and leads to npm retaining its old/outdated copy of node-gyp.
• Also, it seems to be necessary (in my testing) to upgrade node-gyp in two places, as multiple packages directly depend on node-gyp; not only npm itself, but the subdependency npm-lifecycle. One must cd into each package and upgrade its node-gyp dependency manually.

[DeeDeeG]

Updating the wiki page to this should work:

https://gist.githubusercontent.com/DeeDeeG/d95a79b590dae05fa7cbbe4fd34c06b2/raw/7659c3a5f2deacb38dfb63b05e9fee87c36c92b4/Updating-npm's-bundled-node-gyp.md

Diff: https://gist.github.com/DeeDeeG/d95a79b590dae05fa7cbbe4fd34c06b2#file-git-diff-diff

[cclauss]

@DeeDeeG Please review the changes made https://github.com/nodejs/node-gyp/wiki/Updating-npm's-bundled-node-gyp

[cclauss]

On supported versions of Windows, do we need to suggest both which and where?
Or has this been resolved to just one of them?

[DeeDeeG]

@cclauss good catch... We can simplify to just where.

Only Windows 7 and newer work with the latest node-gyp. I think it has been that way for a long time. (The most-recent time I researched this: #2333 (comment))

And I have found a couple of sources saying where was added in Windows Vista:

• https://web.archive.org/web/20070423121442/http://computerbits.wordpress.com/2006/07/21/new-command-line-tools-in-vista-beta-2/
• https://www.lifewire.com/list-of-command-prompt-commands-4092302

So, all versions of Windows that work with the latest node-gyp support where.

[DeeDeeG]

It's a bit more complicated considering both npm 6 and 7, the two major versions that are in wide circulation...

npm 7 doesn't have npm-lifecycle anymore, it has @npmcli/run-script. And during npm explore --global, it sets an env var npm_config_global=true that affects the subshell, meaning our subsequent npm install node-gyp command is effectively npm install --global node-gyp... 🤦

Considering all that, this works on npm 7:

npm explore -g npm/node_modules/@npmcli/run-script -- npm_config_global=false npm install node-gyp@latest

Whereas I can confirm the command you put in the wiki works for npm 6:

npm explore -g npm/node_modules/npm-lifecycle -- npm install node-gyp@latest

[cclauss]

OK. I dropped which in favor of where. Can you please tell me exactly what the last part of the page should say? (I do not have a Windows box.)

npm --version

• If the npm version is less than 7 do X.
• If the npm version is greater than or equal to 7 do Y.

[DeeDeeG]

The difference is where (what directory) to cd into.

Default locations based on the official NodeJS installer:

C:\Program Files\nodejs\node_modules\npm\node_modules\npm-lifecycle (for npm 6) or
C:\Program Files\nodejs\node_modules\npm\node_modules\@npmcli\run-script (for npm 7)

These are the steps I would recommend, for ease of reading/following them...

• In an Administrator cmd prompt...
• where node
  • cd [path\to\node.exe, minus the node.exe], i.e. cd into the folder containing node.exe
    • For example, to cd into the default install path: cd "C:\Program Files\nodejs"
• npm --version
  • If the npm version is less than 7, do:
    • cd node_modules\npm\node_modules\npm-lifecycle
  • If the npm version is 7 do:
    • cd node_modules\npm\node_modules\@npmcli\run-script
• npm install node-gyp@latest

[cclauss]

Is https://github.com/nodejs/node-gyp/wiki/Updating-npm's-bundled-node-gyp the right stuff now?

[DeeDeeG]

Thanks for updating it again.

We need the npm 6 vs npm 7 split in the Unix instructions as well:

npm --version, then

if npm <7 :

$ [sudo] npm explore npm/node_modules/npm-lifecycle -g -- npm install node-gyp@latest

if npm 7:

$ [sudo] npm explore npm/node_modules/@npmcli/run-script -g -- npm_config_global=false npm install node-gyp@latest

Nits:

• It says "i.e." in the Windows instructions, but I think it's better to say "e.g." or "for example", since the actual path can be completely arbitrary, and we are only giving the default example.

• I mildly disagree with saying "npm 7 or greater," because we can't really predict how npm 8 will or won't differ from npm 7. The whole reason the logic is split between 6 and 7 is that they rewrote some of their dependencies from scratch, and we have to update node-gyp under the new dependency for npm 7. (That said, there is some chance npm 8 would use the same dependency, just like it was kept in the transition from npm 5 to npm 6... I dunno). There is a tradeoff between trying to be future-proof vs being conservative and only talking about existing versions of npm.

[cclauss]

Will users be squeamish about upgrading npm?
If it is < v7, why not just recommend npm install -g npm which would simplify the remaining instructions?

[DeeDeeG]

I would be squeamish about it personally, but I will admit I am biased.

For Unix/in general: I am not sure how others feel about it... I personally dislike npm 7, as it changes a lot of the finer points of npm's behavior compared with 6. (It has also had many bugs, several of which I've encountered/reported, especially relating to the strange edge cases a node-gyp contributor in particular can experience...) npm 7 has a new lockfile format, so switching between 6 and 7 re-writes the lockfile with a huge diff each time. So it is a bit obnoxious to mix use of npm 6 and 7 on a multi-developer team. I personally have to really think about which I'm using. For some projects I have deliberately stayed on 6 to match other contributors. In general I am much more confident in npm 6 than 7, so I avoid 7 for less hassle and headaches. (My sense of the "hassle" is amplified though, because I feel a duty to understand and report bugs I encounter. This is a big part of my bias against npm 7: I feel like they should have done more Quality Assurance and called for wider testing before hitting publish, and I don't enjoy doing a lot of it myself.) One point in favor of npm 7: It is faster.

That said, it is mostly drop-in compatible. Some contributors/maintainers don't mind a huge (essentially meaningless) diff to package-lock.json, or they don't track package-lock.json in git. And I do a lot more weird/advanced things with npm than most people probably do, so I hit more bugs than the average person.

For Windows: I would advise against npm install -g npm on Windows. npm install -g npm doesn't work well on Windows, and there are entire programs dedicated to doing it right: https://www.npmjs.com/package/npm-windows-upgrade. I suppose folks can run that, but I think it's more trouble than it's worth...

[DeeDeeG]

Given that there are practical reasons for a user to want (perhaps need) to work with npm 6, and given than npm 6 is shipped with Node 12 and 14 LTS, and given that upgrading is not trivial on Windows, I personally think we should provide instructions for both npm 6 and 7.

[cclauss]

OK… Are we done??

[DeeDeeG]

Tested again reading/copy-pasting straight from the wiki page, and I can confirm it is working as written. Thank you very much!

Test scenarios:

• Ubuntu 20.04, Node/npm installed with n, in a non-root-protected area (no sudo needed for this case)... Instructions work as written for npm 6 and 7, respectively ✔️
• Windows 10, with one copy of Node/npm installed under the protected "Program Files" dir, another copy installed in a non-protected area by extracting a .zip archive... Instructions work as written for npm 6 and 7, respectively. ✔️

Idle thoughts (no action needed) regarding sudo or Administrator cmd.exe prompt: Users with Node installed in a non-protected area in Windows won't need Administrator command prompt, just as Unix users don't need sudo if their copy of Node is installed in a non root-owned/protected area. I think we explain this well enough in the Windows instructions, even if we don't say so explicitly. I trust Linux users to figure it out. Using elevated permissions doesn't hurt in this case, anyhow.

[cclauss]

Thanks massively for your precision in fixing our docs.

[hagaigold]

@DeeDeeG,
For Windows with NPM 6.x.
Doesn't a step is missing here- https://github.com/nodejs/node-gyp/blob/master/docs/Updating-npm-bundled-node-gyp.md

I had also to execute npm install node-gyp@latest in the node_modules\npm while the instruction point to just run it in the node_modules\npm\node_modules\npm-lifecycle folder.

in your earlier draft it was there:

https://gist.githubusercontent.com/DeeDeeG/d95a79b590dae05fa7cbbe4fd34c06b2/raw/7659c3a5f2deacb38dfb63b05e9fee87c36c92b4/Updating-npm's-bundled-node-gyp.md

[DeeDeeG]

@hagaigold if I recall correctly, only the deeper nested copy needs to be updated. That should be the copy the "script runner" under npm-lifecycle calls.

[hagaigold]

@hagaigold if I recall correctly, only the deeper nested copy needs to be updated. That should be the copy the "script runner" under npm-lifecycle calls.

My two cents: I had a different experience when building VS Code project (had to update both locations)