This repository has been archived by the owner. It is now read-only.

npm prune is removing packages installed using git #19356

Closed
jsynowiec opened this Issue Dec 8, 2017 · 16 comments

Comments

Projects
None yet
@jsynowiec
Copy link

jsynowiec commented Dec 8, 2017

I'm opening this issue because:

  • npm is crashing.
  • npm is producing an incorrect install.
  • npm is doing something I don't understand.
  • Other (see below for feature requests):

What's going wrong?

Packages present under the dependencies in the package.json file that were installed using git/github package links, e.g. github:npm/npm#some-tag are being uninstalled when npm prune is executed.

How can the CLI team reproduce the problem?

  1. Install a package using standard syntax, save it as a dependency,
  2. Install another a package using git+<proto>:, git:// or github: format, save it as dependency.
  3. Run npm prune

Expected: Second package is present.
Actual: Second package is missing.

supporting information:

  • npm -v prints: 5.6.0
  • node -v prints: 8.9.1
  • npm config get registry prints: https://registry.npmjs.org/
  • Windows, OS X/macOS, or Linux?: macOS
  • Network issues (unrelated)

additional information:

Workaround - When a tarball url is used, the package is not removed on prune.

@iarna

This comment has been minimized.

Copy link
Member

iarna commented Dec 9, 2017

--only=production is not required to reproduce this.

@jsynowiec

This comment has been minimized.

Copy link
Author

jsynowiec commented Dec 11, 2017

Don't know if this helps but I've noticed that this.idealTree in the Pruner instance lists git dependencies in the childs prop but also as missingDeps.
Also, shouldPrune, especially isExtraneous(child), returns true for those deps, so they end up in toPrune array.

@ValYouW

This comment has been minimized.

Copy link

ValYouW commented Dec 18, 2017

For some reason that doesn't happen on windows... npm v5.5.1
Downgraded all the way to v4.6.1 to restore my linux builds...

@rankida

This comment has been minimized.

Copy link

rankida commented Jan 4, 2018

I just encountered this too. Npm v5.6.0.
If my package is a regular semver (1.x.x) then it is correctly passed over by npm prune --production but if it is a git url it is removed.
I am running MacOS and node v8.9.3

@jsynowiec jsynowiec changed the title npm prune --only=production is removing packages installed with git npm prune is removing packages installed using git Jan 7, 2018

@alphashuro

This comment has been minimized.

Copy link

alphashuro commented Jan 10, 2018

I'm having this problem too

@jsynowiec

This comment has been minimized.

Copy link
Author

jsynowiec commented Jan 11, 2018

@alphashuro @lucasklaassen (and others to follow) please use the 👍 one the issue #19356 (comment) to vote/confirm, rather than comment. You're not adding anything new and all of us are getting notifications about your comments. If you just want to subscribe for updates, use the subscribe button under Notifications that is available in the right column.

@andyghiuta

This comment has been minimized.

Copy link

andyghiuta commented Jan 17, 2018

Here's a partial output of running npm -ddd install somepackage when a package with git is already installed (eg: en-my-package-name):

npm sill currentTree +-- emojis-list@2.1.0
npm sill currentTree +-- en-my-package-name@1.0.5
npm sill currentTree +-- encodeurl@1.0.1

npm sill idealTree +-- emojis-list@2.1.0
npm sill idealTree +-- encodeurl@1.0.1

npm sill diffTrees remove en-my-package-name@1.0.5

npm sill unbuild en-my-package-name@1.0.5
npm info lifecycle en-my-package-name@1.0.5~preuninstall: en-my-package-name@1.0.5
npm info lifecycle en-my-package-name@1.0.5~uninstall: en-my-package-name@1.0.5
npm verb unbuild rmStuff en-my-package-name@1.0.5 from /path/to/my/repo/node_modules
npm info lifecycle en-my-package-name@1.0.5~postuninstall: en-my-package-name@1.0.5

npm sill remove /path/to/my/repo/node_modules/en-my-package-name

As you can see, en-my-package-name is not listed in the silly logs under the idealTree and subsequently removed.

@elliottcrush

This comment has been minimized.

Copy link

elliottcrush commented Jan 19, 2018

I've tried to debug this further and failed to come up with the underlying reason, hopefully some more information might help....

When returning true for my git installed package here -> https://github.com/npm/npm/blob/latest/lib/install/deps.js#L67 I can get the prune to work with a git installed module.

if (fromReq.name === 'my-git-installed-package') return true
Perhaps github are returning new meta data and now rawSpec isn't behaving correctly?

Also could be something related to a version bump from https://github.com/npm/npm-package-arg/blob/master/npa.js ?

@andyghiuta

This comment has been minimized.

Copy link

andyghiuta commented Jan 19, 2018

Indeed the childReq.rawSpec is not the same as requested.rawSpec, so the if on the line https://github.com/npm/npm/blob/latest/lib/install/deps.js#L56 is skipped, although, they are very similar, eg: git+ssh://git@github.com/andyghiuta/test-npm-i.git#83371cdcb95979d0c96027193c4a3e2dfeb51c18 vs git+ssh://git@github.com/andyghiuta/test-npm-i.git. The difference is the commit hash.
It works if after line 56, this would be added:

if (childReq.rawSpec.slice(0, childReq.rawSpec.indexOf('#')) === requested.rawSpec.slice(0, requested.rawSpec.indexOf('#'))) return true

This solution is not ok as it breaks the update of the package, but I don't know enough about npm code to tell where childReq.rawSpec is coming from. I'm guessing from npm-package-arg

@theninj4

This comment has been minimized.

Copy link

theninj4 commented Jan 20, 2018

I've taken a little browse around, this is something of a very simplified view of what may be happening within npm when we install from our github repos to produce the extended package.json files post-install:

const pacote = require('pacote')
var npa = require('npm-package-arg')

const spec = npa('github:some-organisation/some-private-repo#v0.0.0')
pacote.manifest(spec, { cache: './my-cache' }).then(modifiedPackageJson => {
  console.log(modifiedPackageJson) // this looks to be where all the extra attributes come from
})

I'm going to guess the bug here may be something obscure with how the npm <> npm-package-arg <> pacote packages are interacting?

The strange thing here is that my organisation have been using npm prune --production against our projects for several months with no issues, using the same container build image. Then, a few days ago, all our builds started failing due to our private github modules being pruned - private github modules that have been in our projects since the beginning. For reference, said container is built with:

    nvm install 6.11.5 && nvm install 8.9.0 && nvm alias default 8

I can't explain why our build processes would break overnight, it's possible I'm missing something on my end, but I'd suggest something may changed in a network/api response?

@shesek

This comment has been minimized.

Copy link

shesek commented Jan 30, 2018

A simple temporary (and slow) workaround is to npm prune --production && npm install --production, which in my case results in:

removed 157 packages in 24.553s
added 1 package in 65.331s

That 1 package being added back is the one configured to be installed from git.

@sairathb

This comment has been minimized.

Copy link

sairathb commented Feb 5, 2018

Just wanted to add the ripple of this bug, I am using serverless to deploy my project to AWS and now since the prune is removing the git dependencies, my lambda code is failing, @shesek I will not be able to use this as serverless package command is doing it internally,

Is there a work around that I can use for now till this bug is fixed? Thanks

@thenickdude

This comment has been minimized.

Copy link

thenickdude commented Feb 5, 2018

@sairathb I'm in the same situation. Just install npm@5.5, clear your NPM cache, and serverless will work properly after that. The problem is only in version 5.6.

@scottmmjackson

This comment has been minimized.

Copy link

scottmmjackson commented Feb 14, 2018

I was able to reproduce this issue with node@8.9.4 and npm@5.5.1

I rolled back to npm@4.6.1 to resolve the issue.

@saboya

This comment has been minimized.

Copy link

saboya commented Feb 14, 2018

A known workaround is to specify the commit hash in your package.json.

@jsynowiec

This comment has been minimized.

Copy link
Author

jsynowiec commented Feb 22, 2018

Supposedly fixed in 857dab0 and included in npm@5.7 release

figueredo added a commit to CESARBR/knot-gateway-buildroot that referenced this issue Jun 27, 2018

knot-web: avoid removal of git dependencies
npm is removing packages that are specified using a Git repository when
pruning node_modules. This patch is a workaround this bug while a fix
isn't released.

Issue: npm/npm#19356
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.