npm install of git urls overwrites registry packages in cache #4042

Open
arikon opened this Issue Oct 25, 2013 · 10 comments
@arikon

Steps to reproduce:

  1. npm install bem — npm will cache bem@0.6.16
  2. npm install git://github.com/bem/bem-tools.git#6625df0 — npm will cache installed bem@0.6.16' version as bem@0.6.16
  3. npm install bem or npm install bem@0.6.x or npm install bem@~0.6.15 — npm will install previously installed from git 0.6.16' as bem@0.6.16 (!!!) — that is not expected

npm install of git urls should not cache installed packages based on their package.json version value. It could cache based on other features, e.g. treeish or something.

Using npm@1.3.13.

@lo1tuma

We have experienced the same issue on our CI server (jenkins).
Our CI server builds every branch. One feature branch uses a git revision of a dependency which includes breaking changes, but the version number in its package.json is the same version that we are depending on all other branches (from the npm registry). So the builds of all other branches will fail.

@lo1tuma

Our workaround for now is to run npm cache clean for every build.

@robertkowalski
npm member

The best way would be that git-dependencies are not seeing the cache at all, but this needs some refactorings on install.js and cache.js

This patch is more a spike / POC for discussion for a possible quickfix, there are several things like windows support and async calling missing. Also tests are missing, automatic and intensive manual testing.

But a possible fix could delete the cached files in case it is a git repo. I am not that happy with it, because it is not the best solution:

diff --git a/lib/install.js b/lib/install.js
index fa990ab..595ab04 100644
--- a/lib/install.js
+++ b/lib/install.js
@@ -75,6 +75,7 @@ var npm = require("./npm.js")
   , isGitUrl = require("./utils/is-git-url.js")
   , npmInstallChecks = require("npm-install-checks")
   , sortedObject = require("sorted-object")
+  , rimraf = require('rimraf')

 function install (args, cb_) {
   var hasArguments = !!args.length
@@ -897,6 +898,10 @@ function write (target, targetFolder, context, cb_) {
         , JSON.stringify(target, null, 2) + "\n" ]
       , [ lifecycle, target, "preinstall", targetFolder ]
       , function (cb) {
+          if (target._from.indexOf("git://") !== -1) {
+            var cacheToRemove = npm.cache + "/" +  target.name + "/" + target.version
+            rimraf.sync(cacheToRemove)
+          }
           if (!target.bundleDependencies) return cb()

           var bd = path.resolve(targetFolder, "node_modules")
@dominictarr

In npmd@1 I address this by keeping a small database of {url,giturl,module@ver}=> shasum
and then a content addressable store of the actual packages. Every sort of package can be cached
which makes for speedy installs, and it avoids this problem. (the current version depends on leveldb,
which must be compiled, but the plan is to replace that with a pure js "database")

@medikoo

+1 Packages installed directly from repositories should not land in cache. Version provided in package.json should be ignored in such case, definitely it should not be treated as same version as one installed in npm registry. I run into related issues recently

@medikoo medikoo referenced this issue in medikoo/d May 15, 2014
Closed

restore d/d #2

@timoxley
npm member

@robertkowalski haven't looked deep into this, but why is your patch removing the item from cache, rather than simply preventing git url packages being added in the first place?

@grncdr

Just want to chime in here and say that any fix for this should also skip the cache when doing npm install /path/to/some/package. It's a pretty common workflow for trying things out in development and leads to a wonky cache quite quickly.

@othiym23 othiym23 added this to the cache rewrite milestone Jul 24, 2014
@panuhorsmalahti

We recently encountered issue. Any plans on a fix? It's a pretty significant issue.

@kompa3

I agree on this. Npm should not cache any packages that are installed from Git repository master branch. That is because the version at repository may go through a lot of changes until it is published to npm registry.

An exception is of course the tagged versions (@1.2.3) from Git repository that should be identical to the published versions and may therefore safely be cached.

@othiym23

This will be addressed by the cache rewrite project, which will be the next major version (npm@4) after the multi-stage install project (npm@3) lands. The most likely behavior is that what will be installed will have an associated shasum which will be tied to the treeish shasum at the time the cache was packed into a tarball, but I have yet to work out how, exactly, that mapping will work in progress. Look for an npm@4 beta early in 2015.

@othiym23 othiym23 removed the ready label Jan 29, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment