Updating git dependency urls does not work

[felixge]

Currently npm does not notice when the version reference of a previously installed package is upgrade to a git url.

So in order to get the right version one has to rm -rf node_modules/<package> first before running npm install again.

Example: Consider the following package which had some changes in master after version 0.0.2 was released. Among them the Readme was modified:

$ npm install -S utest
utest@0.0.2 ./node_modules/utest 
$ shasum node_modules/utest/Readme.md 
c0f676fb3ccd4aa405c92451e3a9de53f23c39bf  node_modules/utest/Readme.md
$ vim package.json
# Change utest version from "0.0.2" to "git://github.com/felixge/node-utest.git"
$ npm install .

$ shasum node_modules/utest/Readme.md 
c0f676fb3ccd4aa405c92451e3a9de53f23c39bf  node_modules/utest/Readme.md
# ^- Notice how the package has not been updated with the new code from git
$ rm -rf node_modules/utest
$ npm install .
utest@0.0.2 ./node_modules/utest 
$ shasum node_modules/utest/Readme.md 
0167009693fce6aa30e7ef6ea25406da8e1cc0b3  node_modules/utest/Readme.md
# ^- Notice how the package was now updated

Let me know if I can provide further details.

--fg

[isaacs]

In general, npm install (without any args) doesn't update anything that's already present, and only installs missing deps. If you then want it to install again, you need to do npm install <whatever>.

@mikeal asked me about this last night. I think if you and he are getting confused by it, it's the wrong api. Wanna discuss on the npm-@googlegroups.com mailing list?

[felixge]

I've created a discussion on the mailing list, would love to get your input: https://groups.google.com/d/topic/npm-/KnSd2gpA6fw/discussion

[dshaw]

+1 I've been rm -rf-ing to get around this too.

[felixge]

@isaacs: did you have a chance to look at this yet / read the post in the ML?

[isaacs]

I saw the mailing list post, but haven't gotten to it yet. Been super busy trying to get 1.1 finished (with the built-in tar stuff.)

[jperras]

To add my two cents: This becomes something of an issue when attempting to deploy to a PaaS, such as Heroku. And by issue, I mean the only way around it seems to be completely destroying the application and re-deploying it from scratch.

[felixge]

@isaacs np, I understand.

@jperras Good point. I'm rather suprised that heroku doesn't wipe the install on deployment so. It seems like that's what they are advocating:

http://www.12factor.net/
http://blog.heroku.com/archives/2011/6/28/the_new_heroku_4_erosion_resistance_explicit_contracts/

(Great articles btw.)

[juggy]

@jperras are you able to deploy a git dependency on heroku? I am getting: http://stackoverflow.com/questions/8243527/use-git-dependencies-in-with-npm-and-node-on-heroku/8299062#8299062

[jperras]

@juggy Yes, but I have it pointed towards a tarball, e.g.

  "dependencies": {
    "blah": "https://github.com/jperras/repository_name/tarball/v1.2.0",
    ...
  },

[faridnsh]

I'm not sure this is the same issue, but I've seen that git updates treats the git dependencies as npm registry modules, here's an example:

alfred@ubuntu:~/apps/flight-office$ npm update couch-potato

npm WARN eventemitter2@0.4.1 package.json: 'contributers' should probably be 'contributors'
npm http GET https://registry.npmjs.org/couch-potato/1.0.2
npm http 404 https://registry.npmjs.org/couch-potato/1.0.2

npm ERR! 404 'couch-potato' is not in the npm registry.
npm ERR! 404 You should bug the author to publish it
npm ERR! 404 
npm ERR! 404 Note that you can also install from a
npm ERR! 404 tarball, folder, or http url, or git url.
npm ERR! 
npm ERR! System Linux 3.0.0-16-generic
npm ERR! command "node" "/home/alfred/.nvm/v0.6.13/bin/npm" "update" "couch-potato"
npm ERR! cwd /home/alfred/apps/flight-office
npm ERR! node -v v0.6.13
npm ERR! npm -v 1.1.9
npm ERR! code E404
npm ERR! message 404 Not Found: couch-potato
npm ERR! errno {}
npm ERR! 
npm ERR! Additional logging details can be found in:
npm ERR!     /home/alfred/apps/flight-office/npm-debug.log
npm not ok

But it worked with npm install:

alfred@ubuntu:~/apps/flight-office$ npm install couch-potato
couch-potato@1.0.2 ./node_modules/couch-potato 
└── blanket@0.1.0

[jesseditson]

What is the current workaround for this? I tried pointing the dependency at a url like:

 https://github.com/name/repo/tarball/master

but it doesn't seem to update to latest when I run npm update (or npm install).

[isaacs]

@jesseditson Just install it again. That's all update does, but there's no way for it to know that it's out of date. But, the original need for this should be fixed.

[faridnsh]

Well there is, if it detects a change in the url(changing the tag), it should be able to detect that and re-install it.

[isaacs]

Right, so if the package.json's _from attribute doesn't match the url dep, then it should update it, or treat it as an invalid dep. That's why this issue is still open :)

[thesmart]

This is a major gotcha for me as well, and I'm sure many others. We depend on git and npm to update our production code, now there is no point to running npm update ever. rm -rf node_modules && npm install is the work around. Kinda lame. :-/

[youurayy]

Can't npm update just always reinstall the deps where it's not (easily) determinable whether they need updating? This would help a bunch.

[isaacs]

@ypocat +1. Patch welcome.

[Iristyle]

Yeah, running into this as well... deploying Node with Capistrano, and pulled a trick to share node_modules from one release to the next to save time on deploys. Some of the packages have to compile, etc, etc.

However, I'm noticing dependencies are not properly updating.

So back to the slower deploys until theres a good method for doing this.

[asgrim]

+1 to fix this.

I currently work around by doing:

$ npm update
$ npm install mygitdep

where mygitdep is something like:

{
    "dependencies": {
        "mygitdep": "git+ssh://git@our.gitlab.server/mygitdep.git"
    },
}

[ralt]

As far as I understand, it updates if it's not the same version.

So it seems this is filtered out earlier. Can't see where though. Gut-wise, I think the game is over there.

[NickHeiner]

+1 I've been doing rm -rf node_modules/ as well as a workaround.

[isaacs]

+1

[jfirebaugh]

FWIW, rm -rf isn't necessary. Just npm install <module> will get it updated.

[NachoSoto]

+1

[luk-]

I'll have some time this weekend for npm bugs (probably just this one heh). If someone wants to do it before then, feel free.

[Raynos]

pro tip: alias ninstall="rm -rf ./node_modules && npm install"

[fresheneesz]

The question is: why do you want them to share the same instance of D? This sounds an awful lot like a global variable with the usual problems.

[qzaidi]

D has a pool of connections to the DB. Rather than have A B and C have
different pools (each with a certain number of db connections), I prefer to
have a single pool - scales better in our scenario.

On Thu, Jun 5, 2014 at 2:01 PM, fresheneesz notifications@github.com
wrote:

The question is: why do you want them to share the same instance of D?
This sounds an awful lot like a global variable with the usual problems.

—
Reply to this email directly or view it on GitHub
#1727 (comment).

[fresheneesz]

I would recommend creating a way to pass a connection pool to D. That way you can share the same connection pool, not only with D, but with any module that needs it.

[qzaidi]

I already do that, but I will have to do it at multiple places now (in A, B
and C), since now there are multiple instances of D and a pool has to be
passed to all 3.

On Fri, Jun 6, 2014 at 1:39 AM, fresheneesz notifications@github.com
wrote:

I would recommend creating a way to pass a connection pool to D. That way
you can share the same connection pool, not only with D, but with any
module that needs it.

—
Reply to this email directly or view it on GitHub
#1727 (comment).

[fresheneesz]

True, but doing it this way is almost definitely the best way to ensure that other people can use your modules effectively and efficiently. In the cases where modules are using different versions of D, or where a module is using A, B, or C but doesn't require D, it will allow them to still have a single connection pool. Relying on the require cache to handle this for you doesn't allow that. Anyways, I'm sure you'll figure out how to best handle this for your modules.

[jiminikiz]

the temp fix from @asgrim works for me.

[roddik]

Any plans to finally fix this?

[kirill-konshin]

Why is this closed, it still does not work as expected.

[dgreensp]

npm install seemingly won't update a package specified by git URL, ever; I changed the semver from 0.1.0 to 10.0.0 and specified a new commit hash, and nothing, with npm 3.9.3.

Edit: Correction, changing the major version seems to have worked. So I think the current behavior is npm install will update if the git URL changes, and it points to a different semver version, AND that semver version is incompatible with the current one; but I'm not positive.

[yorch]

Same boat here. Using a tarball URL to install a package that gets updated frequently, but NPM does not take the newest version, even thought the new version is already in the local cache (~/.npm/{package-name}/) unless I run npm install {package}.

I'm currently using 3.10.5, but I can swear some older NPM version (maybe 2.x) was actually updating it automatically. In order to make that old NPM version work like that, I had to send a dynamic Last-Modified header when serving the tarball.

[hayesmaker]

same here.. can't get npm install to install anything newer than a 13 day old commit.. we've had 20 commits since then.. we have to specify the commit hash each time we change the dependency for this to work properly?

[ahauser31]

I have the same issue... And in my case, it's quite confusing: I have a github open source public dep and a bitbucket private dep, both managed by me and with the package.json set up exactly the same way. I npm install, it fetches both repos. I make changes to both repos, push them to github and bitbucket and run npm update. It will only find the changed github repo. If I run npm update multiple time, it will each time "update" the github dep, even tho there is no change and the new version info / commit hash is stored in the tags that npm creates in the package.json of my dep in the node_modules.

I guess my question is: Why is there a difference between github and bitbucket (both git repos, forgot to say... no tarball or mercurial etc.)?
And why is the github dep "updated" each time I run npm update even tho there is no change?

Edit:
To clarify, when I added both dependencies, I did not specify any version numbers, commit hashes, etc. that might cause npm to only use this specific version, i.e. I ran e.g npm i git+ssh://git@bitbucket.org:XYZ/XXYYZZ.git -S and npm i git+ssh://git@github.com:ABC/AABBCC.git -S

Edit2:
While it's a bit weird that is always re-downloads the github dep, I rather this would be default for bitbucket as well. As it stands, this inconsistency is making it hard to predict how and when things are updated.

[clark-stevenson]

I think I have the same issue. I have a shared library dependency which project X uses.

package.json
...
"dependencies": {
    "my-project": "bitbucket:my-username/my-project"
  }
...

npm update doesn't get the newest version.

I have to manually write into the command line:

npm install bitbucket:my-username/my-project

So right now, my build tools are kind of dead, however, this seems like an old problem and I am wondering if I am just approaching it wrong.

[narkowicz]

In #3328 (and also somewhere above in this massive thread) there is a suggestion to use branch references such as #master as follows:

package.json
...
  "dependencies": {
    "my-project": "bitbucket:my-username/my-project#master"
  }
...

In my usage with v3.10.8, npm update appears to correctly install the latest commit pushed to the specified branch when the local copy falls behind the remote. Perhaps that will help the situations seen by @clark-stevenson and @ahauser31 ?

There doesn’t appear to be any built in support for versioning of git based dependencies, however a manual strategy such as maintaining and referencing semver style branches such as [n].x.x and ensuring backwards compatibility within those branches could help avoid exposing consumers to breaking changes. Similarly, tagging each specific release allows consistent referencing / locking of minor versions.

If you follow the thread further in #3328 there is the suggestion that support for #1.x.x tag based semver style versioning similar to bower's may be coming for git dependencies. Registry support for private packages and namespaces may make this even less of a priority but… watch that space I guess.

[aneilbaboo]

This is slightly off topic (please chime in if you think I should create a separate issue):

It would be nice to specify the semversion you want from a github url. It would need a new bit of syntax. I'm thinking something like:

"somepackage": "git@github.com:developer/somepackage.git|0.2.x"

The idea is that npm would fetch only package.json from the repo, check and then automatically download and install whatever it finds there if the version number has changed.

This inherits the useful features of the existing versioning mechanism, and some:

1. Package author controls when new functionality is ready to be used
2. Package user can accept bug fixes and avoid large / breaking changes to the API
3. From an efficiency standpoint, it will have the same network overhead as public packages
4. No new data (e.g., commit hashes as in some of the proposals above) need to be stored
5. It should be easy to implement, since the mechanisms are already in place

I haven't spelunked NPM, but isn't the change as simple as:

1. Add/modify code to parse this style of URI
2. Pass the github url and semversion to existing npm functions

?

[kirill-konshin]

@aneilbaboo you can use the following:

"package": "git+http://gtihub.com/user/package.git#0.1.11"

After slash there could be a tag, branch, or even SHA of the commit.

[FossPrime]

If you installed a node package from git, you can force an update by using it's package.json name, without the full url.

For instance if you have

package.json
...
"dependencies": {
    "my-project": "bitbucket:my-username/my-project"
  }
...

You should be able to do npm install my-project

[clark-stevenson]

@rayfoss

The problem for me personally (is still) that npm install my-project is fine, but I don't want to keep having to manually type this.

Once you install from say bitbucket, that appears to be that. There is no way to npm update in a way that can be automated like everything else. If I install my-project it will never know that the project is updated and so there is this manual literal npm install my-project that exists outside of the update process.

It really annoys me, again like I stated, it is for a reason by far more clever folk than me, but since I can't find the reason and can't find the solution, all that remains is rage. 😄

[kongfei605]

Same problem here. 囧

[martunta]

At the end I ended up writing postinstall script that is then executed automatically.

"scripts": { "start": "node scripts/start.js", "build": "node scripts/build.js", "postinstall": "npm update && npm install my-private-lib-name" },
it is silly that we still have to do it