This repository has been archived by the owner. It is now read-only.

Updating git dependency urls does not work #1727

Closed
felixge opened this Issue Nov 17, 2011 · 97 comments

Comments

Projects
None yet
@felixge
Copy link
Contributor

felixge commented Nov 17, 2011

Currently npm does not notice when the version reference of a previously installed package is upgrade to a git url.

So in order to get the right version one has to rm -rf node_modules/<package> first before running npm install again.

Example: Consider the following package which had some changes in master after version 0.0.2 was released. Among them the Readme was modified:

$ npm install -S utest
utest@0.0.2 ./node_modules/utest 
$ shasum node_modules/utest/Readme.md 
c0f676fb3ccd4aa405c92451e3a9de53f23c39bf  node_modules/utest/Readme.md
$ vim package.json
# Change utest version from "0.0.2" to "git://github.com/felixge/node-utest.git"
$ npm install .

$ shasum node_modules/utest/Readme.md 
c0f676fb3ccd4aa405c92451e3a9de53f23c39bf  node_modules/utest/Readme.md
# ^- Notice how the package has not been updated with the new code from git
$ rm -rf node_modules/utest
$ npm install .
utest@0.0.2 ./node_modules/utest 
$ shasum node_modules/utest/Readme.md 
0167009693fce6aa30e7ef6ea25406da8e1cc0b3  node_modules/utest/Readme.md
# ^- Notice how the package was now updated

Let me know if I can provide further details.

--fg

@isaacs

This comment has been minimized.

Copy link
Member

isaacs commented Nov 17, 2011

In general, npm install (without any args) doesn't update anything that's already present, and only installs missing deps. If you then want it to install again, you need to do npm install <whatever>.

@mikeal asked me about this last night. I think if you and he are getting confused by it, it's the wrong api. Wanna discuss on the npm-@googlegroups.com mailing list?

@felixge

This comment has been minimized.

Copy link
Contributor

felixge commented Nov 18, 2011

I've created a discussion on the mailing list, would love to get your input: https://groups.google.com/d/topic/npm-/KnSd2gpA6fw/discussion

@dshaw

This comment has been minimized.

Copy link

dshaw commented Nov 18, 2011

+1 I've been rm -rf-ing to get around this too.

@felixge

This comment has been minimized.

Copy link
Contributor

felixge commented Nov 23, 2011

@isaacs: did you have a chance to look at this yet / read the post in the ML?

@isaacs

This comment has been minimized.

Copy link
Member

isaacs commented Nov 23, 2011

I saw the mailing list post, but haven't gotten to it yet. Been super busy trying to get 1.1 finished (with the built-in tar stuff.)

@jperras

This comment has been minimized.

Copy link

jperras commented Nov 25, 2011

To add my two cents: This becomes something of an issue when attempting to deploy to a PaaS, such as Heroku. And by issue, I mean the only way around it seems to be completely destroying the application and re-deploying it from scratch.

@felixge

This comment has been minimized.

Copy link
Contributor

felixge commented Nov 28, 2011

@isaacs np, I understand.

@jperras Good point. I'm rather suprised that heroku doesn't wipe the install on deployment so. It seems like that's what they are advocating:

http://www.12factor.net/
http://blog.heroku.com/archives/2011/6/28/the_new_heroku_4_erosion_resistance_explicit_contracts/

(Great articles btw.)

@juggy

This comment has been minimized.

Copy link

juggy commented Nov 28, 2011

@jperras

This comment has been minimized.

Copy link

jperras commented Nov 28, 2011

@juggy Yes, but I have it pointed towards a tarball, e.g.

  "dependencies": {
    "blah": "https://github.com/jperras/repository_name/tarball/v1.2.0",
    ...
  },
@alFReD-NSH

This comment has been minimized.

Copy link

alFReD-NSH commented Mar 22, 2012

I'm not sure this is the same issue, but I've seen that git updates treats the git dependencies as npm registry modules, here's an example:

alfred@ubuntu:~/apps/flight-office$ npm update couch-potato

npm WARN eventemitter2@0.4.1 package.json: 'contributers' should probably be 'contributors'
npm http GET https://registry.npmjs.org/couch-potato/1.0.2
npm http 404 https://registry.npmjs.org/couch-potato/1.0.2

npm ERR! 404 'couch-potato' is not in the npm registry.
npm ERR! 404 You should bug the author to publish it
npm ERR! 404 
npm ERR! 404 Note that you can also install from a
npm ERR! 404 tarball, folder, or http url, or git url.
npm ERR! 
npm ERR! System Linux 3.0.0-16-generic
npm ERR! command "node" "/home/alfred/.nvm/v0.6.13/bin/npm" "update" "couch-potato"
npm ERR! cwd /home/alfred/apps/flight-office
npm ERR! node -v v0.6.13
npm ERR! npm -v 1.1.9
npm ERR! code E404
npm ERR! message 404 Not Found: couch-potato
npm ERR! errno {}
npm ERR! 
npm ERR! Additional logging details can be found in:
npm ERR!     /home/alfred/apps/flight-office/npm-debug.log
npm not ok

But it worked with npm install:


alfred@ubuntu:~/apps/flight-office$ npm install couch-potato
couch-potato@1.0.2 ./node_modules/couch-potato 
└── blanket@0.1.0

@jesseditson

This comment has been minimized.

Copy link

jesseditson commented Apr 13, 2012

What is the current workaround for this? I tried pointing the dependency at a url like:

 https://github.com/name/repo/tarball/master

but it doesn't seem to update to latest when I run npm update (or npm install).

@isaacs

This comment has been minimized.

Copy link
Member

isaacs commented Apr 14, 2012

@jesseditson Just install it again. That's all update does, but there's no way for it to know that it's out of date. But, the original need for this should be fixed.

@alFReD-NSH

This comment has been minimized.

Copy link

alFReD-NSH commented Apr 14, 2012

Well there is, if it detects a change in the url(changing the tag), it should be able to detect that and re-install it.

@isaacs

This comment has been minimized.

Copy link
Member

isaacs commented Apr 14, 2012

Right, so if the package.json's _from attribute doesn't match the url dep, then it should update it, or treat it as an invalid dep. That's why this issue is still open :)

@thesmart

This comment has been minimized.

Copy link

thesmart commented Nov 15, 2012

This is a major gotcha for me as well, and I'm sure many others. We depend on git and npm to update our production code, now there is no point to running npm update ever. rm -rf node_modules && npm install is the work around. Kinda lame. :-/

@youurayy

This comment has been minimized.

Copy link

youurayy commented Nov 19, 2012

Can't npm update just always reinstall the deps where it's not (easily) determinable whether they need updating? This would help a bunch.

@isaacs

This comment has been minimized.

Copy link
Member

isaacs commented Nov 29, 2012

@ypocat +1. Patch welcome.

@Iristyle

This comment has been minimized.

Copy link

Iristyle commented Jan 9, 2013

Yeah, running into this as well... deploying Node with Capistrano, and pulled a trick to share node_modules from one release to the next to save time on deploys. Some of the packages have to compile, etc, etc.

However, I'm noticing dependencies are not properly updating.

So back to the slower deploys until theres a good method for doing this.

@asgrim

This comment has been minimized.

Copy link

asgrim commented Feb 27, 2013

+1 to fix this.

I currently work around by doing:

$ npm update
$ npm install mygitdep

where mygitdep is something like:

{
    "dependencies": {
        "mygitdep": "git+ssh://git@our.gitlab.server/mygitdep.git"
    },
}
@ralt

This comment has been minimized.

Copy link
Contributor

ralt commented Mar 9, 2013

As far as I understand, it updates if it's not the same version.

So it seems this is filtered out earlier. Can't see where though. Gut-wise, I think the game is over there.

@NickHeiner

This comment has been minimized.

Copy link

NickHeiner commented Apr 8, 2013

+1 I've been doing rm -rf node_modules/ as well as a workaround.

@isaacs

This comment has been minimized.

Copy link
Member

isaacs commented Apr 8, 2013

+1

@jfirebaugh

This comment has been minimized.

Copy link

jfirebaugh commented Apr 8, 2013

FWIW, rm -rf isn't necessary. Just npm install <module> will get it updated.

@NachoSoto

This comment has been minimized.

Copy link

NachoSoto commented Apr 9, 2013

+1

@luk-

This comment has been minimized.

Copy link
Contributor

luk- commented Apr 9, 2013

I'll have some time this weekend for npm bugs (probably just this one heh). If someone wants to do it before then, feel free.

@Raynos

This comment has been minimized.

Copy link
Contributor

Raynos commented Apr 15, 2013

pro tip: alias ninstall="rm -rf ./node_modules && npm install"

luk- added a commit to luk-/npm that referenced this issue Apr 15, 2013

npm install of git dependency always update
This addresses issue npm#1727. The change ensures that
git dependencies will always update by checking for
_resolved.
@alFReD-NSH

This comment has been minimized.

Copy link

alFReD-NSH commented Mar 13, 2014

@vmkcom Either open a new issue, or wait for one of the maintainers re-open this issue.

@qzaidi

This comment has been minimized.

Copy link

qzaidi commented Jun 4, 2014

This change breaks something for me. I have multiple dependent packages having a common dependency.

A
|---B
| |---D
|---C
| |---D
|---D

D comes from a git repo (isGit is true). Prior to this change (5829ecf), I would end up with a single copy of git. I was sort of depending on this behavior, and it says something like that in npm help install.

npm help install
....
 To avoid this situation, npm flat-out refuses to install any name@version that is already present  anywhere  in  the  tree  of  package folder  ancestors.  A more correct, but more complex, solution would be to symlink the existing version into the new location.  If this ever affects a real use-case, it will be investigated.

My packages are private, so I can't publish them to public npm repo. Tarballs or private npm repo are an option for now, but is there any other alternative?

@fresheneesz

This comment has been minimized.

Copy link

fresheneesz commented Jun 4, 2014

So you probably shouldn't be depending on that behavior. Other packages that require different versions of D could eff things up. If you're depending on, for example, instanceof to work on a member of the D module, you will likely mess stuff up when other packages require a different version of D.

Whats the reason you're depending on that?

@qzaidi

This comment has been minimized.

Copy link

qzaidi commented Jun 5, 2014

B and C can be installed stand alone, or they can be installed as a dependency (of A). When they are installed as a dependency of A, I want all of them to share the same instance of D. Due to require caching, I have then one exported object shared between all packages, and changes to that object are reflected everywhere.

While this behavior persists on npm and tarball packages, this breaks if the package is installed from a git repo. Isn't this going to be a confusing side effect of the above change. The situation described in help for npm install (Limitations of npm Install algorithm) can potentially happen and the doc needs to be updated too. For me, a work around is to run npm dedupe, or to not install from git (setup tarballs or a private repo).

@fresheneesz

This comment has been minimized.

Copy link

fresheneesz commented Jun 5, 2014

The question is: why do you want them to share the same instance of D? This sounds an awful lot like a global variable with the usual problems.

@qzaidi

This comment has been minimized.

Copy link

qzaidi commented Jun 5, 2014

D has a pool of connections to the DB. Rather than have A B and C have
different pools (each with a certain number of db connections), I prefer to
have a single pool - scales better in our scenario.

On Thu, Jun 5, 2014 at 2:01 PM, fresheneesz notifications@github.com
wrote:

The question is: why do you want them to share the same instance of D?
This sounds an awful lot like a global variable with the usual problems.


Reply to this email directly or view it on GitHub
#1727 (comment).

@fresheneesz

This comment has been minimized.

Copy link

fresheneesz commented Jun 5, 2014

I would recommend creating a way to pass a connection pool to D. That way you can share the same connection pool, not only with D, but with any module that needs it.

@qzaidi

This comment has been minimized.

Copy link

qzaidi commented Jun 6, 2014

I already do that, but I will have to do it at multiple places now (in A, B
and C), since now there are multiple instances of D and a pool has to be
passed to all 3.

On Fri, Jun 6, 2014 at 1:39 AM, fresheneesz notifications@github.com
wrote:

I would recommend creating a way to pass a connection pool to D. That way
you can share the same connection pool, not only with D, but with any
module that needs it.


Reply to this email directly or view it on GitHub
#1727 (comment).

@fresheneesz

This comment has been minimized.

Copy link

fresheneesz commented Jun 6, 2014

True, but doing it this way is almost definitely the best way to ensure that other people can use your modules effectively and efficiently. In the cases where modules are using different versions of D, or where a module is using A, B, or C but doesn't require D, it will allow them to still have a single connection pool. Relying on the require cache to handle this for you doesn't allow that. Anyways, I'm sure you'll figure out how to best handle this for your modules.

@jiminikiz

This comment has been minimized.

Copy link

jiminikiz commented Oct 6, 2015

the temp fix from @asgrim works for me.

@roddik

This comment has been minimized.

Copy link

roddik commented Jan 13, 2016

Any plans to finally fix this?

@kirill-konshin

This comment has been minimized.

Copy link

kirill-konshin commented May 20, 2016

Why is this closed, it still does not work as expected.

@dgreensp

This comment has been minimized.

Copy link

dgreensp commented May 26, 2016

npm install seemingly won't update a package specified by git URL, ever; I changed the semver from 0.1.0 to 10.0.0 and specified a new commit hash, and nothing, with npm 3.9.3.

Edit: Correction, changing the major version seems to have worked. So I think the current behavior is npm install will update if the git URL changes, and it points to a different semver version, AND that semver version is incompatible with the current one; but I'm not positive.

@yorch

This comment has been minimized.

Copy link

yorch commented Jul 22, 2016

Same boat here. Using a tarball URL to install a package that gets updated frequently, but NPM does not take the newest version, even thought the new version is already in the local cache (~/.npm/{package-name}/) unless I run npm install {package}.

I'm currently using 3.10.5, but I can swear some older NPM version (maybe 2.x) was actually updating it automatically. In order to make that old NPM version work like that, I had to send a dynamic Last-Modified header when serving the tarball.

@hayesmaker

This comment has been minimized.

Copy link

hayesmaker commented Aug 4, 2016

same here.. can't get npm install to install anything newer than a 13 day old commit.. we've had 20 commits since then.. we have to specify the commit hash each time we change the dependency for this to work properly?

@ahauser31

This comment has been minimized.

Copy link

ahauser31 commented Sep 13, 2016

I have the same issue... And in my case, it's quite confusing: I have a github open source public dep and a bitbucket private dep, both managed by me and with the package.json set up exactly the same way. I npm install, it fetches both repos. I make changes to both repos, push them to github and bitbucket and run npm update. It will only find the changed github repo. If I run npm update multiple time, it will each time "update" the github dep, even tho there is no change and the new version info / commit hash is stored in the tags that npm creates in the package.json of my dep in the node_modules.

I guess my question is: Why is there a difference between github and bitbucket (both git repos, forgot to say... no tarball or mercurial etc.)?
And why is the github dep "updated" each time I run npm update even tho there is no change?

Edit:
To clarify, when I added both dependencies, I did not specify any version numbers, commit hashes, etc. that might cause npm to only use this specific version, i.e. I ran e.g npm i git+ssh://git@bitbucket.org:XYZ/XXYYZZ.git -S and npm i git+ssh://git@github.com:ABC/AABBCC.git -S

Edit2:
While it's a bit weird that is always re-downloads the github dep, I rather this would be default for bitbucket as well. As it stands, this inconsistency is making it hard to predict how and when things are updated.

@clark-stevenson

This comment has been minimized.

Copy link

clark-stevenson commented Oct 19, 2016

I think I have the same issue. I have a shared library dependency which project X uses.

package.json
...
"dependencies": {
    "my-project": "bitbucket:my-username/my-project"
  }
...

npm update doesn't get the newest version.

I have to manually write into the command line:

npm install bitbucket:my-username/my-project

So right now, my build tools are kind of dead, however, this seems like an old problem and I am wondering if I am just approaching it wrong.

@narkowicz

This comment has been minimized.

Copy link

narkowicz commented Oct 31, 2016

In #3328 (and also somewhere above in this massive thread) there is a suggestion to use branch references such as #master as follows:

package.json
...
  "dependencies": {
    "my-project": "bitbucket:my-username/my-project#master"
  }
...

In my usage with v3.10.8, npm update appears to correctly install the latest commit pushed to the specified branch when the local copy falls behind the remote. Perhaps that will help the situations seen by @clark-stevenson and @ahauser31 ?

There doesn’t appear to be any built in support for versioning of git based dependencies, however a manual strategy such as maintaining and referencing semver style branches such as [n].x.x and ensuring backwards compatibility within those branches could help avoid exposing consumers to breaking changes. Similarly, tagging each specific release allows consistent referencing / locking of minor versions.

If you follow the thread further in #3328 there is the suggestion that support for #1.x.x tag based semver style versioning similar to bower's may be coming for git dependencies. Registry support for private packages and namespaces may make this even less of a priority but… watch that space I guess.

@aneilbaboo

This comment has been minimized.

Copy link

aneilbaboo commented Feb 1, 2017

This is slightly off topic (please chime in if you think I should create a separate issue):

It would be nice to specify the semversion you want from a github url. It would need a new bit of syntax. I'm thinking something like:

"somepackage": "git@github.com:developer/somepackage.git|0.2.x"

The idea is that npm would fetch only package.json from the repo, check and then automatically download and install whatever it finds there if the version number has changed.

This inherits the useful features of the existing versioning mechanism, and some:

  1. Package author controls when new functionality is ready to be used
  2. Package user can accept bug fixes and avoid large / breaking changes to the API
  3. From an efficiency standpoint, it will have the same network overhead as public packages
  4. No new data (e.g., commit hashes as in some of the proposals above) need to be stored
  5. It should be easy to implement, since the mechanisms are already in place

I haven't spelunked NPM, but isn't the change as simple as:

  1. Add/modify code to parse this style of URI
  2. Pass the github url and semversion to existing npm functions

?

@kirill-konshin

This comment has been minimized.

Copy link

kirill-konshin commented Feb 1, 2017

@aneilbaboo you can use the following:

"package": "git+http://gtihub.com/user/package.git#0.1.11"

After slash there could be a tag, branch, or even SHA of the commit.

@rayfoss

This comment has been minimized.

Copy link

rayfoss commented Nov 8, 2017

If you installed a node package from git, you can force an update by using it's package.json name, without the full url.

For instance if you have

package.json
...
"dependencies": {
    "my-project": "bitbucket:my-username/my-project"
  }
...

You should be able to do npm install my-project

@clark-stevenson

This comment has been minimized.

Copy link

clark-stevenson commented Nov 8, 2017

@rayfoss

The problem for me personally (is still) that npm install my-project is fine, but I don't want to keep having to manually type this.

Once you install from say bitbucket, that appears to be that. There is no way to npm update in a way that can be automated like everything else. If I install my-project it will never know that the project is updated and so there is this manual literal npm install my-project that exists outside of the update process.

It really annoys me, again like I stated, it is for a reason by far more clever folk than me, but since I can't find the reason and can't find the solution, all that remains is rage. 😄

@kongfei605

This comment has been minimized.

Copy link

kongfei605 commented Nov 29, 2017

Same problem here. 囧

@martunta

This comment has been minimized.

Copy link

martunta commented Dec 27, 2017

At the end I ended up writing postinstall script that is then executed automatically.

"scripts": { "start": "node scripts/start.js", "build": "node scripts/build.js", "postinstall": "npm update && npm install my-private-lib-name" },
it is silly that we still have to do it

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.