This repository has been archived by the owner. It is now read-only.

Git dependencies and npm shrinkwrap don't work together #11736

Closed
bouk opened this Issue Mar 1, 2016 · 11 comments

Comments

Projects
None yet
@bouk

bouk commented Mar 1, 2016

Hi,

It seems npm shrinkwrap and git dependencies are fundamentally incompatible with each other.

echo '{"private": true, "dependencies": {"jquery": "jquery/jquery"}}' > package.json
npm install
npm shrinkwrap
rm -rf node_modules
npm install
npm shrinkwrap

This will display an error similar to

npm ERR! Darwin 15.3.0
npm ERR! argv "/Users/bouke/.nvm/versions/node/v5.5.0/bin/node" "/Users/bouke/.nvm/versions/node/v5.5.0/bin/npm" "shrinkwrap"
npm ERR! node v5.5.0
npm ERR! npm  v3.3.12

npm ERR! Problems were encountered
npm ERR! Please correct and try again.
npm ERR! invalid: have jquery@3.0.0-pre (expected: github:jquery/jquery) /tmp/lolnpm/node_modules/jquery
npm ERR! extraneous: jquery@3.0.0-pre /tmp/lolnpm/node_modules/jquery
npm ERR!
npm ERR! If you need help, you may report this error at:
npm ERR!     <https://github.com/npm/npm/issues>

npm ERR! Please include the following file with any support request:
npm ERR!     /tmp/lolnpm/npm-debug.log

Installing from a shrinkwrap should not cause shrinkwrapping after to fail

@bouk bouk changed the title from Git dependencies and npm shrinkwrap to Git dependencies and npm shrinkwrap don't work together Mar 1, 2016

@othiym23

This comment has been minimized.

Contributor

othiym23 commented Mar 1, 2016

This appears to be a bug in how the installer deals with hosted git dependencies once they've been cached (you can see this just by reinstalling a hosted git package once it's been added to your cache and running npm ls, which will give you an error similar to the one above). Added to big-bug because it produces inconsistent installs and blocks use of shrinkwrap. Thanks for the report!

@XYliang

This comment has been minimized.

XYliang commented Mar 24, 2016

Try using an administrator mode

@kazkovsky

This comment has been minimized.

kazkovsky commented Mar 26, 2016

+1

I have my github repository in shrinkwrap, but it doesn't update on any machine.

cat npm-shrinkwrap.json | grep <my repo>

 "<my repo>": {
      "from": "git+https://<token>:x-oauth-basic@github.com/<my repo>.git",
      "resolved": "git+https://<token>:x-oauth-basic@github.com/<my repo>#9a68c540902e6497855699b990e662e5ab73d3ba"

after running npm install it keeps old version:

npm list

│ └── y18n@3.2.0
└─┬<my repo>@0.2.7 (git+https://<token>:x-oauth-basic@github.com/<my repo>#dd7d3fdf0cb57d8ae9ad1e70326fb4ad98675815)
  ├── canvas@1.2.11
  ├─┬ gulp-plumber@1.1.0
  │ └─┬ through2@2.0.1
@thomsbg

This comment has been minimized.

thomsbg commented Apr 5, 2016

This seems like a dupe (or at least related to) #10502

@sminnee

This comment has been minimized.

Contributor

sminnee commented Apr 14, 2016

Another replication case.

Include this in dependencies:

"dependencies": {
...
  "chosen": "github:harvesthq/chosen#v1.5.1",
...
},

Clean the environment, install & re-shrinkwrap, then keep the shrinkwrap and install from that

 rm -rf node_modules
 rm npm-shrinkwrap.json
 npm install
 npm shrinkwrap --dev
 # commit npm-shrinkwrap.json to source contorl
 rm -rf node_modues
 npm install

Finally, attempt to re-shrinkwrap:

npm ERR! invalid: have chosen@1.5.1 (expected: github:harvesthq/chosen#v1.5.1) /<path>/node_modules/chosen
npm ERR! extraneous: chosen@1.5.1 /<path>/node_modules/chosen

Verified on npm 3.8.6

sminnee added a commit to sminnee/npm that referenced this issue Apr 29, 2016

Fixes npm#11736: better _from values for git remotes
The _from value needs to be the dependency requested prior to any
normalisation. In the case of git remotes, a normalised value as been
passed and this caused errors when re-validating packages check-outs
before a shrinkwrap operation.
@chromakode

This comment has been minimized.

Contributor

chromakode commented Jun 2, 2016

I'm experiencing the same issue. One thing I've noticed: if I individually npm install the specific packages that failed to shrinkwrap, the problem goes away. Diffing the resulting package.jsons may help get to the bottom of this issue.

@chromakode

This comment has been minimized.

Contributor

chromakode commented Jun 2, 2016

Update: using full git URLs with commit hashes (instead of branch names) in package.json seems to work around this issue.

For example, changing "jquery/jquery#master" to "git://github.com/jquery/jquery#73bf35ecf31867d7ed4662374121efa310cf9f8d" shrinkwraps successfully.

@saitodisse

This comment has been minimized.

saitodisse commented Jul 21, 2016

thanks @chromakode, save my life

@mashaalmemon

This comment has been minimized.

mashaalmemon commented Aug 1, 2016

@chromakode, your post saved me alot of time. Full URL and commit hash are the ticket to workaround npm shrinkwrap issue. Note, for anyone else who happens on this post, Also works with https full git URLs as well like so:
"git+https://github.com/jquery/jquery.git#73bf35ecf31867d7ed4662374121efa310cf9f8d"

iarna added a commit that referenced this issue Sep 14, 2016

deps: Fix tagged dependency matching with shrinkwraps
When you depend on a tag, eg `foo@latest`, your `npm-shrinkwrap.json` will
record this in its `_from` field.

When you install under the above circumstances the installed module will
be installed from the associated tarball URL, not by resolving the tag and
installing whatever version is now associated with it. As a result the
`_requested` section of the `package.json` will be based on the tarball URL
and cannot match the `package.json`'s tagged version.

It was _supposed_ to fallback to comparing the `package.json` `_from` field
with but it was doing this by comparing `rawSpec` to `_from`, which _can't_
work, as `rawSpec` is the requested specifier without the name (`latest` in
this case), versus `_from` (`dep@latest` in this case).

Fixes: #13496
Fixes: #11736
PR-URL: #13941
Credit: @iarna

iarna added a commit that referenced this issue Sep 14, 2016

deps: Fix git & tagged dependency matching with shrinkwraps
When you depend on a tag, eg `foo@latest`, your `npm-shrinkwrap.json` will
record this in its `_from` field. Similarly when you depend on a git dependency
with something like `github:foo/bar` it saves this to your `_from`.

When you install under the above circumstances the installed module will be
installed from the associated tarball URL for tagged dependencies or git URL
with commit hash for git dependencies.

As a result the `_requested` section of the `package.json` will be based on
the tarball URL or githash and cann't be matched up to the `package.json`.

It was _supposed_ to fallback to comparing the `package.json` `_from` field
with but it was doing this by comparing `rawSpec` to `_from`, which _can't_
work consistently.

Fixes: #13496
Fixes: #11736
PR-URL: #13941
Credit: @iarna

iarna added a commit that referenced this issue Sep 14, 2016

deps: Fix git & tagged dependency matching with shrinkwraps
When you depend on a tag, eg `foo@latest`, your `npm-shrinkwrap.json` will
record this in its `_from` field. Similarly when you depend on a git dependency
with something like `github:foo/bar` it saves this to your `_from`.

When you install under the above circumstances the installed module will be
installed from the associated tarball URL for tagged dependencies or git URL
with commit hash for git dependencies.

As a result the `_requested` section of the `package.json` will be based on
the tarball URL or githash and cann't be matched up to the `package.json`.

It was _supposed_ to fallback to comparing the `package.json` `_from` field
with but it was doing this by comparing `rawSpec` to `_from`, which _can't_
work consistently.

Fixes: #13496
Fixes: #11736
PR-URL: #13941
Credit: @iarna

iarna added a commit that referenced this issue Sep 14, 2016

deps: Fix git & tagged dependency matching with shrinkwraps
When you depend on a tag, eg `foo@latest`, your `npm-shrinkwrap.json` will
record this in its `_from` field. Similarly when you depend on a git dependency
with something like `github:foo/bar` it saves this to your `_from`.

When you install under the above circumstances the installed module will be
installed from the associated tarball URL for tagged dependencies or git URL
with commit hash for git dependencies.

As a result the `_requested` section of the `package.json` will be based on
the tarball URL or githash and cann't be matched up to the `package.json`.

It was _supposed_ to fallback to comparing the `package.json` `_from` field
with but it was doing this by comparing `rawSpec` to `_from`, which _can't_
work consistently.

Fixes: #13496
Fixes: #11736
PR-URL: #13941
Credit: @iarna

iarna added a commit that referenced this issue Sep 16, 2016

deps: Fix git & tagged dependency matching with shrinkwraps
When you depend on a tag, eg `foo@latest`, your `npm-shrinkwrap.json` will
record this in its `_from` field. Similarly when you depend on a git dependency
with something like `github:foo/bar` it saves this to your `_from`.

When you install under the above circumstances the installed module will be
installed from the associated tarball URL for tagged dependencies or git URL
with commit hash for git dependencies.

As a result the `_requested` section of the `package.json` will be based on
the tarball URL or githash and cann't be matched up to the `package.json`.

It was _supposed_ to fallback to comparing the `package.json` `_from` field
with but it was doing this by comparing `rawSpec` to `_from`, which _can't_
work consistently.

Fixes: #13496
Fixes: #11736
PR-URL: #13941
Credit: @iarna

iarna added a commit that referenced this issue Sep 19, 2016

deps: Fix git & tagged dependency matching with shrinkwraps
When you depend on a tag, eg `foo@latest`, your `npm-shrinkwrap.json` will
record this in its `_from` field. Similarly when you depend on a git dependency
with something like `github:foo/bar` it saves this to your `_from`.

When you install under the above circumstances the installed module will be
installed from the associated tarball URL for tagged dependencies or git URL
with commit hash for git dependencies.

As a result the `_requested` section of the `package.json` will be based on
the tarball URL or githash and cann't be matched up to the `package.json`.

It was _supposed_ to fallback to comparing the `package.json` `_from` field
with but it was doing this by comparing `rawSpec` to `_from`, which _can't_
work consistently.

Fixes: #13496
Fixes: #11736
PR-URL: #13941
Credit: @iarna

christianacca pushed a commit to esvit/ng-table that referenced this issue Sep 23, 2016

@iarna iarna closed this in 3b5eee0 Oct 7, 2016

Glavin001 added a commit to Glavin001/node-email-templates that referenced this issue Oct 23, 2016

Glavin001 added a commit to Glavin001/node-email-templates that referenced this issue Oct 23, 2016

@istrau3

This comment has been minimized.

istrau3 commented Jul 20, 2017

@othiym23
I think this issue has reappeared in the new npm 5+, in other words, when others try running npm install on my project, the install does not obey the shrinkwrap for git dependencies but instead obeys the package.json spec.
Should I create a separate issue?

@cklmercer

This comment has been minimized.

cklmercer commented Jul 20, 2017

Same issue here with npm 5+

using full url and commit hash

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.