npm install doesn't wait for dependencies to be installed

[jbuck]

We found a bug over in filerjs/filer#285 where if you have a dependency (bower in this case) listed in your root package.json as well as another dependency's package.json (filer in this case), it's very easy to hit a race condition where the filer postinstall would fail because npm wouldn't wait for bower to be installed in the root.

STR create a package.json with the following content:

{
  "name": "a-poor-app",
  "version": "1.0.0",
  "dependencies": {
    "bower": "1.3.9",
    "filer": "0.0.28"
  }
}

Then run npm install; echo "*************************** noooooooooooooo npm install why aren't you patient **************************"; npm install and receive the following output:

npm WARN package.json a-poor-app@1.0.0 No description
npm WARN package.json a-poor-app@1.0.0 No repository field.
npm WARN package.json a-poor-app@1.0.0 No README data

> filer@0.0.28 postinstall /Users/jbuck/Documents/Github/a-poor-app/node_modules/filer
> bower install

sh: bower: command not found
npm ERR! filer@0.0.28 postinstall: `bower install`
npm ERR! Exit status 127
npm ERR!
npm ERR! Failed at the filer@0.0.28 postinstall script.
npm ERR! This is most likely a problem with the filer package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR!     bower install
npm ERR! You can get their info via:
npm ERR!     npm owner ls filer
npm ERR! There is likely additional logging output above.

npm ERR! System Darwin 13.3.0
npm ERR! command "node" "/usr/local/bin/npm" "install"
npm ERR! cwd /Users/jbuck/Documents/Github/a-poor-app
npm ERR! node -v v0.10.31
npm ERR! npm -v 1.4.23
npm ERR! code ELIFECYCLE
npm ERR!
npm ERR! Additional logging details can be found in:
npm ERR!     /Users/jbuck/Documents/Github/a-poor-app/npm-debug.log
npm ERR! not ok code 0
*************************** noooooooooooooo npm install why aren't you patient **************************
npm WARN package.json a-poor-app@1.0.0 No description
npm WARN package.json a-poor-app@1.0.0 No repository field.
npm WARN package.json a-poor-app@1.0.0 No README data

> filer@0.0.28 postinstall /Users/jbuck/Documents/Github/a-poor-app/node_modules/filer
> bower install

bower mocha#1.17.1              cached git://github.com/visionmedia/mocha.git#1.17.1
bower mocha#1.17.1            validate 1.17.1 against git://github.com/visionmedia/mocha.git#1.17.1
bower chai#1.9.0                cached git://github.com/chaijs/chai.git#1.9.0
bower chai#1.9.0              validate 1.9.0 against git://github.com/chaijs/chai.git#1.9.0
bower chai#1.9.0               install chai#1.9.0
bower mocha#1.17.1             install mocha#1.17.1

chai#1.9.0 bower_components/chai

mocha#1.17.1 bower_components/mocha
filer@0.0.28 node_modules/filer
└── base64-arraybuffer@0.1.2

[othiym23]

We believe that this issue has been addressed in npm@2.0.0 (backported into npm@1.4.28). Please try one of those two and let us know if it does not address the problem. Thanks!

[jbuck]

@othiym23 This error still occurs with npm@1.4.28 and npm@2.0.0:

Jons-MacBook-Pro:test jbuck$ cat package.json
{
  "name": "a-poor-app",
  "version": "1.0.0",
  "dependencies": {
    "bower": "1.3.9",
    "filer": "0.0.28"
  }
}
Jons-MacBook-Pro:test jbuck$ npm install; echo "*************************** noooooooooooooo npm install why aren't you patient **************************"; npm install
npm WARN package.json a-poor-app@1.0.0 No description
npm WARN package.json a-poor-app@1.0.0 No repository field.
npm WARN package.json a-poor-app@1.0.0 No README data

> filer@0.0.28 postinstall /Users/jbuck/Documents/Github/test/node_modules/filer
> bower install

sh: bower: command not found
npm ERR! Darwin 13.3.0
npm ERR! argv "node" "/usr/local/bin/npm" "install"
npm ERR! node v0.10.32
npm ERR! npm  v2.0.0
npm ERR! code ELIFECYCLE

npm ERR! filer@0.0.28 postinstall: `bower install`
npm ERR! Exit status 127
npm ERR!
npm ERR! Failed at the filer@0.0.28 postinstall script.
npm ERR! This is most likely a problem with the filer package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR!     bower install
npm ERR! You can get their info via:
npm ERR!     npm owner ls filer
npm ERR! There is likely additional logging output above.
*************************** noooooooooooooo npm install why aren't you patient **************************
npm WARN package.json a-poor-app@1.0.0 No description
npm WARN package.json a-poor-app@1.0.0 No repository field.
npm WARN package.json a-poor-app@1.0.0 No README data

> filer@0.0.28 postinstall /Users/jbuck/Documents/Github/test/node_modules/filer
> bower install

bower mocha#1.17.1              cached git://github.com/visionmedia/mocha.git#1.17.1
bower mocha#1.17.1            validate 1.17.1 against git://github.com/visionmedia/mocha.git#1.17.1
bower chai#1.9.0                cached git://github.com/chaijs/chai.git#1.9.0
bower chai#1.9.0              validate 1.9.0 against git://github.com/chaijs/chai.git#1.9.0
bower chai#1.9.0               install chai#1.9.0
bower mocha#1.17.1             install mocha#1.17.1

chai#1.9.0 bower_components/chai

mocha#1.17.1 bower_components/mocha


┌───────────────────────────────────────────┐
│ Update available: 1.3.10 (current: 1.3.9) │
│ Run npm update -g bower to update.        │
└───────────────────────────────────────────┘

filer@0.0.28 node_modules/filer
└── base64-arraybuffer@0.1.2
Jons-MacBook-Pro:test jbuck$
Jons-MacBook-Pro:test jbuck$ npm install; echo "*************************** noooooooooooooo npm install why aren't you patient **************************"; npm install
npm WARN package.json a-poor-app@1.0.0 No description
npm WARN package.json a-poor-app@1.0.0 No repository field.
npm WARN package.json a-poor-app@1.0.0 No README data

> filer@0.0.28 postinstall /Users/jbuck/Documents/Github/test/node_modules/filer
> bower install

sh: bower: command not found
npm ERR! filer@0.0.28 postinstall: `bower install`
npm ERR! Exit status 127
npm ERR!
npm ERR! Failed at the filer@0.0.28 postinstall script.
npm ERR! This is most likely a problem with the filer package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR!     bower install
npm ERR! You can get their info via:
npm ERR!     npm owner ls filer
npm ERR! There is likely additional logging output above.

npm ERR! System Darwin 13.3.0
npm ERR! command "node" "/usr/local/bin/npm" "install"
npm ERR! cwd /Users/jbuck/Documents/Github/test
npm ERR! node -v v0.10.32
npm ERR! npm -v 1.4.28
npm ERR! code ELIFECYCLE
npm ERR! not ok code 0
*************************** noooooooooooooo npm install why aren't you patient **************************
npm WARN package.json a-poor-app@1.0.0 No description
npm WARN package.json a-poor-app@1.0.0 No repository field.
npm WARN package.json a-poor-app@1.0.0 No README data
\
> filer@0.0.28 postinstall /Users/jbuck/Documents/Github/test/node_modules/filer
> bower install

bower chai#1.9.0                cached git://github.com/chaijs/chai.git#1.9.0
bower chai#1.9.0              validate 1.9.0 against git://github.com/chaijs/chai.git#1.9.0
bower mocha#1.17.1              cached git://github.com/visionmedia/mocha.git#1.17.1
bower mocha#1.17.1            validate 1.17.1 against git://github.com/visionmedia/mocha.git#1.17.1
bower chai#1.9.0               install chai#1.9.0
bower mocha#1.17.1             install mocha#1.17.1

chai#1.9.0 bower_components/chai

mocha#1.17.1 bower_components/mocha
filer@0.0.28 node_modules/filer
└── base64-arraybuffer@0.1.2

[othiym23]

Sorry for the hair trigger, @jbuck – we've been cleaning out a bunch of issues related #6043 and press-ganged yours prematurely.

I'm actually not sure what the right thing to do in your case is. Is there a reason you're depending on bower directly? (I'm assuming that there is, and you're using a reduced package.json here.) If you just install filer alone, it installs without issue because it installs bower as its own dependency. It seems like filer should have a peerDependency relationship with bower (and probably grunt), because it does depend on them being installed for its scripts to run. I think this is probably a bug in filer's packaging, but this is also relevant to the work that @iarna is doing on multi-stage install, so I'm sending it her way as well.

[jbuck]

Sorry, I left one part out of my example. a-poor-app and filer both run bower install in their postinstall scripts.

I'm fine with putting bower in as a peerDependency to ensure that it's installed before the postinstall script is run, but I think that should be documented somewhere as a valid use case for peerDependency. I guess it's just weird how the dependency chain is changed when you run npm install in a-poor-app vs filer.

[othiym23]

It's definitely weird, and it's something that npm should handle better than it does. I'm going to add this to the multi-stage install milestone, because one of the things that @iarna is hoping to address with that work is making the whole interaction between installation and lifecycle scripts simpler and more deterministic.

[chrisskilton]

Great thread. has there been any movement on this, or is the peerDependency solution the way to go?

[othiym23]

@drvanhalen The behavior @jbuck is seeing is a bug, but my hope is that multi-stage install / npm@3 will fix this as a byproduct of it being more or less a rewrite of the installer process, and that will solve the problem. It may or may not get fixed prior to the release of npm@3 (which, to be clear, is in the relatively near future).

[chrisskilton]

@othiym23 thanks very much.

[illegalnumbers]

+1

[iarna]

This'll be solved by #6926, so I'm going to close this ticket and further discussion should occur over there.