NPM "lifecycle" scripts order: install/postinstall scripts being run before all dependencies are finished installing

[JamesMGreene]

$ node -v
v0.10.26

$ npm -v
1.4.3

$ sw_vers
ProductName:    Mac OS X
ProductVersion: 10.9.2
BuildVersion:   13C64

I've encountered some problems with a module of my that has an install script: it seems that the script is run before all of the module's dependencies (devDependencies?) are finished installing. This seems like a bug.

This also happens even if I update the "package.json" to execute it during the postinstall phase instead.

[shindakun]

Can you update to 1.4.6 and check to see if it's still occurring?

[JamesMGreene]

@shindakun: Thanks for the suggestion. Still occurring in 1.4.6, though, unfortunately.

[JamesMGreene]

Alternatively: is there any way to install all dependencies (a la npm install) without triggering the preinstall/install/postinstall NPM scripts?

[JamesMGreene]

BTW: Just a hunch but... I suspect this may have something to do with installing dependencies that do not install synchronously (e.g. they have their own install scripts that go download an external resource asynchronously). At least one of my module's dependencies is setup like that.

Ideally, the NPM install/postinstall scripts would not be executed until all of the dependencies' installations are completely resolved.

[timoxley]

they have their own install scripts that go download an external resource asynchronously

What do you mean by "asynchronously"?

Can you give us some concrete samples of what you're doing?

[JamesMGreene]

@timoxley: I'll try to make a test project for you.

[JamesMGreene]

OK, I seem to have been wrong about the asynchronousity being the issue. Rather, it seems like it has to do with a dependency's dependency's dependency (which is also a direct dependency of the main module) not being fully installed before the dependency's install script (which relies on the dependency's dependency) is run.

Logical dependency tree:

flex-sdk@3.0.0-3  (main module)
├── download@0.1.12
|   └── ...
├── playerglobal-latest@0.1.6 node_modules/playerglobal-latest
|   ├── download-github-repo@0.1.2
|   |   └── download@0.1.12
|   |       └── ...
|   └── ...
└── ...

Physical dependency tree after installation is complete:

flex-sdk@3.0.0-3  (main module)
├── download@0.1.12
|   └── ...
├── playerglobal-latest@0.1.6 node_modules/playerglobal-latest
|   ├── download-github-repo@0.1.2
|   └── ...
└── ...

Diff:
Notably, the download module dependency does not get installed within the download-github-repo/node_modules/, rather it seems to rely on the main module's installation of download.

The Problem:
While this difference is fine in theory (once everything is installed), the reality is that the playerglobal-latest module has an install script that attempts to make use of its download-github-repo dependency in order to fetch a GitHub repo upon install. However, as the download-github-repo's dependency on download is not installed as a sub-dependency and just-so-happens to have not completely installed at the ancestral level, the download-github-repo throws an exception due to missing dependencies when the playerglobal-latest module's install script tries to use it.

The Fix:
So, AFAIK, the correct fix for this problem would be to either:

• Not rely on ancestral node_modules installations (i.e. install an identical copy of download under download-github-repo/node_modules/), which could be theoretically be limited to only modules that have install/postinstall NPM scripts configured; or

• Ensure that any module (i.e. playerglobal-latest) that has both

  • a dependency (i.e. download-github-repo) which relies on an ancestral dependency (i.e. download); and
  • an install/postinstall script [that uses dependencies]

  ...does not execute its install/postinstall script unless its full/pertinent dependency tree (including ancestrally stored dependencies) is completely resolved.

Short-Term Workarounds:
If anyone else encounters this before it gets fixed in NPM, there are two effective (though very unsavory) short-term workarounds that came to mind:

• Eliminate one of the shared dependencies, i.e. don't allow the main module (i.e. flex-sdk) to rely on the same shared dependency (i.e. download)
• Update the version requirement for one of the dependencies to not satisfy the other (i.e. too old and exact, or too new). For this to work, you'll need to explore how each dependency's version requirement is specified.

[JamesMGreene]

Most Successful Short-Term Workaround:

• Add a preinstall script that installs the top-level dependency which contains the controversial sub-sub-dependency. In my case, that means preinstall should run npm install playerglobal-latest.

[JamesMGreene]

This seems to be a long-standing issue described in #1341 (comment).

[JamesMGreene]

Another generic workaround that would probably work for most people's use cases:

"preinstall": "npm install --ignore-scripts"

Or just manually running:

npm install --ignore-scripts
npm install

...instead of a normal, standalone npm install.

[Wildhoney]

@JamesMGreene An excellent workaround! Thank you.

[Qix-]

@JamesMGreene That doesn't work for me.

npm ERR! Error: Attempt to unlock ., which hasn't been locked
npm ERR!     at unlock (/usr/local/lib/node_modules/npm/lib/utils/locker.js:44:11)
npm ERR!     at cb (/usr/local/lib/node_modules/npm/lib/cache/add-local.js:30:5)
npm ERR!     at /usr/local/lib/node_modules/npm/lib/cache/add-local.js:47:20
npm ERR!     at /usr/local/lib/node_modules/npm/lib/utils/locker.js:30:7
npm ERR!     at /usr/local/lib/node_modules/npm/node_modules/lockfile/lockfile.js:167:38
npm ERR!     at OpenReq.Req.done (/usr/local/lib/node_modules/npm/node_modules/graceful-fs/graceful-fs.js:144:5)
npm ERR!     at OpenReq.done (/usr/local/lib/node_modules/npm/node_modules/graceful-fs/graceful-fs.js:64:22)
npm ERR!     at Object.oncomplete (fs.js:107:15)

This is something really simple. Not sure why a fix hasn't been proposed yet.

[JamesMGreene]

Looks like the forthcoming multi-stage install (#5919) strategy should resolve this.

[iarna]

Closing this as this is being fixed in #6926

[misak113]

The workaround "preinstall": "npm install --ignore-scripts" above doesn't work for me. It make infinit loop of installing npm dependencies.
I create script which install all deps localy from package.json: buildDependencies & add execute of it to install scripts:

"buildDependencies": {
  "depName": "~v1.x"
},
"scripts": {
  "preinstall": "node preinstall.js",
  "postinstall": "someUsesDepName_v1.x"
}

File preinstall.js:

var npm = require("npm");
var conf = require("./package.json");
var deps = conf.buildDependencies;
var args = [];
Object.keys(deps).forEach(function (name) {
    args.push(name + "@" + deps[name]);
});
npm.load(conf, function (e) {
    if (e) return console.error(e);
    npm.commands.install(args, function (e) {
        if (e) return console.error(e);
        console.info("Installed npm modules: ", args);
    });
});

It definitely works...
Example: Asimplia/module-util@cd2dd3f

[krasimir]

@misak113 nice suggestion. Thanks. However, in my case preinstall.js complains that there is missing module npm. Isn't it that a dependency to your solution. I mean preinstall.js will work if you have already npm installed. Or may you have that module as a global one.

[misak113]

In my version of node.js (v0.10.25) you can have global module required in your app like local. I don't know if it can be required in earlier version.
If you have node application you should have installed npm globaly (not necessery). So this solution should work in most cases.
I acknowledge that solution is not perfect. There are no caching & always is buildDependencies reinstalled forcely after run npm install.
Hope that npm implements something like this using native way. Something like @iarna plans.

[krasimir]

Thanks for the answer. On my machine to use npm programmatically I have to install it as a local dependency (v0.10.33, npm's version: 1.4.28)

[misak113]

Maybe it should be in EnvVar NODE_PATH=/usr/lib/node_modules on some OS (Win etc.) http://stackoverflow.com/questions/15636367/nodejs-require-a-global-module-package #2076 http://stackoverflow.com/questions/12594541/npm-global-install-cannot-find-module
My dev OS is Linux Mint, our server OS is Linux Ubuntu.

[krasimir]

Hm ... In my case I have a postinstall script that runs Gulp. However, my gulp setup requires modules that are listed as dependencies in the upper modules. So npm does not want to install them locally for that particular module. I ended up with the following solution:

"postinstall": "node ./Gulpfile.js make"

And then in my gulpfile.js I just wait till all the dependencies are installed:

if(process.argv && process.argv[2] === 'make') {
  var deps = [
    'gulp', 'gulp-rename', 'gulp-browserify', 'gulp-plumber',
    'gulp-util', 'gulp-uglify', 'ractive', 'gulp-tap'
  ], index = 0;
  (function doWeHaveAllDeps() {
    if(index === deps.length) {
      gulp.start('make');
      return;
    } else if(isModuleExists(deps[index])) {
      index += 1;
      doWeHaveAllDeps();
    } else {
      setTimeout(doWeHaveAllDeps, 500);
    }
  })();
};

function isModuleExists( name ) {
  try { return !!require.resolve(name); }
  catch( e ) { return false }
}