npm@3 wants to be faster

[iarna]

npm@3 is sometimes slower than npm@2, though this is highly tree dependent. It is doing more, but all the same, folks'd like it to be as fast as it can be. Profiling would be grand. ;) This ticket exists as the tracker for npm@3 performance.

• Put together a minimal benchmark to work against
  • Should do a from-scratch install of something big
  • Add a module to a large install
  • Remove a module from a large install
  • List a large install

Ideally I'd like this benchmark to be 2.x and 3.x compatible so we can directly compare different parts.

[okket]

Yes, please improve the speed:

→ time npm -g ls --depth=0
/Users/okketimm/.nvm/versions/node/v0.12.7/lib
├── bower@1.4.1
├── ember-cli@1.13.1
├── eslint@0.24.0
├── npm@3.1.1
└── phantomjs@1.9.17

real 0m14.436s
user 0m14.377s
sys  0m0.616s

→ npm version
{ npm: '3.1.1',
  http_parser: '2.3',
  modules: '14',
  node: '0.12.7',
  openssl: '1.0.1p',
  uv: '1.6.1',
  v8: '3.28.71.19',
  zlib: '1.2.8' }

versus

→ time npm -g ls --depth=0
/Users/okketimm/.nvm/versions/node/v0.12.7/lib
├── bower@1.4.1
├── ember-cli@1.13.1
├── eslint@0.24.0
├── npm@2.11.3
└── phantomjs@1.9.17

real 0m0.570s
user 0m0.531s
sys  0m0.069s

→ npm version
{ npm: '2.13.0',
  http_parser: '2.3',
  modules: '14',
  node: '0.12.7',
  openssl: '1.0.1p',
  uv: '1.6.1',
  v8: '3.28.71.19',
  zlib: '1.2.8' }

[junosuarez]

I ran into a huge slowdown the first time doing a global install after upgrading from 2 to 3. with loglevel=silly on, it spent over a minute on
npm sill install cloneCurrentTreeToIdealTree loadIdealTree

• npm -v = 3.1.0
• node -v = 0.10.36
• win8.1 / x64

[trusktr]

I just wanted to add that I've noticed npm3 is considerably slower. Running npm install in a project where everything is already installed still takes a considerably long time. The expectation would be that npm install finishes super fast because there's nothing to install, nothing to do.

[othiym23]

@trusktr This is one of the biggest differences between npm@2 and npm@3 – in a lot of ways, the mostly flat tree is a side show, and this dependency tree realization process is the primary change to the installer. Before the installer can decide that there's nothing for it to do, it has to both realize the current dependency tree, and construct an ideal dependency tree, and compare the two to see if there's anything that needs to be done. npm@2 didn't do this, which is why running npm install would never pick up any changes below the top-level dependencies. The current installer is a lot safer, but it needs to do a bunch more work to do what it does.

That said, this issue is an acknowledgment that making installs fast is an important feature in its own right, and that we aren't there yet.

[trusktr]

But I know it's getting there and is about to be super awesome. I can't wait for it to be the new default.

By the way, JSPM has flat dependency tree management, and it's fast! Could there be ideas to borrow from it's open source goodness?

[glenjamin]

I'm seeing quite an extreme difference, here's a sample dependency list that reproduces.

  "dependencies": {
    "accounting": "^0.4.1",
    "ampersand-app": "^1.0.4",
    "ampersand-collection": "^1.4.5",
    "ampersand-model": "^5.0.3",
    "ampersand-rest-collection": "^4.0.0",
    "ampersand-router": "^3.0.2",
    "autoprefixer-core": "^5.2.1",
    "babel": "^5.6.23",
    "babel-core": "^5.7.4",
    "babel-loader": "^5.3.2",
    "babelify": "^6.1.3",
    "blue-tape": "^0.1.10",
    "browserify": "^11.0.0",
    "classnames": "^2.1.3",
    "css-loader": "^0.15.5",
    "eslint": "^0.24.1",
    "file-loader": "^0.8.4",
    "float": "^1.0.2",
    "hjs-webpack": "^2.10.0",
    "json-loader": "^0.5.2",
    "local-links": "^1.4.0",
    "lodash.has": "^3.2.1",
    "lodash.isundefined": "^3.0.1",
    "lodash.random": "^3.0.1",
    "lodash.uniqueid": "^3.0.0",
    "node-libs-browser": "^0.5.2",
    "node-sass": "^3.2.0",
    "normalize.scss": "^0.1.0",
    "phantomjs": "^1.9.17",
    "pluralize": "^1.1.2",
    "postcss-loader": "^0.5.1",
    "react": "^0.13.3",
    "react-hot-loader": "^1.2.8",
    "sass-loader": "^1.0.2",
    "style-loader": "^0.12.3",
    "stylus-loader": "^1.2.1",
    "tap-spec": "^4.0.2",
    "tape": "^4.0.1",
    "tape-catch": "^1.0.4",
    "tape-run": "^1.0.0",
    "url-loader": "^0.5.6",
    "validator": "^3.41.3",
    "webpack": "^1.10.1",
    "webpack-dev-server": "^1.10.1",
    "xhr": "^2.0.3"
  }

> PATH=/opt/npm/bin:$PATH time sh -c 'npm ls 2> /dev/null > /dev/null'; npm version
       20.52 real        20.52 user         0.55 sys
{ thing: '1.0.0',
  npm: '2.11.2',
  http_parser: '2.3',
  modules: '14',
  node: '0.12.6',
  openssl: '1.0.1o',
  uv: '1.6.1',
  v8: '3.28.71.19',
  zlib: '1.2.8' }

> time sh -c 'npm ls 2> /dev/null > /dev/null'; npm version

real    0m3.198s
user    0m3.169s
sys 0m0.541s
{ thing: '1.0.0',
  npm: '2.11.2',
  http_parser: '2.3',
  modules: '14',
  node: '0.12.6',
  openssl: '1.0.1o',
  uv: '1.6.1',
  v8: '3.28.71.19',
  zlib: '1.2.8' }

I wonder if it might be feasible to cache the dependency resolution based on a hash of package.json or the declared dependencies?

[mblakele]

@iarna "Profiling would be grand": https://gist.github.com/mblakele/4f1eefd6b4ad5543cfb6

This is something of a noop, running npm install with 3.2.2 when there's no work to be done. It takes about 6-7x longer than npm 2.14.3 does: 30-40 sec vs 5 sec.

As predicted by @othiym23 much of the time seems to be spent realizing packages. I notice fs.stat is prominent, and that's with everything on an SSD. Might be worth checking to make sure that the same packages aren't checked multiple times, though.

[mourner]

Another extreme case, installing eslint is 5 times slower — very very painful :(

time npm install eslint

# v3.3.3
real  0m48.122s
user  0m8.783s
sys   0m2.051s

# v2.4.15
real  0m10.561s
user  0m7.597s
sys   0m2.538s

[badsyntax]

I would go as far as to say most of the installation tasks performed by npm 3 are obviously slower than npm 2. This is a regression! Not progression at all. The title of this issue needs to be changed to npm@v3 needs to be faster because v3 cannot be released with these perf problems.

[othiym23]

Now that npm@3 is in wider release, @iarna is going to spending most of her time fixing critical issues in it as they're found, and analyzing and improving npm's performance. One of the primary purposes of npm 3 was to make the install process more robust and correct, and that means that in many (but not all) cases it's doing substantially more work than older versions were. Improving the performance of the CLI without regressing on robustness is the CLI team's #1 priority right now.

[vjeux]

If you want another data point. react-native is taking 1m53s on npm 3.3.3 and only 40s on npm 2.13.2, a 2.8x increase. The good news is that the number of files went from 25887 to 13261, only half of files :)

Repro steps:

git clone https://github.com/facebook/react-native.git
cd react-native

rm npm-shrinkwrap.json
rm -Rf node_modules
npm install -g npm
time npm install
# real  1m46.743s
# user  0m51.520s
# sys   0m12.608s
find node_modules | wc -l
# 25887

rm npm-shrinkwrap.json
rm -Rf node_modules
npm install -g npm@2.13.2
time npm install 
# real  0m40.008s
# user  0m38.214s
# sys   0m19.352s
find node_modules | wc -l
# 13261

[Vanuan]

From all those time reports it's quite obvious that while user and system times are the same, real time is considerably bigger. It means that npm3 is either waiting on I/O or is not utilizing all CPU cores. It's quite ironic, because the selling point of node was "it's very fast because it's not blocked on I/O, and it doesn't need threads".

So JavaScript is not as efficient for scripting. Even more ironic.

[cchamberlain]

Many of the people on this thread do not seem to understand that npm 3 was a major rearchitecture. Comments about flipping a switch to jump between npm 2 and npm 3 installs are naive (if you want npm 2, install npm 2...). npm is the most platform agnostic package manager in existence and if you think that npm 3 was a regression try using each version on Windows for a few months. npm 2 is a mess on Windows (long file paths...), installs are often insurmountable and require bypassing packages. npm 3 has brought incredible stability. On npm 2, removal of node_modules was a daily occurrence and now I can get faster and more stable results with normal incremental npm install. Bottom line, I spend orders of magnitude less hours hacking npm 3 than npm 2 because it actually works. If you are doing constant npm installs from scratch than maybe you should optimize your build process instead of trolling the awesomely helpful npm team. If you can rewrite the logic for speed and maintain the stability that npm 3 brings than submit your pull request.

[vroudge]

@cchamberlain This is just feedback. Build processes have become a pain in the ass lately because of this issue ; because yes, it is an issue, with speed.

I'm just reporting the great deal of time an install now takes. What is it these days that you can't provide feedback to an open-source project without people saying you're trolling the contrib team.

Am I ungrateful by saying that a product got worse for the use I make of it?

[cchamberlain]

@vroudge - npm 3 is slower than npm 2. It is well documented and no secret. It's slower because it fixes many npm 2 issues. It will get faster but the use you make of it is a very small piece of the puzzle. Speeds great but it's much more important that it does everything it claims it can do without randomly bombing out. Chiming in that it's slower is not useful, if you prefer the npm 2 nested installs you should use that since they are still maintaining it.

[vroudge]

@cchamberlain Does not change the fact that it is an issue and is seen as an issue by the maintainers of the project.

The new modules flat structure is really wonderful and I love it. And npm works wonders, really. It's just really slow, that's it. When you see things like the fact that the progress bar makes it reaaally slower, you just begin to wonder if it could not be made faster, and that's it. I don't chime in just saying that it's slower just to QQ at the fact it's slower, there are many things that make it slower, not just the new tree. I just think it should be investigated. Not that I have the knowledge to do it.

Lets stop this here and stop polluting the issues.

[Vanuan]

@cchamberlain oh, we've got who to blame - windows guys. Why won't you just start using more developer-friendly operating systems

[naholyr]

@cchamberlain it fixes nothing, it brought new issues instead (some conceptual like the fact you can require a sub-dependency now without warning, some very important for an install tool like partial installs with no feedback), and fixed a non-issue for non-Windows developers. It could even have been fixed by a specific command as the too deep folders could still be removed programmatically.

[shehi]

I am on Ubuntu and to install laravel-elixir@5 it took me 30+ mins with 8Mbps connection. It never took this long before when I was in v2. I am working through Docker.

[dsernst]

so it seems like the speed question is pretty well established at this point. Is there a clear path for continuing to move forward? Can we update the OP comment with links to subtasks so we can start knocking them out?

[tfennelly]

Hi @othiym23. Thanks for all your work on npm in general and on npm@3 (making it more robust etc).

Back in December, you said you were hoping that some cycles would be spent on npm@3 performance within the following month or so. Are there any updates on that?

[othiym23]

This is why I typically avoid making date-based estimates. ;)

The CLI team has been making substantial progress in rewriting npm's test suite to make it less hairy, and to pass on Windows. The latter, especially, is an important quality milestone for the project, which is why the team is focused on test work for the time being.

After that, the team will be tackling the list of significant known issues. After that is when we've scheduled getting the whole team working on improving performance, which will most likely include some architectural work as well.

All that said, that's just what the CLI team itself is focusing on. The team continues to land community PRs as quickly as it can; several recent PRs have resulted in significant performance improvements in some cases.

Thanks to all for their contributions and patience.

[iamstarkov]

@othiym23 several months passed. can you share some kind of progress on this issue?

[othiym23]

The CLI team has been making substantial progress in rewriting npm's test suite to make it less hairy, and to pass on Windows.

This is (mostly) done. There's some wiggliness on AppVeyor when running the CI suite, and it's required some TLC from me that I haven't had time to provide. The test suite, however, is in a much better place than it was in in March.

After that, the team will be tackling the list of significant known issues.

This work is still in progress (see the label, which should give you an idea of how many issues are in that bucket). It's finicky, demanding work, and the team remains small, so progress is moderate, but real.

We've also landed several patches, including the new version of gauges and are-we-there-yet in npmlog, with (almost exclusively positive) performance implications. I believe that overall the performance of the CLI is significantly better, with fewer pathological corner cases, than it was in March. That said, there's still a ways to go before the team can turn its energies towards making more wholesale improvements to the CLI's performance at a deeper, more architectural level. When we do start work on that, we'll say as much here.

[othiym23]

Gosh darn it, I keep closing this issue. Freudian close?

[othiym23]

No, wait, it was closed already! ;_; so confused!

[mourner]

I believe that overall the performance of the CLI is significantly better, with fewer pathological corner cases, than it was in March.

There has been some progress, but unfortunately not nearly enough to consider this issue closed. NPM 3 still has consistently 2-3x slower install times than NPM 2, and 5-6x slower times than popular, actively developed alternatives like pnpm and ied (which also flatten the dependency tree) — you can confirm this on any popular Node package, and the alternatives (each having half a thousand commits and tens of contributors) wouldn't even exist in the first place if this wasn't a critical problem.

I'm thinking it will take a month or two to start working on performance, not a year.

It has been 9 months since the comment above...

[mourner]

P.S. I highly appreciate the hard work you're doing and apologize for being annoying. Just trying to keep performance closer on your radar because it significantly affects my life as an open source maintainer.

[vjpr]

Is there an open issue to track progress with performance?

I recently tried out pnpm and it is incredibly fast. It makes Node fun again. I think the community should invest more in pnpm and ied, than expect npm to improve. I have a feeling the performance issues are too deep, and would require a huge amount of re-architecting.

Npm3 has done great work in pushing the idea of the deduped by default node_modules but the non-determinism seems hacky, when pnpm solves this perfectly.

The Node resolver is over-complicated. The pnpm model of having a central store with all modules installed as module-name@version is simple, elegant, and easy to understand.

What would be best is if when requiring a module, the resolver would read the closest package.json file, and find the best version to use from a central store.

If two require('foo') statements resolved to two different absolute paths, then a warning would be shown to help the user update package version ranges to prevent two modules being evaled. At present npm linking causing huge amounts of duplicate module evals. Most importantly this fixes performance issues with npm linking which involves evaling many copies of the same module/version.

I have implemented all of this in https://www.npmjs.com/package/live-perf and have seen huge speedups. The only downside is the resolver needs to be cached, because the dynamic resolving has an overhead as every require statement needs to find its closest package.json file with syscalls.

I think these changes to the Node resolver would make the ultimate package manager.

The end goal should be not to ever worry about whats in node_modules, and be able to rely on auto-installing ala https://github.com/siddharthkp/auto-install.

Imagine being able to write code without ever having to run npm install or rm -rf node_modules.

[halhenke]

Its been a while since i looked at the internals of npm (did a tiny PR) or pnpm (meant to do a fairly big feature but....never did 😞 ) but i think its fair to point out that npm does deal with a lot more edge cases/supported methods of installation (git repos etc) than these other clients.

OTOH i do think there is a fundamental flaw (at least from a performance perspective) in the way that npm3 tries to build a model of the ideal tree before going about the business of pulling down packages...again the details escape me but - as mentioned before the npm 3 package resolution/install algorithm is very complicated and quite tightly coupled to a lot of other parts. My knowledge is fuzzy now and quite possibly out of date but i do think a more or less ground up re-implementation/rethink of how packages are resolved/fetched would be necessary to see notable performance gains. Just my thoughts/opinions.

[pannous]

All the euphemisms and kindness aside:

time npm -g ls
real	0m45.109s

Labels: big-bug good
Closed: why?

These suggestions alleviated the situation dramatically:

npm set progress=false
mv /usr/local/lib/node_modules /usr/local/lib/node_modules.TOO_MANY
cp -r /usr/local/lib/node_modules.TOO_MANY/npm /usr/local/lib/node_modules/

Lesson learned: use global modules very sparingly

[iarna]

@pannous npm ls -g is going to take as long as it takes to list all your global modules in logical tree form, but lots of global modules shouldn't have any impact on any other operation. If you are seeing an impact from having a lot of global modules on any operation other than npm ls -g then please open an issue!

PS disabling your progress bar really REALLY shouldn't have any measurable impact on run time. (I've not seen a benchmark yet where it did.) The progress bar is extremely low impact these days.

[MattFoley]

One thing I've noticed is that performance seems to drop considerably when your package.json contains urls to specific commits, rather than npm packages.

[vvo]

@MattFoley I believe this is something to expect, when specifying a git commit you then have to fire up git command, clone, checkout or something like that. Way slower than downloading a tgz

[MattFoley]

@vvo I think you're right, but maybe there are some improvements that could be made? Maybe a shallow checkout? It may not even be the git checkout that's causing the hang, not sure.

Seems like it hangs on fetchMetadata, although I don't know enough about npm to really say.