npm@3 wants to be faster #8826

Closed
iarna opened this Issue Jul 4, 2015 · 115 comments

Projects

None yet
@iarna
Member
iarna commented Jul 4, 2015

npm@3 is sometimes slower than npm@2, though this is highly tree dependent. It is doing more, but all the same, folks'd like it to be as fast as it can be. Profiling would be grand. ;) This ticket exists as the tracker for npm@3 performance.

  • Put together a minimal benchmark to work against
    • Should do a from-scratch install of something big
    • Add a module to a large install
    • Remove a module from a large install
    • List a large install

Ideally I'd like this benchmark to be 2.x and 3.x compatible so we can directly compare different parts.

@iarna iarna added this to the 3.x milestone Jul 4, 2015
@okket
okket commented Jul 10, 2015

Yes, please improve the speed:

β†’ time npm -g ls --depth=0
/Users/okketimm/.nvm/versions/node/v0.12.7/lib
β”œβ”€β”€ bower@1.4.1
β”œβ”€β”€ ember-cli@1.13.1
β”œβ”€β”€ eslint@0.24.0
β”œβ”€β”€ npm@3.1.1
└── phantomjs@1.9.17

real 0m14.436s
user 0m14.377s
sys  0m0.616s

β†’ npm version
{ npm: '3.1.1',
  http_parser: '2.3',
  modules: '14',
  node: '0.12.7',
  openssl: '1.0.1p',
  uv: '1.6.1',
  v8: '3.28.71.19',
  zlib: '1.2.8' }

versus

β†’ time npm -g ls --depth=0
/Users/okketimm/.nvm/versions/node/v0.12.7/lib
β”œβ”€β”€ bower@1.4.1
β”œβ”€β”€ ember-cli@1.13.1
β”œβ”€β”€ eslint@0.24.0
β”œβ”€β”€ npm@2.11.3
└── phantomjs@1.9.17

real 0m0.570s
user 0m0.531s
sys  0m0.069s

β†’ npm version
{ npm: '2.13.0',
  http_parser: '2.3',
  modules: '14',
  node: '0.12.7',
  openssl: '1.0.1p',
  uv: '1.6.1',
  v8: '3.28.71.19',
  zlib: '1.2.8' }
@jsdnxx
jsdnxx commented Jul 21, 2015

I ran into a huge slowdown the first time doing a global install after upgrading from 2 to 3. with loglevel=silly on, it spent over a minute on
npm sill install cloneCurrentTreeToIdealTree loadIdealTree

  • npm -v = 3.1.0
  • node -v = 0.10.36
  • win8.1 / x64
@trusktr
trusktr commented Jul 29, 2015

I just wanted to add that I've noticed npm3 is considerably slower. Running npm install in a project where everything is already installed still takes a considerably long time. The expectation would be that npm install finishes super fast because there's nothing to install, nothing to do.

@othiym23
Contributor

@trusktr This is one of the biggest differences between npm@2 and npm@3 – in a lot of ways, the mostly flat tree is a side show, and this dependency tree realization process is the primary change to the installer.Β Before the installer can decide that there's nothing for it to do, it has to both realize the current dependency tree, and construct an ideal dependency tree, and compare the two to see if there's anything that needs to be done. npm@2 didn't do this, which is why running npm install would never pick up any changes below the top-level dependencies. The current installer is a lot safer, but it needs to do a bunch more work to do what it does.

That said, this issue is an acknowledgment that making installs fast is an important feature in its own right, and that we aren't there yet.

@trusktr
trusktr commented Jul 29, 2015

But I know it's getting there and is about to be super awesome. I can't wait for it to be the new default.

By the way, JSPM has flat dependency tree management, and it's fast! Could there be ideas to borrow from it's open source goodness?

@glenjamin
Contributor

I'm seeing quite an extreme difference, here's a sample dependency list that reproduces.

  "dependencies": {
    "accounting": "^0.4.1",
    "ampersand-app": "^1.0.4",
    "ampersand-collection": "^1.4.5",
    "ampersand-model": "^5.0.3",
    "ampersand-rest-collection": "^4.0.0",
    "ampersand-router": "^3.0.2",
    "autoprefixer-core": "^5.2.1",
    "babel": "^5.6.23",
    "babel-core": "^5.7.4",
    "babel-loader": "^5.3.2",
    "babelify": "^6.1.3",
    "blue-tape": "^0.1.10",
    "browserify": "^11.0.0",
    "classnames": "^2.1.3",
    "css-loader": "^0.15.5",
    "eslint": "^0.24.1",
    "file-loader": "^0.8.4",
    "float": "^1.0.2",
    "hjs-webpack": "^2.10.0",
    "json-loader": "^0.5.2",
    "local-links": "^1.4.0",
    "lodash.has": "^3.2.1",
    "lodash.isundefined": "^3.0.1",
    "lodash.random": "^3.0.1",
    "lodash.uniqueid": "^3.0.0",
    "node-libs-browser": "^0.5.2",
    "node-sass": "^3.2.0",
    "normalize.scss": "^0.1.0",
    "phantomjs": "^1.9.17",
    "pluralize": "^1.1.2",
    "postcss-loader": "^0.5.1",
    "react": "^0.13.3",
    "react-hot-loader": "^1.2.8",
    "sass-loader": "^1.0.2",
    "style-loader": "^0.12.3",
    "stylus-loader": "^1.2.1",
    "tap-spec": "^4.0.2",
    "tape": "^4.0.1",
    "tape-catch": "^1.0.4",
    "tape-run": "^1.0.0",
    "url-loader": "^0.5.6",
    "validator": "^3.41.3",
    "webpack": "^1.10.1",
    "webpack-dev-server": "^1.10.1",
    "xhr": "^2.0.3"
  }
> PATH=/opt/npm/bin:$PATH time sh -c 'npm ls 2> /dev/null > /dev/null'; npm version
       20.52 real        20.52 user         0.55 sys
{ thing: '1.0.0',
  npm: '2.11.2',
  http_parser: '2.3',
  modules: '14',
  node: '0.12.6',
  openssl: '1.0.1o',
  uv: '1.6.1',
  v8: '3.28.71.19',
  zlib: '1.2.8' }
> time sh -c 'npm ls 2> /dev/null > /dev/null'; npm version

real    0m3.198s
user    0m3.169s
sys 0m0.541s
{ thing: '1.0.0',
  npm: '2.11.2',
  http_parser: '2.3',
  modules: '14',
  node: '0.12.6',
  openssl: '1.0.1o',
  uv: '1.6.1',
  v8: '3.28.71.19',
  zlib: '1.2.8' }

I wonder if it might be feasible to cache the dependency resolution based on a hash of package.json or the declared dependencies?

@vjpr vjpr referenced this issue in live-js/live-cli Aug 9, 2015
Open

module modify workflow #2

@iarna iarna added the blocker label Aug 11, 2015
@iarna iarna modified the milestone: 3.x-next-next, 3.x, 3.x-next Aug 17, 2015
@mblakele
mblakele commented Sep 9, 2015

@iarna "Profiling would be grand": https://gist.github.com/mblakele/4f1eefd6b4ad5543cfb6

This is something of a noop, running npm install with 3.2.2 when there's no work to be done. It takes about 6-7x longer than npm 2.14.3 does: 30-40 sec vs 5 sec.

As predicted by @othiym23 much of the time seems to be spent realizing packages. I notice fs.stat is prominent, and that's with everything on an SSD. Might be worth checking to make sure that the same packages aren't checked multiple times, though.

@iarna iarna modified the milestone: 3.x, 3.x-next Sep 11, 2015
@mourner
mourner commented Sep 18, 2015

Another extreme case, installing eslint is 5 times slower β€” very very painful :(

time npm install eslint

# v3.3.3
real  0m48.122s
user  0m8.783s
sys   0m2.051s

# v2.4.15
real  0m10.561s
user  0m7.597s
sys   0m2.538s
@othiym23 othiym23 added big-bug and removed blocker labels Sep 18, 2015
@badsyntax

I would go as far as to say most of the installation tasks performed by npm 3 are obviously slower than npm 2. This is a regression! Not progression at all. The title of this issue needs to be changed to npm@v3 needs to be faster because v3 cannot be released with these perf problems.

@othiym23
Contributor

Now that npm@3 is in wider release, @iarna is going to spending most of her time fixing critical issues in it as they're found, and analyzing and improving npm's performance. One of the primary purposes of npm 3 was to make the install process more robust and correct, and that means that in many (but not all) cases it's doing substantially more work than older versions were. Improving the performance of the CLI without regressing on robustness is the CLI team's #1 priority right now.

@vjeux
vjeux commented Sep 20, 2015

If you want another data point. react-native is taking 1m53s on npm 3.3.3 and only 40s on npm 2.13.2, a 2.8x increase. The good news is that the number of files went from 25887 to 13261, only half of files :)

Repro steps:

git clone https://github.com/facebook/react-native.git
cd react-native
rm npm-shrinkwrap.json
rm -Rf node_modules
npm install -g npm
time npm install
# real  1m46.743s
# user  0m51.520s
# sys   0m12.608s
find node_modules | wc -l
# 25887
rm npm-shrinkwrap.json
rm -Rf node_modules
npm install -g npm@2.13.2
time npm install 
# real  0m40.008s
# user  0m38.214s
# sys   0m19.352s
find node_modules | wc -l
# 13261
@timdp
timdp commented Sep 22, 2015

Is there any way to get more useful progress reporting? Increasing verbosity is great but it doesn't help in the common case where you basically want an indication of how long the install is going to take. Even a very rough estimate would be better than nothing.

@sindresorhus sindresorhus referenced this issue in PolymerElements/polymer-starter-kit Sep 23, 2015
Closed

audit npm dependencies #337

@sindresorhus

We're seeing a huge regression in npm@3 in polymer-starter-kit.

The following package.json takes ~2 minutes to install on npm@2, but ~10-14 minutes on npm@3.

Node.js 4.1.1. OS X 10.10.

Here's a npm-shrinkwrap.json of the dependencies for reproducibility.

@contra
contra commented Sep 24, 2015

Installs went from ~4mins -> ~12mins when switching from 2.2.18 to 3.3.3

Would recommend anything time sensitive (deployments) stay on 2.x

@mourner
mourner commented Sep 24, 2015

@iarna any progress updates on this? The original issue description suggests it is "sometimes slower" when it's in fact more like "always 3 to 7 times slower", which is pretty critical and prevents a lot of people from upgrading.

@iarna
Member
iarna commented Sep 30, 2015

@mourner Well, contrary to popular belief, it's actually 5-10 times faster in some scenarios too, but apparently none that you're using. =/

But everyone here will be happy to learn that we identified the source of almost all of the slow down and excised it this week. Starting with npm@3.3.6 we should be seeing substantial performance improvements any time your node_modules has a lot of modules in it.

@iarna
Member
iarna commented Sep 30, 2015

The ultimate cause of this was lodash/lodash#1496 – we've addressed this by changing one function to mutate instead of returning a modified clone, and changing another to use a content-aware tree copy function, that can be faster than anything generic.

We are still using lodash's deepclone in a few places, so the work that's gone in there will still be helpful, but it's now less critical.

@iarna iarna added a commit that referenced this issue Sep 30, 2015
@iarna iarna logical-tree: Make it mutate to improve performance
The clone we were doing to save ourselves from mutation proved to be
excessively slow. While lodash is being updated to not have the
same scaling issues, it was determined that we weren't gaining
anything from not mutating here, so there was no reason to pay
even a lesser price.

PR-URL: #9803
Fixes: #8826
725d28a
@iarna iarna added a commit that referenced this issue Sep 30, 2015
@iarna iarna install: Switch from clone to a tree-copier that's aware of our datas…
…tructures

Lodash's deep-clone is being updated to not have the catostrophic scalaing issues
it had when this patch was written, but even still, using something tailor-made
will necessarily be faster.

PR-URL: #9803
Fixes: #8826
fa77770
@vvo
vvo commented Sep 30, 2015

Niiice, I know this is stupid but I had good performance on simple json.parse(json.stringify()), did you try it?

@donaldpipowitch

Awesome. Thank you for the update!

@mourner
mourner commented Oct 1, 2015

@iarna wow, that's really great to hear! And surprising β€” I was afraid it's bound by network operations... Glad that was not the case. Looking forward to the update.

@iarna iarna added a commit that closed this issue Oct 1, 2015
@iarna iarna logical-tree: Make it mutate to improve performance
The clone we were doing to save ourselves from mutation proved to be
excessively slow. While lodash is being updated to not have the
same scaling issues, it was determined that we weren't gaining
anything from not mutating here, so there was no reason to pay
even a lesser price.

PR-URL: #9803
Fixes: #8826
cf42217
@iarna iarna closed this in cf42217 Oct 1, 2015
@iarna iarna added a commit that referenced this issue Oct 1, 2015
@iarna iarna install: Switch from clone to a tree-copier that's aware of our datas…
…tructures

Lodash's deep-clone is being updated to not have the catostrophic scalaing issues
it had when this patch was written, but even still, using something tailor-made
will necessarily be faster.

PR-URL: #9803
Fixes: #8826
7bc0d4c
@mgol
mgol commented Oct 2, 2015

in npm@3.3.6 time npm -g ls went down for me from ~20 seconds to ~5. Still more than in npm@2 (I understand this is unavoidable with the new architecture) but it's definitely easier to swallow!

@JaKXz
Contributor
JaKXz commented Oct 2, 2015

Any ideas why I'm not getting at v3.3.6 with these commands? I ran both just before time of writing (13:15 EST).

$ npm install -g npm
/usr/local/bin/npm -> /usr/local/lib/node_modules/npm/bin/npm-cli.js
/usr/local/lib
└── npm@3.3.5

$ npm install -g npm@latest
/usr/local/bin/npm -> /usr/local/lib/node_modules/npm/bin/npm-cli.js
/usr/local/lib
└── npm@3.3.5
@SimenB
Contributor
SimenB commented Oct 2, 2015

npm i -g npm@next
Run npm show npm dist-tags to see what's available

@sindresorhus

@JaKXz Not the place to ask support questions. Open a new issue.

@greenkeeperio-bot greenkeeperio-bot referenced this issue in allain/servicify Oct 9, 2015
Closed

Update npm to version 3.3.6 πŸš€ #3

@just-boris

Installed the latest available version 3.3.8. But 2.x still seems to be faster.

@just-boris

Also, here is few numbers. Tested in docker latest node image:
docker run -it node:4.1.2 bash

Test setup:

git clone https://github.com/kriasoft/react-starter-kit.git
cd react-starter-kit
npm install
rm -rf node_modules/

First install is required to warm up npm cache and reduce network effects

Results
npm 2.14.4

real    3m34.238s
user    3m50.120s
sys 2m5.650s

npm 3.3.8

real    7m1.421s
user    3m29.200s
sys 0m40.610s
@tomitrescak

On windows the performace decrease is even more striking. What I timed as 67 seconds on NPM2 takes 4 minutes on NPM 3 ;(

@iarna
Member
iarna commented Oct 13, 2015

npm@3 has to be slower because it does a lot more (npm@2 would see a module in your tree and assume all of its dependencies were ok, npm@3 actually goes and checks, this obviously takes time). That doesn't mean we don't want it to be faster, but it does mean that some operations are never going to match the npm@2 times without cutting the features added w/ npm@3.

@tomitrescak That difference in times seems more extreme than I would expect. What version of did you see this with? Can you share the package.json you were using?

@nmccready

Anyway to set a global setting in .bashrc to disable these set of features? I think what most people are getting at is that they prefer the speed over the features. I mean no dis-respect at all but until it is optional or sped up I plan on using npm@2.X .

It might be better yet to opt-in to these features to double check for the extra safety if needed.

@goloroden

@nmccready +1 … IMHO speed is the non-plus-ultra here. If I really want to be sure that all dependencies are correctly met, I run

$ rm -rf node_modules
$ npm install

and I'm fine with that, because this is the only way to ensure you get the same result twice. Hence this has to be fast. In times of Docker and immutable environments I think that installing from scratch is much more important than e.g. updating a previous installation.

As @nmccready already said, I too do not want to be disrespectful, but for cases like the ones described above the (dramatically) slower speed of npm@3 is a show-stopper for now.

@iarna
Member
iarna commented Oct 13, 2015

@nmccready It's not as simple as putting in an if to restore npm@2's behavior, as it was a complete rearchitecting of the install paths, but something like that isn't impossible, if perhaps more difficult then it appears at first.

@goloroden Eh, I think speeds in the same ballpark of npm@2 are achievable for fresh installs like that.

That said "because this is the only way to ensure you get the same result twice" is exactly what npm@3 is intended to work toward eliminating as a necessary pattern. (A command that says "make my node_modules look like it would if there wasn't one here when I started" is a likely addition, and actually possible w/ npm@3.)

@iarna
Member
iarna commented Oct 13, 2015

(Please don't take the closing of this ticket as a sign that we don't care about further performance improvements. This was closed because we found the thing that was making npm go non-linear and thus actually impossible to use. Like, commands that could take days to complete.)

@kidwm
kidwm commented Oct 14, 2015

@iarna Thank you for above explanations and awesome work!

@nelix
nelix commented Oct 27, 2015

I think npm@3 is great.
All I need is a fast npm install-exactly-shrinkwrap option for CI
I don't mind if commands that change shrinkwrap or package.json is slow.

@timruffles

@iarna out of interest, what was causing the non-linearity?

@iarna
Member
iarna commented Oct 27, 2015

@timruffles lodash.cloneDeep had a cycle-detection algorithm that involved doing a linear search of all objects seen previously.

@mourner
mourner commented Oct 27, 2015

@iarna for a use case of a clean install of a module, does NPM 3 still do much more work than NPM 2? npm install eslint is still taking about 4-5 times as much as it did with NPM 2, and most of the time seems to be network requests.

@thisguychris

I found that npm3's performance is very close to npm2. Sometimes even faster. As long as you do
npm cache clean. I think the reason why a lot of people seeing npm2 is faster is that most of the time it just links the modules that's already there. Zapping the cache would level the playing field.

@mourner
mourner commented Nov 2, 2015

@deezahyn @iarna OK, I just repeated the test, doing both cached and uncached npm install eslint on NPM 2 and NPM 3. Please take a look at the log:

~/projects/test β†’ npm -v
2.14.4
~/projects/test β†’ npm cache clean
~/projects/test β†’ rm -rf node_modules/
~/projects/test β†’ time npm install eslint
...
real    0m14.305s
user    0m10.596s
sys 0m3.621s
~/projects/test β†’ rm -rf node_modules/
~/projects/test β†’ time npm install eslint
...
real    0m11.009s
user    0m9.259s
sys 0m3.012s
~/projects/test β†’ npm install -g npm@latest
/usr/local/bin/npm -> /usr/local/lib/node_modules/npm/bin/npm-cli.js
npm@3.3.10 /usr/local/lib/node_modules/npm
~/projects/test β†’ npm -v
3.3.10
~/projects/test β†’ npm cache clean
~/projects/test β†’ rm -rf node_modules/
~/projects/test β†’ time npm install eslint
...
real    0m44.157s
user    0m9.972s
sys 0m2.328s
~/projects/test β†’ rm -rf node_modules/
~/projects/test β†’ time npm install eslint
...
real    0m30.585s
user    0m9.407s
sys 0m2.208s

Full log here: https://gist.github.com/mourner/d4e6f81da5e974aa9275

NPM 3 is 3x slower than NPM 2 in both cases (cached and uncached). Can you explain what's going on? Should I repeat the test case with some additional commands?

@naholyr
naholyr commented Nov 4, 2015

The test being each time rm -rf node_modules; time npm install, I have npm 3 being 3 times slower than npm 2 too, in all cases.

Without npm cache clean:

  • npm@2 npm install 81,31s user 10,32s system 131% cpu 1:09,65 total
  • npm@latest npm install 132,62s user 7,32s system 58% cpu 4:00,74 total
  • npm@next npm install 131,06s user 7,45s system 66% cpu 3:27,69 total

With npm cache clean:

  • npm@2 npm install 82,84s user 11,37s system 50% cpu 3:06,38 total
  • npm@latest npm install 143,66s user 9,66s system 27% cpu 9:21,68 total
  • npm@next npm install 132,42s user 8,05s system 28% cpu 8:18,89 total

The package.json:

{
  "dependencies": {
    "angular": "^1.4.7",
    "angular-animate": "^1.4.7",
    "angular-bootstrap-colorpicker": "^3.0.19",
    "angular-masonry": "github:delapouite/angular-masonry#nojquery",
    "angular-off-click": "github:lmtm/angular-off-click#npm",
    "angular-sanitize": "^1.4.7",
    "angular-socket-io": "^0.7.0",
    "angular-ui-bootstrap": "^0.14.3",
    "angular-ui-router": "^0.2.15",
    "async": "^1.5.0",
    "bluebird": "^3.0.5",
    "body-parser": "^1.14.1",
    "bootstrap": "^3.3.5",
    "bootstrap-styl": "^4.0.5",
    "busboy": "^0.2.11",
    "chalk": "^1.1.1",
    "commander": "^2.9.0",
    "connect-assets": "^5.0.1",
    "connect-flash": "~0.1.1",
    "connect-mongo": "~0.8.2",
    "connect-redis": "^3.0.1",
    "cors": "^2.7.1",
    "debug": "^2.2.0",
    "deepmerge": "~0.2.10",
    "ect": "~0.5.9",
    "eslint": "^1.8.0",
    "express": "^4.13.3",
    "express-session": "^1.12.1",
    "file-loader": "^0.8.4",
    "gm": "^1.21.1",
    "hiredis": "~0.4.1",
    "html-loader": "^0.3.0",
    "imap": "^0.8.16",
    "imports-loader": "^0.6.5",
    "jquery": "^2.1.4",
    "less": "^2.5.3",
    "lodash": "~3.10.1",
    "mailparser": "0.5.3",
    "masonry-layout": "^3.3.2",
    "method-override": "^2.3.5",
    "mime": "^1.3.4",
    "minimatch": "^3.0.0",
    "moment": "^2.10.6",
    "mongoose": "~4.2.4",
    "mongoose-schema-extend": "^0.1.7",
    "mongoose-timestamp": "^0.4.0",
    "nconf": "~0.8.2",
    "ng-annotate-loader": "0.0.10",
    "ng-file-upload": "^9.1.2",
    "ngtemplate-loader": "^1.3.1",
    "node-transloadit": "0.0.4",
    "node-uuid": "~1.4.3",
    "nodemailer": "^1.8.0",
    "passport": "^0.3.0",
    "passport-local": "^1.0.0",
    "random-string": "~0.1.2",
    "raw-loader": "^0.5.1",
    "redis": "^2.3.0",
    "request": "^2.65.0",
    "serve-favicon": "^2.3.0",
    "serve-static": "^1.10.0",
    "slugify2": "0.0.2",
    "socket.io": "^1.3.7",
    "socket.io-client": "^1.3.7",
    "striptags": "^2.0.4",
    "talon": "^2.3.2",
    "temporary": "0.0.8",
    "transloadit-api": "^1.0.0",
    "wb-angular-slugify": "^1.2.0",
    "winston": "^2.1.0"
  },
  "devDependencies": {
    "aggregate-commands": "^2.1.0",
    "awsbox": "^0.7.0",
    "babel-core": "^6.0.20",
    "babel-eslint": "^4.1.3",
    "babel-loader": "^6.0.1",
    "chai": "^3.4.0",
    "chai-as-promised": "^5.1.0",
    "compression-webpack-plugin": "^0.2.0",
    "css-loader": "^0.21.0",
    "env-test": "^1.0.0",
    "http-server": "^0.8.5",
    "mocha": "^2.3.3",
    "nodemon": "^1.8.1",
    "protractor": "^2.2.0",
    "rewire": "^2.3.4",
    "style-loader": "^0.13.0",
    "stylus": "^0.52.4",
    "stylus-loader": "^1.4.2",
    "supertest": "^1.1.0",
    "watch-exec": "^1.2.0",
    "webpack": "^1.12.2",
    "webpack-dev-server": "^1.12.1",
    "xhr2": "^0.1.3"
  }
}
@iamstarkov

still an issue

@donaldpipowitch

Yeah, we made the same joke.

@mciparelli

Google redirected me here as soon as I was looking for answers to the extreme slowness I'm experiencing. I was sure it was something I was doing wrong. Looks like it wasn't like that. More like npm maintainers decided to release a major version that makes you wait. A lot.
I don't know what's inside (regarding code and architechture) guys, but version 2 has been working pretty well for me. You should've tested a bit harder before releasing npm@3.

@nmccready

Why is this issue closed again?

@naholyr
naholyr commented Nov 5, 2015

@nmccready because all this has been fixed since 3.3.6 obviously πŸ™Š

@nmccready

I take it that is a jk cause obviously the community feels it's still sub par.

@naholyr
naholyr commented Nov 5, 2015

Yeah sorry if it was not obvious, I even gave some data here yesterday about that slowness ;)

@andreldm
andreldm commented Nov 7, 2015

When I started playing with Node.js one of the greatest things was npm's lightning speed compared to Ruby's gem and Python's pip. But now 3.x is painfully slow, I'm just trying to install simple tools like yo and its generators but this feat takes more than 10 minutes. I can't run a benchmark right now, but npm 2.x used to be much more faster(certainly no more than 2 min). I truly appreciate all the work you guys are doing, a tree as flat as possible is great, but I'm really considering to stick to 2.x until installing anything doesn't require me to halt my workflow and find something else to do meanwhile.

@vjpr
vjpr commented Nov 7, 2015

@andre I had the exact same thought the other day. I remember how awesome
npm felt compared to gem. But now it's painfully slow. Even running a local
npm cache like npm lazy doesn't help. I have a large project that takes
20mins for fresh npm install using 3. Any additional npm install something
takes at least 1 minute.

On Sat, 7 Nov 2015 at 11:12 PM, AndrΓ© Miranda notifications@github.com
wrote:

When I started playing with Node.js one of the greatest things was npm's
lightning speed compared to Ruby's gem and Python's pip. But now 3.x is
painfully slow, I'm just trying to install simple tools like yo and its
generators but this feat takes more than 10 minutes. I can't run a
benchmark right now, but npm 2.x used to be much more faster(certainly no
more than 2 min). I truly appreciate all the work you guys are doing, a
tree as flat as possible is great, but I'm really considering to stick to
2.x until installing anything doesn't require me to halt my workflow and
find something else to do meanwhile.

β€”
Reply to this email directly or view it on GitHub
#8826 (comment).

@jkroso
jkroso commented Nov 8, 2015

Does npm even use its cache? Forget comparing it to v2, it should be easy to make it properly quick.

@othiym23
Contributor
othiym23 commented Nov 8, 2015

Does npm even use its cache?

Yes†. However, for each dependency@version package ID, the CLI fires off a request to the registry to ensure that its cache already contains the newest version of that particular package / version pair. This is necessary because some private registry software allows users to push new versions of tarballs as the same version (as did the primary npm registry, until early last year). If you run npm install --loglevel=http to get the old-style logging, you'll see that the vast majority of these requests come back nearly instantly with HTTP 304 - Not Modified, which allows the rest of the install code to use the cached version of package metadata and tarballs.

Three things take time in the npm installer:

  1. Collecting metadata about dependencies from the registry and other places where dependencies are hosted and computing the ideal tree.
  2. Downloading tarballs into npm's cache.
  3. Extracting and placing tarballs.

npm@3 is frequently slower than npm@2 because it looks at the whole dependency tree, instead of just the top-level dependencies. This fixes a huge swath of inconsistencies, edge cases, and bugs in the npm@2 installer, but since it's doing more work, it takes longer. All three of the network access, computation, and filesystem activity are expensive -- there isn't a single magic wand we can wave to make the installer as fast as it was in npm@2.

There are a few things we can do to improve npm's network performance:

  1. Rewrite npm's cache to be content-addressable. This would allow us to do things like write package tarball content hashes into npm-shrinkwrap.json, thus removing the need to poll the registry or redownload (probably unchanged) package tarballs.
  2. Replace request with a dedicated, npm-specific download manager. This is actually more of a robustness than a performance fix, as limiting the number of concurrent connections could reduce performance on fast, reliable connections, but on the average, I think it will probably reduce overhead significantly.
  3. Start looking into using SPDY or HTTP 2 to bundle requests. This could result in substantial improvements, but it requires changes to the primary registry that are fairly complex, and that team has its own road map that's pretty full for a while.

There are also a host of things we can do to improve the performance of the rest of the installer, which is where the biggest gains in performance in npm@3 have thus far been realized. The team is concerned about npm@3's performance, and will continue to work on improving it. At the same time, we have our hands full ensuring that npm@3 is as stable and reliable as possible, and right now our attention is on robustness more than performance. If people want to try making changes to improve npm's performance, or want to see if they can find specific hot spots within the installer where performance could be improved, that would be helpful. More general-purpose benchmarks don't really do much except tell us that npm@3 is slower than npm@2 in many common cases, which we already know.

† Except in the case of shrinkwrapped packages, where the resolved tarball needs to be downloaded each time to ensure that you get the same bits for subsequent installs as you did when you originally shrinkwrapped the package. This behavior is unchanged from npm@2 and older, though.

@nmccready

npm@3 is frequently slower than npm@2 because it looks at the whole dependency tree, instead of just the top-level dependencies

How does one make this optional? IE where does this code begin so someone can dive in and work on this.

@iamstarkov

i dont think it is possible to make it optional, ’cause it is the whole point of v3

@iamstarkov

however I also want to know, where are the benchmarks, how to run them and so on.

@othiym23
Contributor
othiym23 commented Nov 8, 2015

How does one make this optional?

@iamstarkov is right – dependency tree realization is the point of npm@3, and isn't a behavior we're going to make optional. However, if you read this section of the release process, you'll see that npm@2 is going to get bug fixes for more or less the duration of Node 4's LTS release cycle, so you can stick with that.

however I also want to know, where are the benchmarks, how to run them and so on.

An outstanding project for the CLI team is to build a usable benchmarking environment. Simply timing wall-clock time of installs (with or without primed caches) doesn't really say a whole lot about how npm is going to run for other users; ideally we'd have some kind of prebaked AMI that had tools like perf or DTrace included so that we could get an accurate picture of how much network I/O, disk I/O, memory usage, GC activity, and CPU time were consumed by CLI runs. One of the CLI team will probably get around to making something like this eventually, but if somebody really wanted to do us (and the developer community) a favor, they could take a stab at putting such a thing together.

@nmccready

Well thanks for clearing that up. Well I look forward to mentioned possible improvements. Thanks guys for your hard work.

@mourner
mourner commented Nov 8, 2015

npm@3 is frequently slower than npm@2 because it looks at the whole dependency tree, instead of just the top-level dependencies. This fixes a huge swath of inconsistencies, edge cases, and bugs in the npm@2 installer, but since it's doing more work, it takes longer.

@othiym23 specifically in the case of a clean install from scratch, isn't npm@2 downloading all packages from the dependency tree as well, just because there aren't any? Why would npm@3 be consistently 3-5 times slower in this case if it doesn't fetch more packages (and may even fetch less because of the flattened structure in case of shared deps)?

I can understand the slowdown for things like npm update, but for a clean install, it's absolutely not clear why it would do more work, especially 3-5 times more work, which is a severe regression that NPM devs don't seem to acknowledge.

@glenjamin
Contributor

I've started doing a bit of profiling / analysis on this - the angular heavy package.json posted above appears to be particularly pathological.

One initial observation is that npm@3 splits install into distinct phases - loadDeps is likely network (concurrent connections & bandwidth) limited, but the subsequent step extractTarballs will be limited on local IO.

It is conceptually possible to overlap these phases - but I've yet to look into the code so I have no idea how much more complex this might make the implementation.

@glenjamin
Contributor

Here's some output of my initial analysis: https://github.com/glenjamin/slow-npm

I've noticed that there's a few modules where their package information is requested multiple times from the registry in a single execution - from the numbers shown in https://github.com/glenjamin/slow-npm/blob/fff311e2caecc4466bb847709c170833e06afc2c/duplicate-requests.txt this probably won't make a huge difference - but does seem like an area which is doing more work than is needed.

The CPU profiling graph doesn't show anything particularly scary at first glance, although are-we-there-yet is higher than I would have expected it doesn't seem like optimisations there would make a large impact - any major improvements will likely need to be architectural changes rather than code execution optimisations.

Requesting the registry metadata does seem like it dominates execution time, I wonder if some ideas could be borrowed from https://www.youtube.com/watch?v=JjYAnBhF2JU - specifically the part about keeping track of overall registry changes via a global version number. This could allow provision of an endpoint which would allow clients to ask "have any of these modules changed since version X"

@naholyr
naholyr commented Nov 9, 2015

As I imagined it, I thought npm3 was following those steps :

  • fetch the dependency tree (should be fast)
  • optimize it (should be very fast)
  • install packages from optimized tree (should be faster than npm2 as there are less packages to download)

Maybe it's what is done, but I dont get how it can be so slower than npm2 unless it makes thousands of requests to fetch the tree. Wouldn't a qui fix be to build an HTTP endpoint which, given a list of packages, returns this whole tree in one single request?

@glenjamin
Contributor
  • fetch the dependency tree (should be fast)

This makes one request per package - this is actually beneficial from an HTTP caching point of view - making an endpoint which bundles these together will ruin any transport layer cachability.

  • optimize it (should be very fast)

This is CPU bound, and potentially computationally expensive, but doesn't appear to be a bottleneck at the moment

  • install packages from optimized tree (should be faster than npm2 as there are less packages to download)

should be the same, yes - Judging by the progress bar this is actually split into download / unpack / preinstall / postinstall.

I think npm2 interleaved these stages more than npm3 currently does - and I would imagine the clearer separation makes the code / process much easier to follow. There might be space to overlap these phases at a sub-tree level as a future optimisation.

@naholyr
naholyr commented Nov 9, 2015

When you talk about caching you're refering to server-side caching right ? Because the client has actually more benefit running an uncached fast single request than dozens of (even if faster) multiple requests.

@iamstarkov

I'm not sure it's possible to cache npm queries because their have to be relevant at any time. if some of the deps has been updated then cache for current package should be cleared. and all the versions of all the packages have to be handled this way. it's incredible I think

@othiym23
Contributor
othiym23 commented Nov 9, 2015

Building the logical ideal tree includes:

  1. reading the installed physical tree from disk
  2. fetching basic metadata for packages, if changed
  3. scanning packages for shrinkwrap files and applying shrinkwrap data to the tree
  4. scanning packages for bundledDependencies and deciding what to do with the bundled versions (which may not match package.json, in which case npm should replace them (while warning))

So while in theory, it would be nice to have a clear separation between fetching pure metadata and downloading or extracting package tarballs, in practice that's not always possible without complicated conditional logic. Some or all of this could be accelerated by moving more of the data into registry end points, and the team has plans to do that eventually (at least for some cases). It isn't easy or quick, though, because changes to the registry require the registry's time and careful design to ensure the abilities of old versions of the CLI to keep working.

Older versions of npm did in fact intermingle the different phases, and that had at least two major consequences. One is that it was indeed incredibly hard to understand what and why things were happening in the installer, especially once shrinkwrap and bundledDependencies were involved (the latter, in fact, was an endless source of subtle bugs and subtler regressions as bugs were patched). The other is that the self-bootstrapping package tree generated an enormous number of simultaneous network requests, which as packages have grown larger and more complex has started putting unrealistic levels of load on devs' network stacks and connections. npm@3 gives us tools to manage both these things, but it's going to take a while before the team has both robustness and performance nailed down.

@iamstarkov

looking forward for this and please don't underestimate community: just provide tools and benchmarks with some intro docs and eventually someone will fix it for all of us

@andrenarchy

On my machine (Ubuntu 14.04.3 64bit), the progress bar (npm@3.3.6) is reproducibly responsible for a factor 4 slowdown:

$ npm set progress=false
$ rm -rf node_modules
$ time npm install
[...]
real    5m10.601s
user    4m56.435s
sys 0m36.003s
$ npm set progress=true
$ rm -rf node_modules
$ time npm install
[...]
real    21m13.415s
user    20m36.752s
sys 0m42.010s

Maybe the progress bar's console i/o is a hand brake? ;)

@mourner
mourner commented Nov 11, 2015

@andrenarchy I was hopeful when I saw you comment, but unfortunately it doesn't improve performance on my use cases like npm install eslint. It could be a factor when installing a huge amount of tiny modules though.

@andrenarchy

@mourner probably there's more than one issue here. :)

I forgot to mention my package.json.

@nschloe
nschloe commented Nov 11, 2015

I see about 70% longer install user times with progress=true:

$ npm set progress=false
$ rm -rf node_modules
$ time npm install
[...]
real    3m26.846s
user    1m29.196s
sys 0m9.560s
$ npm set progress=true
$ rm -rf node_modules
$ time npm install
[...]
real    4m49.531s
user    2m33.868s
sys 0m10.616s
@mourner
mourner commented Nov 11, 2015

The progress bar issue should be very easy to fix β€” just throttling progress updates to e.g. 100ms.

@othiym23
Contributor

See #10380 for a detailed discussion of the whys and wherefores of npm@3 network performance and how it affects the overall performance of the CLI.

@jadbox
jadbox commented Nov 29, 2015

NPM3 & 3.5: I can catch an Uber ride faster than my project's npm install.
(v2 @ ~45s vs ~5m) (i7/SSD/16gb ddr3)

@trusktr
trusktr commented Dec 9, 2015

Talking of speed, everyone should check this out: https://github.com/alexanderGugel/ied (it's being renamed to "nom" soon).

@alistairjcbrown

Just to add some more stats for disabling the progress bar. Using npm v3.3.12 (bundled with node v5.2.0). This project has a shrinkwrap file. The install cache was warmed with an initial install.

Disabling the progress bar reduced install time from 13.6 minutes to 2.9 minutes (~20% of the original time)

$ npm set progress=true
$ rm -rf node_modules
$ time npm install
...
npm install  818.98s user 58.21s system 103% cpu 14:09.81 total
$ npm set progress=false
$ rm -rf node_modules
$ time npm install
...
npm install  171.19s user 54.13s system 119% cpu 3:08.65 total
@iamstarkov

my stats for generator-travis:

$ node -v && npm -v
v5.2.0
3.5.2
npm i 1:14 74s 100% # progress=true (default)
npm i 1:08 68s  91% # progress=false
npm i 1:26 86s 116% # --legacy-bundling progress=true 
npm i 1:23 83s 112% # --legacy-bundling progress=false

apparently, --legacy-bundling make things even worser. Disabling progress gains only 9%

@Kurtz1993

npm@3 still slow as a snail! Even for removing packages.
I think you might remove all those logs when performing an operation.

@vroudge
vroudge commented Dec 16, 2015

Crazy slow here too.

@ankitduseja

Same here. Npm2 was way faster. Even facebook's react-native official documentation (https://facebook.github.io/react-native/docs/getting-started.html) is now referring to use npm2. That landed me here.

@donaldpipowitch

Really? It is their official recommendation? :(

@mourner
mourner commented Dec 16, 2015

The new open ticket about this is here: #10380

The basic gist of it is that it's 3-5 times slower because it "makes it much easier to understand what npm is doing". Maintainers say there are no obvious ways to bring NPM3 on par with NPM2 performance, and they'll rather be focusing on compatibility issues for the next year at least. :(

@btecu
btecu commented Dec 16, 2015

There's also ied soon to be renamed to nom: https://github.com/alexanderGugel/ied

@Kurtz1993

I think thy should let us choice if we want or not know what's going under the hood.
In my case, I don't really care about NPM's internal operations.

@nmccready

Probably too late for that. What would be best is making 4.X off 2.X and
start over. Cut your losses or be phased out by newer coolness. Speed is
what matters.

On Wed, Dec 16, 2015, 12:16 Luis Rogelio HernΓ‘ndez LΓ³pez <
notifications@github.com> wrote:

I think thy should let us choice if we want or not know what's going under
the hood.
In my case, I don't really care about NPM's internal operations.

β€”
Reply to this email directly or view it on GitHub
#8826 (comment).

VR,

Sent from my mobile device

Nicholas McCready
nmccready@gmail.com

@othiym23
Contributor

rather be focusing on compatibility issues for the next year at least.

Right now,the CLI team is working on getting its test suite passing consistently under Travis on all versions of Node it supports. Once that's done, we're doing the same, but for Windows as well as UNIX, which will involve setting up Windows CI. This is long-overdue payment of some significant technical debt. After that work is complete, our two next priorities are fixing the outstanding bugs in the installer (there are a fair few we already know about), and improving npm@3's performance. Software takes as long as it takes, but I'm thinking it will take a month or two to start working on performance, not a year.

The basic gist of it is that it's 3-5 times slower because it "makes it much easier to understand what npm is doing".

That's a fairly ungenerous reading of what @iarna and I have said. npm@3's installer has a goal to be robust, and part of that is making it understandable so we can troubleshoot and debug it when it's behaving erratically, and so developers like you, who lack our intimate knowledge of the installer's design and implementation, can have a better understanding of why npm is doing what it's doing. Previous versions of the installer were exceedingly difficult to understand when e.g. the bundle dependency algorithm was doing unexpected things.

Maintainers say there are no obvious ways to bring NPM3 on par with NPM2 performance

This simply isn't true. It will require the involvement of more than the CLI team at npm, and it's going to take some time, but I'm fully confident that we'll return npm's performance to at least on par with npm@2.

@othiym23
Contributor

In my case, I don't really care about NPM's internal operations.

You don't care until it does something unexpected or wrong, and then you (in general, maybe not you in particular) care a great deal, because until somebody understands what's happening, you're blocked (potentially along with the rest of your team). 90+% of the time, npm-debug.log is the only tool available to the CLI team to understand buggy behavior, making it all but essential that it capture as much information as possible about what's going on with your installs.

@Kurtz1993

@othiym23 that's why I said this: "They should let us choice if we want or not know what's going under the hood."
If a problem occurs I will want to know what went wrong, of course.

@iamstarkov

@othiym23 is there any details/reasons behind chosen priorities? I mean why tech debt have been considered more important than community's problems. I don't know full picture, so I will be happy to know

@othiym23
Contributor

is there any details/reasons behind chosen priorities?

The npm CLI team includes one manager (me) and two full-time engineers, and has a very heavy workload. We don't have dedicated QA resources, so our only indicators of the quality of the product we're shipping are the number of inbound issues we see and our test suite. If we don't trust that the test suite is telling us that our code base is in good shape, we're in trouble. Right now, the team has insufficient faith in the test suite. So we need to fix that!

Also, it tremendously simplifies the process of landing contributions from the community if we can look at the Travis build and know whether the new code causes any problems without pulling down changes and running the tests ourselves. And it's an important part of our commitment to Windows users to have continuous integration testing things on their platform.

Finally, the internals of npm are sufficiently difficult to understand that most users of npm never make the effort, and when they do, very few of them master it enough to make meaningful contributions to the core. The architecture is complex and subtle, and part of the reason npm@3 exists is to make it more comprehensible, so that the CLI team is no longer so much of a gatekeeper for finding and fixing issues in the installer.

We've put off paying down this debt for way too long, to the point that it's significantly impairing the ability of the project to add new features (and increase the size of the contributor base). It's not that this technical debt is more important than the community's problems, it's that it is a community problem. The community problem, in my estimation.

@iamstarkov

@othiym23 thank you

@Zarel
Zarel commented Jan 26, 2016

You don't care until it does something unexpected or wrong, and then you (in general, maybe not you in particular) care a great deal, because until somebody understands what's happening, you're blocked (potentially along with the rest of your team). 90+% of the time, npm-debug.log is the only tool available to the CLI team to understand buggy behavior, making it all but essential that it capture as much information as possible about what's going on with your installs.

It sounds like this should be off by default, then. If a command fails, the user can re-run it with logging enabled, thus solving both the performance problem and the debugging problem.

@josephgodwinkimani

I have experienced the same problem for the last two days trying to npm install dependencies for meanjs but I opted to use Windows command processor. It did it in record time obviously the progress bar got halfway..hope we can find a solution. issue #11283 and issue #8826

@Vanuan
Vanuan commented Jan 28, 2016

From all those time reports it's quite obvious that while user and system times are the same, real time is considerably bigger. It means that npm3 is either waiting on I/O or is not utilizing all CPU cores. It's quite ironic, because the selling point of node was "it's very fast because it's not blocked on I/O, and it doesn't need threads".

So JavaScript is not as efficient for scripting. Even more ironic. :trollface:

@cchamberlain

Many of the people on this thread do not seem to understand that npm 3 was a major rearchitecture. Comments about flipping a switch to jump between npm 2 and npm 3 installs are naive (if you want npm 2, install npm 2...). npm is the most platform agnostic package manager in existence and if you think that npm 3 was a regression try using each version on Windows for a few months. npm 2 is a mess on Windows (long file paths...), installs are often insurmountable and require bypassing packages. npm 3 has brought incredible stability. On npm 2, removal of node_modules was a daily occurrence and now I can get faster and more stable results with normal incremental npm install. Bottom line, I spend orders of magnitude less hours hacking npm 3 than npm 2 because it actually works. If you are doing constant npm installs from scratch than maybe you should optimize your build process instead of trolling the awesomely helpful npm team. If you can rewrite the logic for speed and maintain the stability that npm 3 brings than submit your pull request.

@vroudge
vroudge commented Jan 28, 2016

@cchamberlain This is just feedback. Build processes have become a pain in the ass lately because of this issue ; because yes, it is an issue, with speed.

I'm just reporting the great deal of time an install now takes. What is it these days that you can't provide feedback to an open-source project without people saying you're trolling the contrib team.

Am I ungrateful by saying that a product got worse for the use I make of it?

@cchamberlain

@vroudge - npm 3 is slower than npm 2. It is well documented and no secret. It's slower because it fixes many npm 2 issues. It will get faster but the use you make of it is a very small piece of the puzzle. Speeds great but it's much more important that it does everything it claims it can do without randomly bombing out. Chiming in that it's slower is not useful, if you prefer the npm 2 nested installs you should use that since they are still maintaining it.

@vroudge
vroudge commented Jan 28, 2016

@cchamberlain Does not change the fact that it is an issue and is seen as an issue by the maintainers of the project.

The new modules flat structure is really wonderful and I love it. And npm works wonders, really. It's just really slow, that's it. When you see things like the fact that the progress bar makes it reaaally slower, you just begin to wonder if it could not be made faster, and that's it. I don't chime in just saying that it's slower just to QQ at the fact it's slower, there are many things that make it slower, not just the new tree. I just think it should be investigated. Not that I have the knowledge to do it.

Lets stop this here and stop polluting the issues.

@Vanuan
Vanuan commented Jan 28, 2016

@cchamberlain oh, we've got who to blame - windows guys. Why won't you just start using more developer-friendly operating systems :trollface:

@naholyr
naholyr commented Jan 28, 2016

@cchamberlain it fixes nothing, it brought new issues instead (some conceptual like the fact you can require a sub-dependency now without warning, some very important for an install tool like partial installs with no feedback), and fixed a non-issue for non-Windows developers. It could even have been fixed by a specific command as the too deep folders could still be removed programmatically.

@davidknezic davidknezic referenced this issue in axa-ch/style-guide Jan 31, 2016
Merged

Improve Docker build times #445

2 of 5 tasks complete
@shehi
shehi commented Feb 28, 2016

I am on Ubuntu and to install laravel-elixir@5 it took me 30+ mins with 8Mbps connection. It never took this long before when I was in v2. I am working through Docker.

@dsernst
dsernst commented Mar 2, 2016

so it seems like the speed question is pretty well established at this point. Is there a clear path for continuing to move forward? Can we update the OP comment with links to subtasks so we can start knocking them out?

@tfennelly

Hi @othiym23. Thanks for all your work on npm in general and on npm@3 (making it more robust etc).

Back in December, you said you were hoping that some cycles would be spent on npm@3 performance within the following month or so. Are there any updates on that?

@othiym23
Contributor
othiym23 commented Mar 7, 2016

This is why I typically avoid making date-based estimates. ;)

The CLI team has been making substantial progress in rewriting npm's test suite to make it less hairy, and to pass on Windows. The latter, especially, is an important quality milestone for the project, which is why the team is focused on test work for the time being.

After that, the team will be tackling the list of significant known issues. After that is when we've scheduled getting the whole team working on improving performance, which will most likely include some architectural work as well.

All that said, that's just what the CLI team itself is focusing on. The team continues to land community PRs as quickly as it can; several recent PRs have resulted in significant performance improvements in some cases.

Thanks to all for their contributions and patience.

@iamstarkov

@othiym23 several months passed. can you share some kind of progress on this issue?

@othiym23
Contributor
othiym23 commented Sep 7, 2016

The CLI team has been making substantial progress in rewriting npm's test suite to make it less hairy, and to pass on Windows.

This is (mostly) done. There's some wiggliness on AppVeyor when running the CI suite, and it's required some TLC from me that I haven't had time to provide. The test suite, however, is in a much better place than it was in in March.

After that, the team will be tackling the list of significant known issues.

This work is still in progress (see the label, which should give you an idea of how many issues are in that bucket). It's finicky, demanding work, and the team remains small, so progress is moderate, but real.

We've also landed several patches, including the new version of gauges and are-we-there-yet in npmlog, with (almost exclusively positive) performance implications. I believe that overall the performance of the CLI is significantly better, with fewer pathological corner cases, than it was in March. That said, there's still a ways to go before the team can turn its energies towards making more wholesale improvements to the CLI's performance at a deeper, more architectural level. When we do start work on that, we'll say as much here.

@othiym23
Contributor
othiym23 commented Sep 7, 2016

Gosh darn it, I keep closing this issue. Freudian close?

@othiym23 othiym23 reopened this Sep 7, 2016
@othiym23
Contributor
othiym23 commented Sep 7, 2016

No, wait, it was closed already! ;_; so confused!

@othiym23 othiym23 closed this Sep 7, 2016
@mourner
mourner commented Sep 7, 2016

I believe that overall the performance of the CLI is significantly better, with fewer pathological corner cases, than it was in March.

There has been some progress, but unfortunately not nearly enough to consider this issue closed. NPM 3 still has consistently 2-3x slower install times than NPM 2, and 5-6x slower times than popular, actively developed alternatives like pnpm and ied (which also flatten the dependency tree) β€” you can confirm this on any popular Node package, and the alternatives (each having half a thousand commits and tens of contributors) wouldn't even exist in the first place if this wasn't a critical problem.

I'm thinking it will take a month or two to start working on performance, not a year.

It has been 9 months since the comment above...

@mourner
mourner commented Sep 7, 2016

P.S. I highly appreciate the hard work you're doing and apologize for being annoying. Just trying to keep performance closer on your radar because it significantly affects my life as an open source maintainer.

@vjpr
vjpr commented Sep 7, 2016 edited

Is there an open issue to track progress with performance?

I recently tried out pnpm and it is incredibly fast. It makes Node fun again. I think the community should invest more in pnpm and ied, than expect npm to improve. I have a feeling the performance issues are too deep, and would require a huge amount of re-architecting.

Npm3 has done great work in pushing the idea of the deduped by default node_modules but the non-determinism seems hacky, when pnpm solves this perfectly.

The Node resolver is over-complicated. The pnpm model of having a central store with all modules installed as module-name@version is simple, elegant, and easy to understand.

What would be best is if when requiring a module, the resolver would read the closest package.json file, and find the best version to use from a central store.

If two require('foo') statements resolved to two different absolute paths, then a warning would be shown to help the user update package version ranges to prevent two modules being evaled. At present npm linking causing huge amounts of duplicate module evals. Most importantly this fixes performance issues with npm linking which involves evaling many copies of the same module/version.

I have implemented all of this in https://www.npmjs.com/package/live-perf and have seen huge speedups. The only downside is the resolver needs to be cached, because the dynamic resolving has an overhead as every require statement needs to find its closest package.json file with syscalls.

I think these changes to the Node resolver would make the ultimate package manager.

The end goal should be not to ever worry about whats in node_modules, and be able to rely on auto-installing ala https://github.com/siddharthkp/auto-install.

Imagine being able to write code without ever having to run npm install or rm -rf node_modules.

@halhenke
Contributor
halhenke commented Sep 8, 2016

Its been a while since i looked at the internals of npm (did a tiny PR) or pnpm (meant to do a fairly big feature but....never did 😞 ) but i think its fair to point out that npm does deal with a lot more edge cases/supported methods of installation (git repos etc) than these other clients.

OTOH i do think there is a fundamental flaw (at least from a performance perspective) in the way that npm3 tries to build a model of the ideal tree before going about the business of pulling down packages...again the details escape me but - as mentioned before the npm 3 package resolution/install algorithm is very complicated and quite tightly coupled to a lot of other parts. My knowledge is fuzzy now and quite possibly out of date but i do think a more or less ground up re-implementation/rethink of how packages are resolved/fetched would be necessary to see notable performance gains. Just my thoughts/opinions.

@pannous
pannous commented Dec 29, 2016 edited

All the euphemisms and kindness aside:

time npm -g ls
real	0m45.109s

Labels: big-bug good
Closed: why?

These suggestions alleviated the situation dramatically:

npm set progress=false
mv /usr/local/lib/node_modules /usr/local/lib/node_modules.TOO_MANY
cp -r /usr/local/lib/node_modules.TOO_MANY/npm /usr/local/lib/node_modules/

Lesson learned: use global modules very sparingly

@iarna
Member
iarna commented Jan 2, 2017 edited

@pannous npm ls -g is going to take as long as it takes to list all your global modules in logical tree form, but lots of global modules shouldn't have any impact on any other operation. If you are seeing an impact from having a lot of global modules on any operation other than npm ls -g then please open an issue!

PS disabling your progress bar really REALLY shouldn't have any measurable impact on run time. (I've not seen a benchmark yet where it did.) The progress bar is extremely low impact these days.

@MattFoley

One thing I've noticed is that performance seems to drop considerably when your package.json contains urls to specific commits, rather than npm packages.

@vvo
vvo commented Feb 14, 2017

@MattFoley I believe this is something to expect, when specifying a git commit you then have to fire up git command, clone, checkout or something like that. Way slower than downloading a tgz

@MattFoley

@vvo I think you're right, but maybe there are some improvements that could be made? Maybe a shallow checkout? It may not even be the git checkout that's causing the hang, not sure.

Seems like it hangs on fetchMetadata, although I don't know enough about npm to really say.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment