npm@3 wants to be faster #8826

iarna · Jul 4, 2015

npm@3 is sometimes slower than npm@2, though this is highly tree dependent. It is doing more, but all the same, folks'd like it to be as fast as it can be. Profiling would be grand. ;) This ticket exists as the tracker for npm@3 performance.

Put together a minimal benchmark to work against
- Should do a from-scratch install of something big
- Add a module to a large install
- Remove a module from a large install
- List a large install

Ideally I'd like this benchmark to be 2.x and 3.x compatible so we can directly compare different parts.

Yes, please improve the speed:

→ time npm -g ls --depth=0
/Users/okketimm/.nvm/versions/node/v0.12.7/lib
├── bower@1.4.1
├── ember-cli@1.13.1
├── eslint@0.24.0
├── npm@3.1.1
└── phantomjs@1.9.17

real 0m14.436s
user 0m14.377s
sys  0m0.616s

→ npm version
{ npm: '3.1.1',
  http_parser: '2.3',
  modules: '14',
  node: '0.12.7',
  openssl: '1.0.1p',
  uv: '1.6.1',
  v8: '3.28.71.19',
  zlib: '1.2.8' }

versus

→ time npm -g ls --depth=0
/Users/okketimm/.nvm/versions/node/v0.12.7/lib
├── bower@1.4.1
├── ember-cli@1.13.1
├── eslint@0.24.0
├── npm@2.11.3
└── phantomjs@1.9.17

real 0m0.570s
user 0m0.531s
sys  0m0.069s

→ npm version
{ npm: '2.13.0',
  http_parser: '2.3',
  modules: '14',
  node: '0.12.7',
  openssl: '1.0.1p',
  uv: '1.6.1',
  v8: '3.28.71.19',
  zlib: '1.2.8' }

okket · Jul 10, 2015

Yes, please improve the speed:

→ time npm -g ls --depth=0
/Users/okketimm/.nvm/versions/node/v0.12.7/lib
├── bower@1.4.1
├── ember-cli@1.13.1
├── eslint@0.24.0
├── npm@3.1.1
└── phantomjs@1.9.17

real 0m14.436s
user 0m14.377s
sys  0m0.616s

→ npm version
{ npm: '3.1.1',
  http_parser: '2.3',
  modules: '14',
  node: '0.12.7',
  openssl: '1.0.1p',
  uv: '1.6.1',
  v8: '3.28.71.19',
  zlib: '1.2.8' }

versus

→ time npm -g ls --depth=0
/Users/okketimm/.nvm/versions/node/v0.12.7/lib
├── bower@1.4.1
├── ember-cli@1.13.1
├── eslint@0.24.0
├── npm@2.11.3
└── phantomjs@1.9.17

real 0m0.570s
user 0m0.531s
sys  0m0.069s

→ npm version
{ npm: '2.13.0',
  http_parser: '2.3',
  modules: '14',
  node: '0.12.7',
  openssl: '1.0.1p',
  uv: '1.6.1',
  v8: '3.28.71.19',
  zlib: '1.2.8' }

I ran into a huge slowdown the first time doing a global install after upgrading from 2 to 3. with loglevel=silly on, it spent over a minute on
npm sill install cloneCurrentTreeToIdealTree loadIdealTree

npm -v = 3.1.0
node -v = 0.10.36
win8.1 / x64

suarasaur · Jul 21, 2015

I ran into a huge slowdown the first time doing a global install after upgrading from 2 to 3. with loglevel=silly on, it spent over a minute on
npm sill install cloneCurrentTreeToIdealTree loadIdealTree

npm -v = 3.1.0
node -v = 0.10.36
win8.1 / x64

I just wanted to add that I've noticed npm3 is considerably slower. Running npm install in a project where everything is already installed still takes a considerably long time. The expectation would be that npm install finishes super fast because there's nothing to install, nothing to do.

trusktr · Jul 29, 2015

I just wanted to add that I've noticed npm3 is considerably slower. Running npm install in a project where everything is already installed still takes a considerably long time. The expectation would be that npm install finishes super fast because there's nothing to install, nothing to do.

@trusktr This is one of the biggest differences between npm@2 and npm@3 – in a lot of ways, the mostly flat tree is a side show, and this dependency tree realization process is the primary change to the installer. Before the installer can decide that there's nothing for it to do, it has to both realize the current dependency tree, and construct an ideal dependency tree, and compare the two to see if there's anything that needs to be done. npm@2 didn't do this, which is why running npm install would never pick up any changes below the top-level dependencies. The current installer is a lot safer, but it needs to do a bunch more work to do what it does.

That said, this issue is an acknowledgment that making installs fast is an important feature in its own right, and that we aren't there yet.

othiym23 · Jul 29, 2015

@trusktr This is one of the biggest differences between npm@2 and npm@3 – in a lot of ways, the mostly flat tree is a side show, and this dependency tree realization process is the primary change to the installer. Before the installer can decide that there's nothing for it to do, it has to both realize the current dependency tree, and construct an ideal dependency tree, and compare the two to see if there's anything that needs to be done. npm@2 didn't do this, which is why running npm install would never pick up any changes below the top-level dependencies. The current installer is a lot safer, but it needs to do a bunch more work to do what it does.

That said, this issue is an acknowledgment that making installs fast is an important feature in its own right, and that we aren't there yet.

But I know it's getting there and is about to be super awesome. I can't wait for it to be the new default.

By the way, JSPM has flat dependency tree management, and it's fast! Could there be ideas to borrow from it's open source goodness?

trusktr · Jul 29, 2015

But I know it's getting there and is about to be super awesome. I can't wait for it to be the new default.

By the way, JSPM has flat dependency tree management, and it's fast! Could there be ideas to borrow from it's open source goodness?

I'm seeing quite an extreme difference, here's a sample dependency list that reproduces.

  "dependencies": {
    "accounting": "^0.4.1",
    "ampersand-app": "^1.0.4",
    "ampersand-collection": "^1.4.5",
    "ampersand-model": "^5.0.3",
    "ampersand-rest-collection": "^4.0.0",
    "ampersand-router": "^3.0.2",
    "autoprefixer-core": "^5.2.1",
    "babel": "^5.6.23",
    "babel-core": "^5.7.4",
    "babel-loader": "^5.3.2",
    "babelify": "^6.1.3",
    "blue-tape": "^0.1.10",
    "browserify": "^11.0.0",
    "classnames": "^2.1.3",
    "css-loader": "^0.15.5",
    "eslint": "^0.24.1",
    "file-loader": "^0.8.4",
    "float": "^1.0.2",
    "hjs-webpack": "^2.10.0",
    "json-loader": "^0.5.2",
    "local-links": "^1.4.0",
    "lodash.has": "^3.2.1",
    "lodash.isundefined": "^3.0.1",
    "lodash.random": "^3.0.1",
    "lodash.uniqueid": "^3.0.0",
    "node-libs-browser": "^0.5.2",
    "node-sass": "^3.2.0",
    "normalize.scss": "^0.1.0",
    "phantomjs": "^1.9.17",
    "pluralize": "^1.1.2",
    "postcss-loader": "^0.5.1",
    "react": "^0.13.3",
    "react-hot-loader": "^1.2.8",
    "sass-loader": "^1.0.2",
    "style-loader": "^0.12.3",
    "stylus-loader": "^1.2.1",
    "tap-spec": "^4.0.2",
    "tape": "^4.0.1",
    "tape-catch": "^1.0.4",
    "tape-run": "^1.0.0",
    "url-loader": "^0.5.6",
    "validator": "^3.41.3",
    "webpack": "^1.10.1",
    "webpack-dev-server": "^1.10.1",
    "xhr": "^2.0.3"
  }

> PATH=/opt/npm/bin:$PATH time sh -c 'npm ls 2> /dev/null > /dev/null'; npm version
       20.52 real        20.52 user         0.55 sys
{ thing: '1.0.0',
  npm: '2.11.2',
  http_parser: '2.3',
  modules: '14',
  node: '0.12.6',
  openssl: '1.0.1o',
  uv: '1.6.1',
  v8: '3.28.71.19',
  zlib: '1.2.8' }

> time sh -c 'npm ls 2> /dev/null > /dev/null'; npm version

real    0m3.198s
user    0m3.169s
sys 0m0.541s
{ thing: '1.0.0',
  npm: '2.11.2',
  http_parser: '2.3',
  modules: '14',
  node: '0.12.6',
  openssl: '1.0.1o',
  uv: '1.6.1',
  v8: '3.28.71.19',
  zlib: '1.2.8' }

I wonder if it might be feasible to cache the dependency resolution based on a hash of package.json or the declared dependencies?

glenjamin · Jul 29, 2015

I'm seeing quite an extreme difference, here's a sample dependency list that reproduces.

  "dependencies": {
    "accounting": "^0.4.1",
    "ampersand-app": "^1.0.4",
    "ampersand-collection": "^1.4.5",
    "ampersand-model": "^5.0.3",
    "ampersand-rest-collection": "^4.0.0",
    "ampersand-router": "^3.0.2",
    "autoprefixer-core": "^5.2.1",
    "babel": "^5.6.23",
    "babel-core": "^5.7.4",
    "babel-loader": "^5.3.2",
    "babelify": "^6.1.3",
    "blue-tape": "^0.1.10",
    "browserify": "^11.0.0",
    "classnames": "^2.1.3",
    "css-loader": "^0.15.5",
    "eslint": "^0.24.1",
    "file-loader": "^0.8.4",
    "float": "^1.0.2",
    "hjs-webpack": "^2.10.0",
    "json-loader": "^0.5.2",
    "local-links": "^1.4.0",
    "lodash.has": "^3.2.1",
    "lodash.isundefined": "^3.0.1",
    "lodash.random": "^3.0.1",
    "lodash.uniqueid": "^3.0.0",
    "node-libs-browser": "^0.5.2",
    "node-sass": "^3.2.0",
    "normalize.scss": "^0.1.0",
    "phantomjs": "^1.9.17",
    "pluralize": "^1.1.2",
    "postcss-loader": "^0.5.1",
    "react": "^0.13.3",
    "react-hot-loader": "^1.2.8",
    "sass-loader": "^1.0.2",
    "style-loader": "^0.12.3",
    "stylus-loader": "^1.2.1",
    "tap-spec": "^4.0.2",
    "tape": "^4.0.1",
    "tape-catch": "^1.0.4",
    "tape-run": "^1.0.0",
    "url-loader": "^0.5.6",
    "validator": "^3.41.3",
    "webpack": "^1.10.1",
    "webpack-dev-server": "^1.10.1",
    "xhr": "^2.0.3"
  }

> PATH=/opt/npm/bin:$PATH time sh -c 'npm ls 2> /dev/null > /dev/null'; npm version
       20.52 real        20.52 user         0.55 sys
{ thing: '1.0.0',
  npm: '2.11.2',
  http_parser: '2.3',
  modules: '14',
  node: '0.12.6',
  openssl: '1.0.1o',
  uv: '1.6.1',
  v8: '3.28.71.19',
  zlib: '1.2.8' }

> time sh -c 'npm ls 2> /dev/null > /dev/null'; npm version

real    0m3.198s
user    0m3.169s
sys 0m0.541s
{ thing: '1.0.0',
  npm: '2.11.2',
  http_parser: '2.3',
  modules: '14',
  node: '0.12.6',
  openssl: '1.0.1o',
  uv: '1.6.1',
  v8: '3.28.71.19',
  zlib: '1.2.8' }

I wonder if it might be feasible to cache the dependency resolution based on a hash of package.json or the declared dependencies?

@iarna "Profiling would be grand": https://gist.github.com/mblakele/4f1eefd6b4ad5543cfb6

This is something of a noop, running npm install with 3.2.2 when there's no work to be done. It takes about 6-7x longer than npm 2.14.3 does: 30-40 sec vs 5 sec.

As predicted by @othiym23 much of the time seems to be spent realizing packages. I notice fs.stat is prominent, and that's with everything on an SSD. Might be worth checking to make sure that the same packages aren't checked multiple times, though.

mblakele · Sep 9, 2015

@iarna "Profiling would be grand": https://gist.github.com/mblakele/4f1eefd6b4ad5543cfb6

This is something of a noop, running npm install with 3.2.2 when there's no work to be done. It takes about 6-7x longer than npm 2.14.3 does: 30-40 sec vs 5 sec.

As predicted by @othiym23 much of the time seems to be spent realizing packages. I notice fs.stat is prominent, and that's with everything on an SSD. Might be worth checking to make sure that the same packages aren't checked multiple times, though.

Another extreme case, installing eslint is 5 times slower — very very painful :(

time npm install eslint

# v3.3.3
real  0m48.122s
user  0m8.783s
sys   0m2.051s

# v2.4.15
real  0m10.561s
user  0m7.597s
sys   0m2.538s

mourner · Sep 18, 2015

Another extreme case, installing eslint is 5 times slower — very very painful :(

time npm install eslint

# v3.3.3
real  0m48.122s
user  0m8.783s
sys   0m2.051s

# v2.4.15
real  0m10.561s
user  0m7.597s
sys   0m2.538s

I would go as far as to say most of the installation tasks performed by npm 3 are obviously slower than npm 2. This is a regression! Not progression at all. The title of this issue needs to be changed to npm@v3 needs to be faster because v3 cannot be released with these perf problems.

badsyntax · Sep 18, 2015

I would go as far as to say most of the installation tasks performed by npm 3 are obviously slower than npm 2. This is a regression! Not progression at all. The title of this issue needs to be changed to npm@v3 needs to be faster because v3 cannot be released with these perf problems.

Now that npm@3 is in wider release, @iarna is going to spending most of her time fixing critical issues in it as they're found, and analyzing and improving npm's performance. One of the primary purposes of npm 3 was to make the install process more robust and correct, and that means that in many (but not all) cases it's doing substantially more work than older versions were. Improving the performance of the CLI without regressing on robustness is the CLI team's #1 priority right now.

othiym23 · Sep 18, 2015

Now that npm@3 is in wider release, @iarna is going to spending most of her time fixing critical issues in it as they're found, and analyzing and improving npm's performance. One of the primary purposes of npm 3 was to make the install process more robust and correct, and that means that in many (but not all) cases it's doing substantially more work than older versions were. Improving the performance of the CLI without regressing on robustness is the CLI team's #1 priority right now.

If you want another data point. react-native is taking 1m53s on npm 3.3.3 and only 40s on npm 2.13.2, a 2.8x increase. The good news is that the number of files went from 25887 to 13261, only half of files :)

Repro steps:

git clone https://github.com/facebook/react-native.git
cd react-native

rm npm-shrinkwrap.json
rm -Rf node_modules
npm install -g npm
time npm install
# real  1m46.743s
# user  0m51.520s
# sys   0m12.608s
find node_modules | wc -l
# 25887

rm npm-shrinkwrap.json
rm -Rf node_modules
npm install -g npm@2.13.2
time npm install 
# real  0m40.008s
# user  0m38.214s
# sys   0m19.352s
find node_modules | wc -l
# 13261

vjeux · Sep 20, 2015

If you want another data point. react-native is taking 1m53s on npm 3.3.3 and only 40s on npm 2.13.2, a 2.8x increase. The good news is that the number of files went from 25887 to 13261, only half of files :)

Repro steps:

git clone https://github.com/facebook/react-native.git
cd react-native

rm npm-shrinkwrap.json
rm -Rf node_modules
npm install -g npm
time npm install
# real  1m46.743s
# user  0m51.520s
# sys   0m12.608s
find node_modules | wc -l
# 25887

rm npm-shrinkwrap.json
rm -Rf node_modules
npm install -g npm@2.13.2
time npm install 
# real  0m40.008s
# user  0m38.214s
# sys   0m19.352s
find node_modules | wc -l
# 13261

Is there any way to get more useful progress reporting? Increasing verbosity is great but it doesn't help in the common case where you basically want an indication of how long the install is going to take. Even a very rough estimate would be better than nothing.

timdp · Sep 22, 2015

Is there any way to get more useful progress reporting? Increasing verbosity is great but it doesn't help in the common case where you basically want an indication of how long the install is going to take. Even a very rough estimate would be better than nothing.

We're seeing a huge regression in npm@3 in polymer-starter-kit.

The following package.json takes ~2 minutes to install on npm@2, but ~10-14 minutes on npm@3.

Node.js 4.1.1. OS X 10.10.

Here's a npm-shrinkwrap.json of the dependencies for reproducibility.

sindresorhus · Sep 23, 2015

We're seeing a huge regression in npm@3 in polymer-starter-kit.

The following package.json takes ~2 minutes to install on npm@2, but ~10-14 minutes on npm@3.

Node.js 4.1.1. OS X 10.10.

Here's a npm-shrinkwrap.json of the dependencies for reproducibility.

Installs went from ~4mins -> ~12mins when switching from 2.2.18 to 3.3.3

Would recommend anything time sensitive (deployments) stay on 2.x

contra · Sep 24, 2015

Installs went from ~4mins -> ~12mins when switching from 2.2.18 to 3.3.3

Would recommend anything time sensitive (deployments) stay on 2.x

@iarna any progress updates on this? The original issue description suggests it is "sometimes slower" when it's in fact more like "always 3 to 7 times slower", which is pretty critical and prevents a lot of people from upgrading.

mourner · Sep 24, 2015

@iarna any progress updates on this? The original issue description suggests it is "sometimes slower" when it's in fact more like "always 3 to 7 times slower", which is pretty critical and prevents a lot of people from upgrading.

@mourner Well, contrary to popular belief, it's actually 5-10 times faster in some scenarios too, but apparently none that you're using. =/

But everyone here will be happy to learn that we identified the source of almost all of the slow down and excised it this week. Starting with npm@3.3.6 we should be seeing substantial performance improvements any time your node_modules has a lot of modules in it.

iarna · Sep 30, 2015

@mourner Well, contrary to popular belief, it's actually 5-10 times faster in some scenarios too, but apparently none that you're using. =/

But everyone here will be happy to learn that we identified the source of almost all of the slow down and excised it this week. Starting with npm@3.3.6 we should be seeing substantial performance improvements any time your node_modules has a lot of modules in it.

Probably too late for that. What would be best is making 4.X off 2.X and
start over. Cut your losses or be phased out by newer coolness. Speed is
what matters.

On Wed, Dec 16, 2015, 12:16 Luis Rogelio Hernández López <
notifications@github.com> wrote:

I think thy should let us choice if we want or not know what's going under
the hood.
In my case, I don't really care about NPM's internal operations.

—
Reply to this email directly or view it on GitHub
#8826 (comment).

VR,

Sent from my mobile device

Nicholas McCready
nmccready@gmail.com

nmccready · Dec 16, 2015

Probably too late for that. What would be best is making 4.X off 2.X and
start over. Cut your losses or be phased out by newer coolness. Speed is
what matters.

On Wed, Dec 16, 2015, 12:16 Luis Rogelio Hernández López <
notifications@github.com> wrote:

I think thy should let us choice if we want or not know what's going under
the hood.
In my case, I don't really care about NPM's internal operations.

—
Reply to this email directly or view it on GitHub
#8826 (comment).

VR,

Sent from my mobile device

Nicholas McCready
nmccready@gmail.com

rather be focusing on compatibility issues for the next year at least.

Right now,the CLI team is working on getting its test suite passing consistently under Travis on all versions of Node it supports. Once that's done, we're doing the same, but for Windows as well as UNIX, which will involve setting up Windows CI. This is long-overdue payment of some significant technical debt. After that work is complete, our two next priorities are fixing the outstanding bugs in the installer (there are a fair few we already know about), and improving npm@3's performance. Software takes as long as it takes, but I'm thinking it will take a month or two to start working on performance, not a year.

The basic gist of it is that it's 3-5 times slower because it "makes it much easier to understand what npm is doing".

That's a fairly ungenerous reading of what @iarna and I have said. npm@3's installer has a goal to be robust, and part of that is making it understandable so we can troubleshoot and debug it when it's behaving erratically, and so developers like you, who lack our intimate knowledge of the installer's design and implementation, can have a better understanding of why npm is doing what it's doing. Previous versions of the installer were exceedingly difficult to understand when e.g. the bundle dependency algorithm was doing unexpected things.

Maintainers say there are no obvious ways to bring NPM3 on par with NPM2 performance

This simply isn't true. It will require the involvement of more than the CLI team at npm, and it's going to take some time, but I'm fully confident that we'll return npm's performance to at least on par with npm@2.

othiym23 · Dec 16, 2015

rather be focusing on compatibility issues for the next year at least.

Right now,the CLI team is working on getting its test suite passing consistently under Travis on all versions of Node it supports. Once that's done, we're doing the same, but for Windows as well as UNIX, which will involve setting up Windows CI. This is long-overdue payment of some significant technical debt. After that work is complete, our two next priorities are fixing the outstanding bugs in the installer (there are a fair few we already know about), and improving npm@3's performance. Software takes as long as it takes, but I'm thinking it will take a month or two to start working on performance, not a year.

The basic gist of it is that it's 3-5 times slower because it "makes it much easier to understand what npm is doing".

That's a fairly ungenerous reading of what @iarna and I have said. npm@3's installer has a goal to be robust, and part of that is making it understandable so we can troubleshoot and debug it when it's behaving erratically, and so developers like you, who lack our intimate knowledge of the installer's design and implementation, can have a better understanding of why npm is doing what it's doing. Previous versions of the installer were exceedingly difficult to understand when e.g. the bundle dependency algorithm was doing unexpected things.

Maintainers say there are no obvious ways to bring NPM3 on par with NPM2 performance

This simply isn't true. It will require the involvement of more than the CLI team at npm, and it's going to take some time, but I'm fully confident that we'll return npm's performance to at least on par with npm@2.

In my case, I don't really care about NPM's internal operations.

You don't care until it does something unexpected or wrong, and then you (in general, maybe not you in particular) care a great deal, because until somebody understands what's happening, you're blocked (potentially along with the rest of your team). 90+% of the time, npm-debug.log is the only tool available to the CLI team to understand buggy behavior, making it all but essential that it capture as much information as possible about what's going on with your installs.

othiym23 · Dec 16, 2015

In my case, I don't really care about NPM's internal operations.

You don't care until it does something unexpected or wrong, and then you (in general, maybe not you in particular) care a great deal, because until somebody understands what's happening, you're blocked (potentially along with the rest of your team). 90+% of the time, npm-debug.log is the only tool available to the CLI team to understand buggy behavior, making it all but essential that it capture as much information as possible about what's going on with your installs.

@othiym23 that's why I said this: "They should let us choice if we want or not know what's going under the hood."
If a problem occurs I will want to know what went wrong, of course.

Kurtz1993 · Dec 16, 2015

@othiym23 that's why I said this: "They should let us choice if we want or not know what's going under the hood."
If a problem occurs I will want to know what went wrong, of course.

@othiym23 is there any details/reasons behind chosen priorities? I mean why tech debt have been considered more important than community's problems. I don't know full picture, so I will be happy to know

iamstarkov · Dec 16, 2015

@othiym23 is there any details/reasons behind chosen priorities? I mean why tech debt have been considered more important than community's problems. I don't know full picture, so I will be happy to know

is there any details/reasons behind chosen priorities?

The npm CLI team includes one manager (me) and two full-time engineers, and has a very heavy workload. We don't have dedicated QA resources, so our only indicators of the quality of the product we're shipping are the number of inbound issues we see and our test suite. If we don't trust that the test suite is telling us that our code base is in good shape, we're in trouble. Right now, the team has insufficient faith in the test suite. So we need to fix that!

Also, it tremendously simplifies the process of landing contributions from the community if we can look at the Travis build and know whether the new code causes any problems without pulling down changes and running the tests ourselves. And it's an important part of our commitment to Windows users to have continuous integration testing things on their platform.

Finally, the internals of npm are sufficiently difficult to understand that most users of npm never make the effort, and when they do, very few of them master it enough to make meaningful contributions to the core. The architecture is complex and subtle, and part of the reason npm@3 exists is to make it more comprehensible, so that the CLI team is no longer so much of a gatekeeper for finding and fixing issues in the installer.

We've put off paying down this debt for way too long, to the point that it's significantly impairing the ability of the project to add new features (and increase the size of the contributor base). It's not that this technical debt is more important than the community's problems, it's that it is a community problem. The community problem, in my estimation.

othiym23 · Dec 16, 2015

is there any details/reasons behind chosen priorities?

The npm CLI team includes one manager (me) and two full-time engineers, and has a very heavy workload. We don't have dedicated QA resources, so our only indicators of the quality of the product we're shipping are the number of inbound issues we see and our test suite. If we don't trust that the test suite is telling us that our code base is in good shape, we're in trouble. Right now, the team has insufficient faith in the test suite. So we need to fix that!

Also, it tremendously simplifies the process of landing contributions from the community if we can look at the Travis build and know whether the new code causes any problems without pulling down changes and running the tests ourselves. And it's an important part of our commitment to Windows users to have continuous integration testing things on their platform.

Finally, the internals of npm are sufficiently difficult to understand that most users of npm never make the effort, and when they do, very few of them master it enough to make meaningful contributions to the core. The architecture is complex and subtle, and part of the reason npm@3 exists is to make it more comprehensible, so that the CLI team is no longer so much of a gatekeeper for finding and fixing issues in the installer.

We've put off paying down this debt for way too long, to the point that it's significantly impairing the ability of the project to add new features (and increase the size of the contributor base). It's not that this technical debt is more important than the community's problems, it's that it is a community problem. The community problem, in my estimation.

@othiym23 thank you

iamstarkov · Dec 16, 2015

@othiym23 thank you

You don't care until it does something unexpected or wrong, and then you (in general, maybe not you in particular) care a great deal, because until somebody understands what's happening, you're blocked (potentially along with the rest of your team). 90+% of the time, npm-debug.log is the only tool available to the CLI team to understand buggy behavior, making it all but essential that it capture as much information as possible about what's going on with your installs.

It sounds like this should be off by default, then. If a command fails, the user can re-run it with logging enabled, thus solving both the performance problem and the debugging problem.

Zarel · Jan 26, 2016

You don't care until it does something unexpected or wrong, and then you (in general, maybe not you in particular) care a great deal, because until somebody understands what's happening, you're blocked (potentially along with the rest of your team). 90+% of the time, npm-debug.log is the only tool available to the CLI team to understand buggy behavior, making it all but essential that it capture as much information as possible about what's going on with your installs.

It sounds like this should be off by default, then. If a command fails, the user can re-run it with logging enabled, thus solving both the performance problem and the debugging problem.

I have experienced the same problem for the last two days trying to npm install dependencies for meanjs but I opted to use Windows command processor. It did it in record time obviously the progress bar got halfway..hope we can find a solution. issue #11283 and issue #8826

josephgodwinkimani · Jan 27, 2016

I have experienced the same problem for the last two days trying to npm install dependencies for meanjs but I opted to use Windows command processor. It did it in record time obviously the progress bar got halfway..hope we can find a solution. issue #11283 and issue #8826

From all those time reports it's quite obvious that while user and system times are the same, real time is considerably bigger. It means that npm3 is either waiting on I/O or is not utilizing all CPU cores. It's quite ironic, because the selling point of node was "it's very fast because it's not blocked on I/O, and it doesn't need threads".

So JavaScript is not as efficient for scripting. Even more ironic.

Vanuan · Jan 28, 2016

From all those time reports it's quite obvious that while user and system times are the same, real time is considerably bigger. It means that npm3 is either waiting on I/O or is not utilizing all CPU cores. It's quite ironic, because the selling point of node was "it's very fast because it's not blocked on I/O, and it doesn't need threads".

So JavaScript is not as efficient for scripting. Even more ironic.

Many of the people on this thread do not seem to understand that npm 3 was a major rearchitecture. Comments about flipping a switch to jump between npm 2 and npm 3 installs are naive (if you want npm 2, install npm 2...). npm is the most platform agnostic package manager in existence and if you think that npm 3 was a regression try using each version on Windows for a few months. npm 2 is a mess on Windows (long file paths...), installs are often insurmountable and require bypassing packages. npm 3 has brought incredible stability. On npm 2, removal of node_modules was a daily occurrence and now I can get faster and more stable results with normal incremental npm install. Bottom line, I spend orders of magnitude less hours hacking npm 3 than npm 2 because it actually works. If you are doing constant npm installs from scratch than maybe you should optimize your build process instead of trolling the awesomely helpful npm team. If you can rewrite the logic for speed and maintain the stability that npm 3 brings than submit your pull request.

cchamberlain · Jan 28, 2016

Many of the people on this thread do not seem to understand that npm 3 was a major rearchitecture. Comments about flipping a switch to jump between npm 2 and npm 3 installs are naive (if you want npm 2, install npm 2...). npm is the most platform agnostic package manager in existence and if you think that npm 3 was a regression try using each version on Windows for a few months. npm 2 is a mess on Windows (long file paths...), installs are often insurmountable and require bypassing packages. npm 3 has brought incredible stability. On npm 2, removal of node_modules was a daily occurrence and now I can get faster and more stable results with normal incremental npm install. Bottom line, I spend orders of magnitude less hours hacking npm 3 than npm 2 because it actually works. If you are doing constant npm installs from scratch than maybe you should optimize your build process instead of trolling the awesomely helpful npm team. If you can rewrite the logic for speed and maintain the stability that npm 3 brings than submit your pull request.

@cchamberlain This is just feedback. Build processes have become a pain in the ass lately because of this issue ; because yes, it is an issue, with speed.

I'm just reporting the great deal of time an install now takes. What is it these days that you can't provide feedback to an open-source project without people saying you're trolling the contrib team.

Am I ungrateful by saying that a product got worse for the use I make of it?

vroudge · Jan 28, 2016

@cchamberlain This is just feedback. Build processes have become a pain in the ass lately because of this issue ; because yes, it is an issue, with speed.

I'm just reporting the great deal of time an install now takes. What is it these days that you can't provide feedback to an open-source project without people saying you're trolling the contrib team.

Am I ungrateful by saying that a product got worse for the use I make of it?

@vroudge - npm 3 is slower than npm 2. It is well documented and no secret. It's slower because it fixes many npm 2 issues. It will get faster but the use you make of it is a very small piece of the puzzle. Speeds great but it's much more important that it does everything it claims it can do without randomly bombing out. Chiming in that it's slower is not useful, if you prefer the npm 2 nested installs you should use that since they are still maintaining it.

cchamberlain · Jan 28, 2016

@vroudge - npm 3 is slower than npm 2. It is well documented and no secret. It's slower because it fixes many npm 2 issues. It will get faster but the use you make of it is a very small piece of the puzzle. Speeds great but it's much more important that it does everything it claims it can do without randomly bombing out. Chiming in that it's slower is not useful, if you prefer the npm 2 nested installs you should use that since they are still maintaining it.

@cchamberlain Does not change the fact that it is an issue and is seen as an issue by the maintainers of the project.

The new modules flat structure is really wonderful and I love it. And npm works wonders, really. It's just really slow, that's it. When you see things like the fact that the progress bar makes it reaaally slower, you just begin to wonder if it could not be made faster, and that's it. I don't chime in just saying that it's slower just to QQ at the fact it's slower, there are many things that make it slower, not just the new tree. I just think it should be investigated. Not that I have the knowledge to do it.

Lets stop this here and stop polluting the issues.

vroudge · Jan 28, 2016

@cchamberlain Does not change the fact that it is an issue and is seen as an issue by the maintainers of the project.

The new modules flat structure is really wonderful and I love it. And npm works wonders, really. It's just really slow, that's it. When you see things like the fact that the progress bar makes it reaaally slower, you just begin to wonder if it could not be made faster, and that's it. I don't chime in just saying that it's slower just to QQ at the fact it's slower, there are many things that make it slower, not just the new tree. I just think it should be investigated. Not that I have the knowledge to do it.

Lets stop this here and stop polluting the issues.

@cchamberlain oh, we've got who to blame - windows guys. Why won't you just start using more developer-friendly operating systems

Vanuan · Jan 28, 2016

@cchamberlain oh, we've got who to blame - windows guys. Why won't you just start using more developer-friendly operating systems

@cchamberlain it fixes nothing, it brought new issues instead (some conceptual like the fact you can require a sub-dependency now without warning, some very important for an install tool like partial installs with no feedback), and fixed a non-issue for non-Windows developers. It could even have been fixed by a specific command as the too deep folders could still be removed programmatically.

naholyr · Jan 28, 2016

@cchamberlain it fixes nothing, it brought new issues instead (some conceptual like the fact you can require a sub-dependency now without warning, some very important for an install tool like partial installs with no feedback), and fixed a non-issue for non-Windows developers. It could even have been fixed by a specific command as the too deep folders could still be removed programmatically.

I am on Ubuntu and to install laravel-elixir@5 it took me 30+ mins with 8Mbps connection. It never took this long before when I was in v2. I am working through Docker.

shehi · Feb 28, 2016

I am on Ubuntu and to install laravel-elixir@5 it took me 30+ mins with 8Mbps connection. It never took this long before when I was in v2. I am working through Docker.

so it seems like the speed question is pretty well established at this point. Is there a clear path for continuing to move forward? Can we update the OP comment with links to subtasks so we can start knocking them out?

dsernst · Mar 2, 2016

so it seems like the speed question is pretty well established at this point. Is there a clear path for continuing to move forward? Can we update the OP comment with links to subtasks so we can start knocking them out?

Hi @othiym23. Thanks for all your work on npm in general and on npm@3 (making it more robust etc).

Back in December, you said you were hoping that some cycles would be spent on npm@3 performance within the following month or so. Are there any updates on that?

tfennelly · Mar 7, 2016

Hi @othiym23. Thanks for all your work on npm in general and on npm@3 (making it more robust etc).

Back in December, you said you were hoping that some cycles would be spent on npm@3 performance within the following month or so. Are there any updates on that?

This is why I typically avoid making date-based estimates. ;)

The CLI team has been making substantial progress in rewriting npm's test suite to make it less hairy, and to pass on Windows. The latter, especially, is an important quality milestone for the project, which is why the team is focused on test work for the time being.

After that, the team will be tackling the list of significant known issues. After that is when we've scheduled getting the whole team working on improving performance, which will most likely include some architectural work as well.

All that said, that's just what the CLI team itself is focusing on. The team continues to land community PRs as quickly as it can; several recent PRs have resulted in significant performance improvements in some cases.

Thanks to all for their contributions and patience.

othiym23 · Mar 7, 2016

This is why I typically avoid making date-based estimates. ;)

The CLI team has been making substantial progress in rewriting npm's test suite to make it less hairy, and to pass on Windows. The latter, especially, is an important quality milestone for the project, which is why the team is focused on test work for the time being.

After that, the team will be tackling the list of significant known issues. After that is when we've scheduled getting the whole team working on improving performance, which will most likely include some architectural work as well.

All that said, that's just what the CLI team itself is focusing on. The team continues to land community PRs as quickly as it can; several recent PRs have resulted in significant performance improvements in some cases.

Thanks to all for their contributions and patience.

@othiym23 several months passed. can you share some kind of progress on this issue?

iamstarkov · Sep 6, 2016

@othiym23 several months passed. can you share some kind of progress on this issue?

The CLI team has been making substantial progress in rewriting npm's test suite to make it less hairy, and to pass on Windows.

This is (mostly) done. There's some wiggliness on AppVeyor when running the CI suite, and it's required some TLC from me that I haven't had time to provide. The test suite, however, is in a much better place than it was in in March.

After that, the team will be tackling the list of significant known issues.

This work is still in progress (see the label, which should give you an idea of how many issues are in that bucket). It's finicky, demanding work, and the team remains small, so progress is moderate, but real.

We've also landed several patches, including the new version of gauges and are-we-there-yet in npmlog, with (almost exclusively positive) performance implications. I believe that overall the performance of the CLI is significantly better, with fewer pathological corner cases, than it was in March. That said, there's still a ways to go before the team can turn its energies towards making more wholesale improvements to the CLI's performance at a deeper, more architectural level. When we do start work on that, we'll say as much here.

othiym23 · Sep 7, 2016

The CLI team has been making substantial progress in rewriting npm's test suite to make it less hairy, and to pass on Windows.

This is (mostly) done. There's some wiggliness on AppVeyor when running the CI suite, and it's required some TLC from me that I haven't had time to provide. The test suite, however, is in a much better place than it was in in March.

After that, the team will be tackling the list of significant known issues.

This work is still in progress (see the label, which should give you an idea of how many issues are in that bucket). It's finicky, demanding work, and the team remains small, so progress is moderate, but real.

We've also landed several patches, including the new version of gauges and are-we-there-yet in npmlog, with (almost exclusively positive) performance implications. I believe that overall the performance of the CLI is significantly better, with fewer pathological corner cases, than it was in March. That said, there's still a ways to go before the team can turn its energies towards making more wholesale improvements to the CLI's performance at a deeper, more architectural level. When we do start work on that, we'll say as much here.

Gosh darn it, I keep closing this issue. Freudian close?

othiym23 · Sep 7, 2016

Gosh darn it, I keep closing this issue. Freudian close?

No, wait, it was closed already! ;_; so confused!

othiym23 · Sep 7, 2016

No, wait, it was closed already! ;_; so confused!

I believe that overall the performance of the CLI is significantly better, with fewer pathological corner cases, than it was in March.

There has been some progress, but unfortunately not nearly enough to consider this issue closed. NPM 3 still has consistently 2-3x slower install times than NPM 2, and 5-6x slower times than popular, actively developed alternatives like pnpm and ied (which also flatten the dependency tree) — you can confirm this on any popular Node package, and the alternatives (each having half a thousand commits and tens of contributors) wouldn't even exist in the first place if this wasn't a critical problem.

I'm thinking it will take a month or two to start working on performance, not a year.

It has been 9 months since the comment above...

mourner · Sep 7, 2016

I believe that overall the performance of the CLI is significantly better, with fewer pathological corner cases, than it was in March.

There has been some progress, but unfortunately not nearly enough to consider this issue closed. NPM 3 still has consistently 2-3x slower install times than NPM 2, and 5-6x slower times than popular, actively developed alternatives like pnpm and ied (which also flatten the dependency tree) — you can confirm this on any popular Node package, and the alternatives (each having half a thousand commits and tens of contributors) wouldn't even exist in the first place if this wasn't a critical problem.

I'm thinking it will take a month or two to start working on performance, not a year.

It has been 9 months since the comment above...

P.S. I highly appreciate the hard work you're doing and apologize for being annoying. Just trying to keep performance closer on your radar because it significantly affects my life as an open source maintainer.

mourner · Sep 7, 2016

P.S. I highly appreciate the hard work you're doing and apologize for being annoying. Just trying to keep performance closer on your radar because it significantly affects my life as an open source maintainer.

Is there an open issue to track progress with performance?

I recently tried out pnpm and it is incredibly fast. It makes Node fun again. I think the community should invest more in pnpm and ied, than expect npm to improve. I have a feeling the performance issues are too deep, and would require a huge amount of re-architecting.

Npm3 has done great work in pushing the idea of the deduped by default node_modules but the non-determinism seems hacky, when pnpm solves this perfectly.

The Node resolver is over-complicated. The pnpm model of having a central store with all modules installed as module-name@version is simple, elegant, and easy to understand.

What would be best is if when requiring a module, the resolver would read the closest package.json file, and find the best version to use from a central store.

If two require('foo') statements resolved to two different absolute paths, then a warning would be shown to help the user update package version ranges to prevent two modules being evaled. At present npm linking causing huge amounts of duplicate module evals. Most importantly this fixes performance issues with npm linking which involves evaling many copies of the same module/version.

I have implemented all of this in https://www.npmjs.com/package/live-perf and have seen huge speedups. The only downside is the resolver needs to be cached, because the dynamic resolving has an overhead as every require statement needs to find its closest package.json file with syscalls.

I think these changes to the Node resolver would make the ultimate package manager.

The end goal should be not to ever worry about whats in node_modules, and be able to rely on auto-installing ala https://github.com/siddharthkp/auto-install.

Imagine being able to write code without ever having to run npm install or rm -rf node_modules.

vjpr · Sep 7, 2016