This repository has been archived by the owner. It is now read-only.

npm@3 wants to be faster #8826

Closed
iarna opened this Issue Jul 4, 2015 · 115 comments

Comments

Projects
None yet
@iarna
Member

iarna commented Jul 4, 2015

npm@3 is sometimes slower than npm@2, though this is highly tree dependent. It is doing more, but all the same, folks'd like it to be as fast as it can be. Profiling would be grand. ;) This ticket exists as the tracker for npm@3 performance.

  • Put together a minimal benchmark to work against
    • Should do a from-scratch install of something big
    • Add a module to a large install
    • Remove a module from a large install
    • List a large install

Ideally I'd like this benchmark to be 2.x and 3.x compatible so we can directly compare different parts.

@iarna iarna added this to the 3.x milestone Jul 4, 2015

@okket

This comment has been minimized.

Show comment
Hide comment
@okket

okket Jul 10, 2015

Yes, please improve the speed:

time npm -g ls --depth=0
/Users/okketimm/.nvm/versions/node/v0.12.7/lib
├── bower@1.4.1
├── ember-cli@1.13.1
├── eslint@0.24.0
├── npm@3.1.1
└── phantomjs@1.9.17

real 0m14.436s
user 0m14.377s
sys  0m0.616s

→ npm version
{ npm: '3.1.1',
  http_parser: '2.3',
  modules: '14',
  node: '0.12.7',
  openssl: '1.0.1p',
  uv: '1.6.1',
  v8: '3.28.71.19',
  zlib: '1.2.8' }

versus

time npm -g ls --depth=0
/Users/okketimm/.nvm/versions/node/v0.12.7/lib
├── bower@1.4.1
├── ember-cli@1.13.1
├── eslint@0.24.0
├── npm@2.11.3
└── phantomjs@1.9.17

real 0m0.570s
user 0m0.531s
sys  0m0.069s

→ npm version
{ npm: '2.13.0',
  http_parser: '2.3',
  modules: '14',
  node: '0.12.7',
  openssl: '1.0.1p',
  uv: '1.6.1',
  v8: '3.28.71.19',
  zlib: '1.2.8' }

okket commented Jul 10, 2015

Yes, please improve the speed:

time npm -g ls --depth=0
/Users/okketimm/.nvm/versions/node/v0.12.7/lib
├── bower@1.4.1
├── ember-cli@1.13.1
├── eslint@0.24.0
├── npm@3.1.1
└── phantomjs@1.9.17

real 0m14.436s
user 0m14.377s
sys  0m0.616s

→ npm version
{ npm: '3.1.1',
  http_parser: '2.3',
  modules: '14',
  node: '0.12.7',
  openssl: '1.0.1p',
  uv: '1.6.1',
  v8: '3.28.71.19',
  zlib: '1.2.8' }

versus

time npm -g ls --depth=0
/Users/okketimm/.nvm/versions/node/v0.12.7/lib
├── bower@1.4.1
├── ember-cli@1.13.1
├── eslint@0.24.0
├── npm@2.11.3
└── phantomjs@1.9.17

real 0m0.570s
user 0m0.531s
sys  0m0.069s

→ npm version
{ npm: '2.13.0',
  http_parser: '2.3',
  modules: '14',
  node: '0.12.7',
  openssl: '1.0.1p',
  uv: '1.6.1',
  v8: '3.28.71.19',
  zlib: '1.2.8' }
@suarasaur

This comment has been minimized.

Show comment
Hide comment
@suarasaur

suarasaur Jul 21, 2015

I ran into a huge slowdown the first time doing a global install after upgrading from 2 to 3. with loglevel=silly on, it spent over a minute on
npm sill install cloneCurrentTreeToIdealTree loadIdealTree

  • npm -v = 3.1.0
  • node -v = 0.10.36
  • win8.1 / x64

suarasaur commented Jul 21, 2015

I ran into a huge slowdown the first time doing a global install after upgrading from 2 to 3. with loglevel=silly on, it spent over a minute on
npm sill install cloneCurrentTreeToIdealTree loadIdealTree

  • npm -v = 3.1.0
  • node -v = 0.10.36
  • win8.1 / x64
@trusktr

This comment has been minimized.

Show comment
Hide comment
@trusktr

trusktr Jul 29, 2015

I just wanted to add that I've noticed npm3 is considerably slower. Running npm install in a project where everything is already installed still takes a considerably long time. The expectation would be that npm install finishes super fast because there's nothing to install, nothing to do.

trusktr commented Jul 29, 2015

I just wanted to add that I've noticed npm3 is considerably slower. Running npm install in a project where everything is already installed still takes a considerably long time. The expectation would be that npm install finishes super fast because there's nothing to install, nothing to do.

@othiym23

This comment has been minimized.

Show comment
Hide comment
@othiym23

othiym23 Jul 29, 2015

Contributor

@trusktr This is one of the biggest differences between npm@2 and npm@3 – in a lot of ways, the mostly flat tree is a side show, and this dependency tree realization process is the primary change to the installer. Before the installer can decide that there's nothing for it to do, it has to both realize the current dependency tree, and construct an ideal dependency tree, and compare the two to see if there's anything that needs to be done. npm@2 didn't do this, which is why running npm install would never pick up any changes below the top-level dependencies. The current installer is a lot safer, but it needs to do a bunch more work to do what it does.

That said, this issue is an acknowledgment that making installs fast is an important feature in its own right, and that we aren't there yet.

Contributor

othiym23 commented Jul 29, 2015

@trusktr This is one of the biggest differences between npm@2 and npm@3 – in a lot of ways, the mostly flat tree is a side show, and this dependency tree realization process is the primary change to the installer. Before the installer can decide that there's nothing for it to do, it has to both realize the current dependency tree, and construct an ideal dependency tree, and compare the two to see if there's anything that needs to be done. npm@2 didn't do this, which is why running npm install would never pick up any changes below the top-level dependencies. The current installer is a lot safer, but it needs to do a bunch more work to do what it does.

That said, this issue is an acknowledgment that making installs fast is an important feature in its own right, and that we aren't there yet.

@trusktr

This comment has been minimized.

Show comment
Hide comment
@trusktr

trusktr Jul 29, 2015

But I know it's getting there and is about to be super awesome. I can't wait for it to be the new default.

By the way, JSPM has flat dependency tree management, and it's fast! Could there be ideas to borrow from it's open source goodness?

trusktr commented Jul 29, 2015

But I know it's getting there and is about to be super awesome. I can't wait for it to be the new default.

By the way, JSPM has flat dependency tree management, and it's fast! Could there be ideas to borrow from it's open source goodness?

@glenjamin

This comment has been minimized.

Show comment
Hide comment
@glenjamin

glenjamin Jul 29, 2015

Contributor

I'm seeing quite an extreme difference, here's a sample dependency list that reproduces.

  "dependencies": {
    "accounting": "^0.4.1",
    "ampersand-app": "^1.0.4",
    "ampersand-collection": "^1.4.5",
    "ampersand-model": "^5.0.3",
    "ampersand-rest-collection": "^4.0.0",
    "ampersand-router": "^3.0.2",
    "autoprefixer-core": "^5.2.1",
    "babel": "^5.6.23",
    "babel-core": "^5.7.4",
    "babel-loader": "^5.3.2",
    "babelify": "^6.1.3",
    "blue-tape": "^0.1.10",
    "browserify": "^11.0.0",
    "classnames": "^2.1.3",
    "css-loader": "^0.15.5",
    "eslint": "^0.24.1",
    "file-loader": "^0.8.4",
    "float": "^1.0.2",
    "hjs-webpack": "^2.10.0",
    "json-loader": "^0.5.2",
    "local-links": "^1.4.0",
    "lodash.has": "^3.2.1",
    "lodash.isundefined": "^3.0.1",
    "lodash.random": "^3.0.1",
    "lodash.uniqueid": "^3.0.0",
    "node-libs-browser": "^0.5.2",
    "node-sass": "^3.2.0",
    "normalize.scss": "^0.1.0",
    "phantomjs": "^1.9.17",
    "pluralize": "^1.1.2",
    "postcss-loader": "^0.5.1",
    "react": "^0.13.3",
    "react-hot-loader": "^1.2.8",
    "sass-loader": "^1.0.2",
    "style-loader": "^0.12.3",
    "stylus-loader": "^1.2.1",
    "tap-spec": "^4.0.2",
    "tape": "^4.0.1",
    "tape-catch": "^1.0.4",
    "tape-run": "^1.0.0",
    "url-loader": "^0.5.6",
    "validator": "^3.41.3",
    "webpack": "^1.10.1",
    "webpack-dev-server": "^1.10.1",
    "xhr": "^2.0.3"
  }
> PATH=/opt/npm/bin:$PATH time sh -c 'npm ls 2> /dev/null > /dev/null'; npm version
       20.52 real        20.52 user         0.55 sys
{ thing: '1.0.0',
  npm: '2.11.2',
  http_parser: '2.3',
  modules: '14',
  node: '0.12.6',
  openssl: '1.0.1o',
  uv: '1.6.1',
  v8: '3.28.71.19',
  zlib: '1.2.8' }
> time sh -c 'npm ls 2> /dev/null > /dev/null'; npm version

real    0m3.198s
user    0m3.169s
sys 0m0.541s
{ thing: '1.0.0',
  npm: '2.11.2',
  http_parser: '2.3',
  modules: '14',
  node: '0.12.6',
  openssl: '1.0.1o',
  uv: '1.6.1',
  v8: '3.28.71.19',
  zlib: '1.2.8' }

I wonder if it might be feasible to cache the dependency resolution based on a hash of package.json or the declared dependencies?

Contributor

glenjamin commented Jul 29, 2015

I'm seeing quite an extreme difference, here's a sample dependency list that reproduces.

  "dependencies": {
    "accounting": "^0.4.1",
    "ampersand-app": "^1.0.4",
    "ampersand-collection": "^1.4.5",
    "ampersand-model": "^5.0.3",
    "ampersand-rest-collection": "^4.0.0",
    "ampersand-router": "^3.0.2",
    "autoprefixer-core": "^5.2.1",
    "babel": "^5.6.23",
    "babel-core": "^5.7.4",
    "babel-loader": "^5.3.2",
    "babelify": "^6.1.3",
    "blue-tape": "^0.1.10",
    "browserify": "^11.0.0",
    "classnames": "^2.1.3",
    "css-loader": "^0.15.5",
    "eslint": "^0.24.1",
    "file-loader": "^0.8.4",
    "float": "^1.0.2",
    "hjs-webpack": "^2.10.0",
    "json-loader": "^0.5.2",
    "local-links": "^1.4.0",
    "lodash.has": "^3.2.1",
    "lodash.isundefined": "^3.0.1",
    "lodash.random": "^3.0.1",
    "lodash.uniqueid": "^3.0.0",
    "node-libs-browser": "^0.5.2",
    "node-sass": "^3.2.0",
    "normalize.scss": "^0.1.0",
    "phantomjs": "^1.9.17",
    "pluralize": "^1.1.2",
    "postcss-loader": "^0.5.1",
    "react": "^0.13.3",
    "react-hot-loader": "^1.2.8",
    "sass-loader": "^1.0.2",
    "style-loader": "^0.12.3",
    "stylus-loader": "^1.2.1",
    "tap-spec": "^4.0.2",
    "tape": "^4.0.1",
    "tape-catch": "^1.0.4",
    "tape-run": "^1.0.0",
    "url-loader": "^0.5.6",
    "validator": "^3.41.3",
    "webpack": "^1.10.1",
    "webpack-dev-server": "^1.10.1",
    "xhr": "^2.0.3"
  }
> PATH=/opt/npm/bin:$PATH time sh -c 'npm ls 2> /dev/null > /dev/null'; npm version
       20.52 real        20.52 user         0.55 sys
{ thing: '1.0.0',
  npm: '2.11.2',
  http_parser: '2.3',
  modules: '14',
  node: '0.12.6',
  openssl: '1.0.1o',
  uv: '1.6.1',
  v8: '3.28.71.19',
  zlib: '1.2.8' }
> time sh -c 'npm ls 2> /dev/null > /dev/null'; npm version

real    0m3.198s
user    0m3.169s
sys 0m0.541s
{ thing: '1.0.0',
  npm: '2.11.2',
  http_parser: '2.3',
  modules: '14',
  node: '0.12.6',
  openssl: '1.0.1o',
  uv: '1.6.1',
  v8: '3.28.71.19',
  zlib: '1.2.8' }

I wonder if it might be feasible to cache the dependency resolution based on a hash of package.json or the declared dependencies?

@iarna iarna added the blocker label Aug 11, 2015

@iarna iarna modified the milestones: 3.x-next-next, 3.x, 3.x-next Aug 17, 2015

@mblakele

This comment has been minimized.

Show comment
Hide comment
@mblakele

mblakele Sep 9, 2015

@iarna "Profiling would be grand": https://gist.github.com/mblakele/4f1eefd6b4ad5543cfb6

This is something of a noop, running npm install with 3.2.2 when there's no work to be done. It takes about 6-7x longer than npm 2.14.3 does: 30-40 sec vs 5 sec.

As predicted by @othiym23 much of the time seems to be spent realizing packages. I notice fs.stat is prominent, and that's with everything on an SSD. Might be worth checking to make sure that the same packages aren't checked multiple times, though.

mblakele commented Sep 9, 2015

@iarna "Profiling would be grand": https://gist.github.com/mblakele/4f1eefd6b4ad5543cfb6

This is something of a noop, running npm install with 3.2.2 when there's no work to be done. It takes about 6-7x longer than npm 2.14.3 does: 30-40 sec vs 5 sec.

As predicted by @othiym23 much of the time seems to be spent realizing packages. I notice fs.stat is prominent, and that's with everything on an SSD. Might be worth checking to make sure that the same packages aren't checked multiple times, though.

@iarna iarna modified the milestones: 3.x, 3.x-next Sep 11, 2015

@mourner

This comment has been minimized.

Show comment
Hide comment
@mourner

mourner Sep 18, 2015

Another extreme case, installing eslint is 5 times slower — very very painful :(

time npm install eslint

# v3.3.3
real  0m48.122s
user  0m8.783s
sys   0m2.051s

# v2.4.15
real  0m10.561s
user  0m7.597s
sys   0m2.538s

mourner commented Sep 18, 2015

Another extreme case, installing eslint is 5 times slower — very very painful :(

time npm install eslint

# v3.3.3
real  0m48.122s
user  0m8.783s
sys   0m2.051s

# v2.4.15
real  0m10.561s
user  0m7.597s
sys   0m2.538s

@othiym23 othiym23 added big-bug and removed blocker labels Sep 18, 2015

@badsyntax

This comment has been minimized.

Show comment
Hide comment
@badsyntax

badsyntax Sep 18, 2015

I would go as far as to say most of the installation tasks performed by npm 3 are obviously slower than npm 2. This is a regression! Not progression at all. The title of this issue needs to be changed to npm@v3 needs to be faster because v3 cannot be released with these perf problems.

badsyntax commented Sep 18, 2015

I would go as far as to say most of the installation tasks performed by npm 3 are obviously slower than npm 2. This is a regression! Not progression at all. The title of this issue needs to be changed to npm@v3 needs to be faster because v3 cannot be released with these perf problems.

@othiym23

This comment has been minimized.

Show comment
Hide comment
@othiym23

othiym23 Sep 18, 2015

Contributor

Now that npm@3 is in wider release, @iarna is going to spending most of her time fixing critical issues in it as they're found, and analyzing and improving npm's performance. One of the primary purposes of npm 3 was to make the install process more robust and correct, and that means that in many (but not all) cases it's doing substantially more work than older versions were. Improving the performance of the CLI without regressing on robustness is the CLI team's #1 priority right now.

Contributor

othiym23 commented Sep 18, 2015

Now that npm@3 is in wider release, @iarna is going to spending most of her time fixing critical issues in it as they're found, and analyzing and improving npm's performance. One of the primary purposes of npm 3 was to make the install process more robust and correct, and that means that in many (but not all) cases it's doing substantially more work than older versions were. Improving the performance of the CLI without regressing on robustness is the CLI team's #1 priority right now.

@vjeux

This comment has been minimized.

Show comment
Hide comment
@vjeux

vjeux Sep 20, 2015

If you want another data point. react-native is taking 1m53s on npm 3.3.3 and only 40s on npm 2.13.2, a 2.8x increase. The good news is that the number of files went from 25887 to 13261, only half of files :)

Repro steps:

git clone https://github.com/facebook/react-native.git
cd react-native
rm npm-shrinkwrap.json
rm -Rf node_modules
npm install -g npm
time npm install
# real  1m46.743s
# user  0m51.520s
# sys   0m12.608s
find node_modules | wc -l
# 25887
rm npm-shrinkwrap.json
rm -Rf node_modules
npm install -g npm@2.13.2
time npm install 
# real  0m40.008s
# user  0m38.214s
# sys   0m19.352s
find node_modules | wc -l
# 13261

vjeux commented Sep 20, 2015

If you want another data point. react-native is taking 1m53s on npm 3.3.3 and only 40s on npm 2.13.2, a 2.8x increase. The good news is that the number of files went from 25887 to 13261, only half of files :)

Repro steps:

git clone https://github.com/facebook/react-native.git
cd react-native
rm npm-shrinkwrap.json
rm -Rf node_modules
npm install -g npm
time npm install
# real  1m46.743s
# user  0m51.520s
# sys   0m12.608s
find node_modules | wc -l
# 25887
rm npm-shrinkwrap.json
rm -Rf node_modules
npm install -g npm@2.13.2
time npm install 
# real  0m40.008s
# user  0m38.214s
# sys   0m19.352s
find node_modules | wc -l
# 13261
@timdp

This comment has been minimized.

Show comment
Hide comment
@timdp

timdp Sep 22, 2015

Is there any way to get more useful progress reporting? Increasing verbosity is great but it doesn't help in the common case where you basically want an indication of how long the install is going to take. Even a very rough estimate would be better than nothing.

timdp commented Sep 22, 2015

Is there any way to get more useful progress reporting? Increasing verbosity is great but it doesn't help in the common case where you basically want an indication of how long the install is going to take. Even a very rough estimate would be better than nothing.

@sindresorhus

This comment has been minimized.

Show comment
Hide comment
@sindresorhus

sindresorhus Sep 23, 2015

We're seeing a huge regression in npm@3 in polymer-starter-kit.

The following package.json takes ~2 minutes to install on npm@2, but ~10-14 minutes on npm@3.

Node.js 4.1.1. OS X 10.10.

Here's a npm-shrinkwrap.json of the dependencies for reproducibility.

sindresorhus commented Sep 23, 2015

We're seeing a huge regression in npm@3 in polymer-starter-kit.

The following package.json takes ~2 minutes to install on npm@2, but ~10-14 minutes on npm@3.

Node.js 4.1.1. OS X 10.10.

Here's a npm-shrinkwrap.json of the dependencies for reproducibility.

@mcnameej mcnameej referenced this issue Sep 23, 2015

Closed

Discussion: LTS & v5 release planning #3000

7 of 8 tasks complete
@contra

This comment has been minimized.

Show comment
Hide comment
@contra

contra Sep 24, 2015

Installs went from ~4mins -> ~12mins when switching from 2.2.18 to 3.3.3

Would recommend anything time sensitive (deployments) stay on 2.x

contra commented Sep 24, 2015

Installs went from ~4mins -> ~12mins when switching from 2.2.18 to 3.3.3

Would recommend anything time sensitive (deployments) stay on 2.x

@mourner

This comment has been minimized.

Show comment
Hide comment
@mourner

mourner Sep 24, 2015

@iarna any progress updates on this? The original issue description suggests it is "sometimes slower" when it's in fact more like "always 3 to 7 times slower", which is pretty critical and prevents a lot of people from upgrading.

mourner commented Sep 24, 2015

@iarna any progress updates on this? The original issue description suggests it is "sometimes slower" when it's in fact more like "always 3 to 7 times slower", which is pretty critical and prevents a lot of people from upgrading.

@iarna

This comment has been minimized.

Show comment
Hide comment
@iarna

iarna Sep 30, 2015

Member

@mourner Well, contrary to popular belief, it's actually 5-10 times faster in some scenarios too, but apparently none that you're using. =/

But everyone here will be happy to learn that we identified the source of almost all of the slow down and excised it this week. Starting with npm@3.3.6 we should be seeing substantial performance improvements any time your node_modules has a lot of modules in it.

Member

iarna commented Sep 30, 2015

@mourner Well, contrary to popular belief, it's actually 5-10 times faster in some scenarios too, but apparently none that you're using. =/

But everyone here will be happy to learn that we identified the source of almost all of the slow down and excised it this week. Starting with npm@3.3.6 we should be seeing substantial performance improvements any time your node_modules has a lot of modules in it.

@nmccready

This comment has been minimized.

Show comment
Hide comment
@nmccready

nmccready Dec 16, 2015

Probably too late for that. What would be best is making 4.X off 2.X and
start over. Cut your losses or be phased out by newer coolness. Speed is
what matters.

On Wed, Dec 16, 2015, 12:16 Luis Rogelio Hernández López <
notifications@github.com> wrote:

I think thy should let us choice if we want or not know what's going under
the hood.
In my case, I don't really care about NPM's internal operations.


Reply to this email directly or view it on GitHub
#8826 (comment).

VR,

Sent from my mobile device

Nicholas McCready
nmccready@gmail.com

nmccready commented Dec 16, 2015

Probably too late for that. What would be best is making 4.X off 2.X and
start over. Cut your losses or be phased out by newer coolness. Speed is
what matters.

On Wed, Dec 16, 2015, 12:16 Luis Rogelio Hernández López <
notifications@github.com> wrote:

I think thy should let us choice if we want or not know what's going under
the hood.
In my case, I don't really care about NPM's internal operations.


Reply to this email directly or view it on GitHub
#8826 (comment).

VR,

Sent from my mobile device

Nicholas McCready
nmccready@gmail.com

@othiym23

This comment has been minimized.

Show comment
Hide comment
@othiym23

othiym23 Dec 16, 2015

Contributor

rather be focusing on compatibility issues for the next year at least.

Right now,the CLI team is working on getting its test suite passing consistently under Travis on all versions of Node it supports. Once that's done, we're doing the same, but for Windows as well as UNIX, which will involve setting up Windows CI. This is long-overdue payment of some significant technical debt. After that work is complete, our two next priorities are fixing the outstanding bugs in the installer (there are a fair few we already know about), and improving npm@3's performance. Software takes as long as it takes, but I'm thinking it will take a month or two to start working on performance, not a year.

The basic gist of it is that it's 3-5 times slower because it "makes it much easier to understand what npm is doing".

That's a fairly ungenerous reading of what @iarna and I have said. npm@3's installer has a goal to be robust, and part of that is making it understandable so we can troubleshoot and debug it when it's behaving erratically, and so developers like you, who lack our intimate knowledge of the installer's design and implementation, can have a better understanding of why npm is doing what it's doing. Previous versions of the installer were exceedingly difficult to understand when e.g. the bundle dependency algorithm was doing unexpected things.

Maintainers say there are no obvious ways to bring NPM3 on par with NPM2 performance

This simply isn't true. It will require the involvement of more than the CLI team at npm, and it's going to take some time, but I'm fully confident that we'll return npm's performance to at least on par with npm@2.

Contributor

othiym23 commented Dec 16, 2015

rather be focusing on compatibility issues for the next year at least.

Right now,the CLI team is working on getting its test suite passing consistently under Travis on all versions of Node it supports. Once that's done, we're doing the same, but for Windows as well as UNIX, which will involve setting up Windows CI. This is long-overdue payment of some significant technical debt. After that work is complete, our two next priorities are fixing the outstanding bugs in the installer (there are a fair few we already know about), and improving npm@3's performance. Software takes as long as it takes, but I'm thinking it will take a month or two to start working on performance, not a year.

The basic gist of it is that it's 3-5 times slower because it "makes it much easier to understand what npm is doing".

That's a fairly ungenerous reading of what @iarna and I have said. npm@3's installer has a goal to be robust, and part of that is making it understandable so we can troubleshoot and debug it when it's behaving erratically, and so developers like you, who lack our intimate knowledge of the installer's design and implementation, can have a better understanding of why npm is doing what it's doing. Previous versions of the installer were exceedingly difficult to understand when e.g. the bundle dependency algorithm was doing unexpected things.

Maintainers say there are no obvious ways to bring NPM3 on par with NPM2 performance

This simply isn't true. It will require the involvement of more than the CLI team at npm, and it's going to take some time, but I'm fully confident that we'll return npm's performance to at least on par with npm@2.

@othiym23

This comment has been minimized.

Show comment
Hide comment
@othiym23

othiym23 Dec 16, 2015

Contributor

In my case, I don't really care about NPM's internal operations.

You don't care until it does something unexpected or wrong, and then you (in general, maybe not you in particular) care a great deal, because until somebody understands what's happening, you're blocked (potentially along with the rest of your team). 90+% of the time, npm-debug.log is the only tool available to the CLI team to understand buggy behavior, making it all but essential that it capture as much information as possible about what's going on with your installs.

Contributor

othiym23 commented Dec 16, 2015

In my case, I don't really care about NPM's internal operations.

You don't care until it does something unexpected or wrong, and then you (in general, maybe not you in particular) care a great deal, because until somebody understands what's happening, you're blocked (potentially along with the rest of your team). 90+% of the time, npm-debug.log is the only tool available to the CLI team to understand buggy behavior, making it all but essential that it capture as much information as possible about what's going on with your installs.

@Kurtz1993

This comment has been minimized.

Show comment
Hide comment
@Kurtz1993

Kurtz1993 Dec 16, 2015

@othiym23 that's why I said this: "They should let us choice if we want or not know what's going under the hood."
If a problem occurs I will want to know what went wrong, of course.

Kurtz1993 commented Dec 16, 2015

@othiym23 that's why I said this: "They should let us choice if we want or not know what's going under the hood."
If a problem occurs I will want to know what went wrong, of course.

@iamstarkov

This comment has been minimized.

Show comment
Hide comment
@iamstarkov

iamstarkov Dec 16, 2015

@othiym23 is there any details/reasons behind chosen priorities? I mean why tech debt have been considered more important than community's problems. I don't know full picture, so I will be happy to know

iamstarkov commented Dec 16, 2015

@othiym23 is there any details/reasons behind chosen priorities? I mean why tech debt have been considered more important than community's problems. I don't know full picture, so I will be happy to know

@othiym23

This comment has been minimized.

Show comment
Hide comment
@othiym23

othiym23 Dec 16, 2015

Contributor

is there any details/reasons behind chosen priorities?

The npm CLI team includes one manager (me) and two full-time engineers, and has a very heavy workload. We don't have dedicated QA resources, so our only indicators of the quality of the product we're shipping are the number of inbound issues we see and our test suite. If we don't trust that the test suite is telling us that our code base is in good shape, we're in trouble. Right now, the team has insufficient faith in the test suite. So we need to fix that!

Also, it tremendously simplifies the process of landing contributions from the community if we can look at the Travis build and know whether the new code causes any problems without pulling down changes and running the tests ourselves. And it's an important part of our commitment to Windows users to have continuous integration testing things on their platform.

Finally, the internals of npm are sufficiently difficult to understand that most users of npm never make the effort, and when they do, very few of them master it enough to make meaningful contributions to the core. The architecture is complex and subtle, and part of the reason npm@3 exists is to make it more comprehensible, so that the CLI team is no longer so much of a gatekeeper for finding and fixing issues in the installer.

We've put off paying down this debt for way too long, to the point that it's significantly impairing the ability of the project to add new features (and increase the size of the contributor base). It's not that this technical debt is more important than the community's problems, it's that it is a community problem. The community problem, in my estimation.

Contributor

othiym23 commented Dec 16, 2015

is there any details/reasons behind chosen priorities?

The npm CLI team includes one manager (me) and two full-time engineers, and has a very heavy workload. We don't have dedicated QA resources, so our only indicators of the quality of the product we're shipping are the number of inbound issues we see and our test suite. If we don't trust that the test suite is telling us that our code base is in good shape, we're in trouble. Right now, the team has insufficient faith in the test suite. So we need to fix that!

Also, it tremendously simplifies the process of landing contributions from the community if we can look at the Travis build and know whether the new code causes any problems without pulling down changes and running the tests ourselves. And it's an important part of our commitment to Windows users to have continuous integration testing things on their platform.

Finally, the internals of npm are sufficiently difficult to understand that most users of npm never make the effort, and when they do, very few of them master it enough to make meaningful contributions to the core. The architecture is complex and subtle, and part of the reason npm@3 exists is to make it more comprehensible, so that the CLI team is no longer so much of a gatekeeper for finding and fixing issues in the installer.

We've put off paying down this debt for way too long, to the point that it's significantly impairing the ability of the project to add new features (and increase the size of the contributor base). It's not that this technical debt is more important than the community's problems, it's that it is a community problem. The community problem, in my estimation.

@iamstarkov

This comment has been minimized.

Show comment
Hide comment
@iamstarkov

iamstarkov commented Dec 16, 2015

@othiym23 thank you

@Zarel

This comment has been minimized.

Show comment
Hide comment
@Zarel

Zarel Jan 26, 2016

Contributor

You don't care until it does something unexpected or wrong, and then you (in general, maybe not you in particular) care a great deal, because until somebody understands what's happening, you're blocked (potentially along with the rest of your team). 90+% of the time, npm-debug.log is the only tool available to the CLI team to understand buggy behavior, making it all but essential that it capture as much information as possible about what's going on with your installs.

It sounds like this should be off by default, then. If a command fails, the user can re-run it with logging enabled, thus solving both the performance problem and the debugging problem.

Contributor

Zarel commented Jan 26, 2016

You don't care until it does something unexpected or wrong, and then you (in general, maybe not you in particular) care a great deal, because until somebody understands what's happening, you're blocked (potentially along with the rest of your team). 90+% of the time, npm-debug.log is the only tool available to the CLI team to understand buggy behavior, making it all but essential that it capture as much information as possible about what's going on with your installs.

It sounds like this should be off by default, then. If a command fails, the user can re-run it with logging enabled, thus solving both the performance problem and the debugging problem.

@josephgodwinkimani

This comment has been minimized.

Show comment
Hide comment
@josephgodwinkimani

josephgodwinkimani Jan 27, 2016

I have experienced the same problem for the last two days trying to npm install dependencies for meanjs but I opted to use Windows command processor. It did it in record time obviously the progress bar got halfway..hope we can find a solution. issue #11283 and issue #8826

josephgodwinkimani commented Jan 27, 2016

I have experienced the same problem for the last two days trying to npm install dependencies for meanjs but I opted to use Windows command processor. It did it in record time obviously the progress bar got halfway..hope we can find a solution. issue #11283 and issue #8826

@Vanuan

This comment has been minimized.

Show comment
Hide comment
@Vanuan

Vanuan Jan 28, 2016

From all those time reports it's quite obvious that while user and system times are the same, real time is considerably bigger. It means that npm3 is either waiting on I/O or is not utilizing all CPU cores. It's quite ironic, because the selling point of node was "it's very fast because it's not blocked on I/O, and it doesn't need threads".

So JavaScript is not as efficient for scripting. Even more ironic. :trollface:

Vanuan commented Jan 28, 2016

From all those time reports it's quite obvious that while user and system times are the same, real time is considerably bigger. It means that npm3 is either waiting on I/O or is not utilizing all CPU cores. It's quite ironic, because the selling point of node was "it's very fast because it's not blocked on I/O, and it doesn't need threads".

So JavaScript is not as efficient for scripting. Even more ironic. :trollface:

@cchamberlain

This comment has been minimized.

Show comment
Hide comment
@cchamberlain

cchamberlain Jan 28, 2016

Many of the people on this thread do not seem to understand that npm 3 was a major rearchitecture. Comments about flipping a switch to jump between npm 2 and npm 3 installs are naive (if you want npm 2, install npm 2...). npm is the most platform agnostic package manager in existence and if you think that npm 3 was a regression try using each version on Windows for a few months. npm 2 is a mess on Windows (long file paths...), installs are often insurmountable and require bypassing packages. npm 3 has brought incredible stability. On npm 2, removal of node_modules was a daily occurrence and now I can get faster and more stable results with normal incremental npm install. Bottom line, I spend orders of magnitude less hours hacking npm 3 than npm 2 because it actually works. If you are doing constant npm installs from scratch than maybe you should optimize your build process instead of trolling the awesomely helpful npm team. If you can rewrite the logic for speed and maintain the stability that npm 3 brings than submit your pull request.

cchamberlain commented Jan 28, 2016

Many of the people on this thread do not seem to understand that npm 3 was a major rearchitecture. Comments about flipping a switch to jump between npm 2 and npm 3 installs are naive (if you want npm 2, install npm 2...). npm is the most platform agnostic package manager in existence and if you think that npm 3 was a regression try using each version on Windows for a few months. npm 2 is a mess on Windows (long file paths...), installs are often insurmountable and require bypassing packages. npm 3 has brought incredible stability. On npm 2, removal of node_modules was a daily occurrence and now I can get faster and more stable results with normal incremental npm install. Bottom line, I spend orders of magnitude less hours hacking npm 3 than npm 2 because it actually works. If you are doing constant npm installs from scratch than maybe you should optimize your build process instead of trolling the awesomely helpful npm team. If you can rewrite the logic for speed and maintain the stability that npm 3 brings than submit your pull request.

@vroudge

This comment has been minimized.

Show comment
Hide comment
@vroudge

vroudge Jan 28, 2016

@cchamberlain This is just feedback. Build processes have become a pain in the ass lately because of this issue ; because yes, it is an issue, with speed.

I'm just reporting the great deal of time an install now takes. What is it these days that you can't provide feedback to an open-source project without people saying you're trolling the contrib team.

Am I ungrateful by saying that a product got worse for the use I make of it?

vroudge commented Jan 28, 2016

@cchamberlain This is just feedback. Build processes have become a pain in the ass lately because of this issue ; because yes, it is an issue, with speed.

I'm just reporting the great deal of time an install now takes. What is it these days that you can't provide feedback to an open-source project without people saying you're trolling the contrib team.

Am I ungrateful by saying that a product got worse for the use I make of it?

@cchamberlain

This comment has been minimized.

Show comment
Hide comment
@cchamberlain

cchamberlain Jan 28, 2016

@vroudge - npm 3 is slower than npm 2. It is well documented and no secret. It's slower because it fixes many npm 2 issues. It will get faster but the use you make of it is a very small piece of the puzzle. Speeds great but it's much more important that it does everything it claims it can do without randomly bombing out. Chiming in that it's slower is not useful, if you prefer the npm 2 nested installs you should use that since they are still maintaining it.

cchamberlain commented Jan 28, 2016

@vroudge - npm 3 is slower than npm 2. It is well documented and no secret. It's slower because it fixes many npm 2 issues. It will get faster but the use you make of it is a very small piece of the puzzle. Speeds great but it's much more important that it does everything it claims it can do without randomly bombing out. Chiming in that it's slower is not useful, if you prefer the npm 2 nested installs you should use that since they are still maintaining it.

@vroudge

This comment has been minimized.

Show comment
Hide comment
@vroudge

vroudge Jan 28, 2016

@cchamberlain Does not change the fact that it is an issue and is seen as an issue by the maintainers of the project.

The new modules flat structure is really wonderful and I love it. And npm works wonders, really. It's just really slow, that's it. When you see things like the fact that the progress bar makes it reaaally slower, you just begin to wonder if it could not be made faster, and that's it. I don't chime in just saying that it's slower just to QQ at the fact it's slower, there are many things that make it slower, not just the new tree. I just think it should be investigated. Not that I have the knowledge to do it.

Lets stop this here and stop polluting the issues.

vroudge commented Jan 28, 2016

@cchamberlain Does not change the fact that it is an issue and is seen as an issue by the maintainers of the project.

The new modules flat structure is really wonderful and I love it. And npm works wonders, really. It's just really slow, that's it. When you see things like the fact that the progress bar makes it reaaally slower, you just begin to wonder if it could not be made faster, and that's it. I don't chime in just saying that it's slower just to QQ at the fact it's slower, there are many things that make it slower, not just the new tree. I just think it should be investigated. Not that I have the knowledge to do it.

Lets stop this here and stop polluting the issues.

@Vanuan

This comment has been minimized.

Show comment
Hide comment
@Vanuan

Vanuan Jan 28, 2016

@cchamberlain oh, we've got who to blame - windows guys. Why won't you just start using more developer-friendly operating systems :trollface:

Vanuan commented Jan 28, 2016

@cchamberlain oh, we've got who to blame - windows guys. Why won't you just start using more developer-friendly operating systems :trollface:

@naholyr

This comment has been minimized.

Show comment
Hide comment
@naholyr

naholyr Jan 28, 2016

@cchamberlain it fixes nothing, it brought new issues instead (some conceptual like the fact you can require a sub-dependency now without warning, some very important for an install tool like partial installs with no feedback), and fixed a non-issue for non-Windows developers. It could even have been fixed by a specific command as the too deep folders could still be removed programmatically.

naholyr commented Jan 28, 2016

@cchamberlain it fixes nothing, it brought new issues instead (some conceptual like the fact you can require a sub-dependency now without warning, some very important for an install tool like partial installs with no feedback), and fixed a non-issue for non-Windows developers. It could even have been fixed by a specific command as the too deep folders could still be removed programmatically.

@davidknezic davidknezic referenced this issue Jan 31, 2016

Merged

Improve Docker build times #445

2 of 5 tasks complete
@shehi

This comment has been minimized.

Show comment
Hide comment
@shehi

shehi Feb 28, 2016

I am on Ubuntu and to install laravel-elixir@5 it took me 30+ mins with 8Mbps connection. It never took this long before when I was in v2. I am working through Docker.

shehi commented Feb 28, 2016

I am on Ubuntu and to install laravel-elixir@5 it took me 30+ mins with 8Mbps connection. It never took this long before when I was in v2. I am working through Docker.

@dsernst

This comment has been minimized.

Show comment
Hide comment
@dsernst

dsernst Mar 2, 2016

so it seems like the speed question is pretty well established at this point. Is there a clear path for continuing to move forward? Can we update the OP comment with links to subtasks so we can start knocking them out?

dsernst commented Mar 2, 2016

so it seems like the speed question is pretty well established at this point. Is there a clear path for continuing to move forward? Can we update the OP comment with links to subtasks so we can start knocking them out?

@tfennelly

This comment has been minimized.

Show comment
Hide comment
@tfennelly

tfennelly Mar 7, 2016

Hi @othiym23. Thanks for all your work on npm in general and on npm@3 (making it more robust etc).

Back in December, you said you were hoping that some cycles would be spent on npm@3 performance within the following month or so. Are there any updates on that?

tfennelly commented Mar 7, 2016

Hi @othiym23. Thanks for all your work on npm in general and on npm@3 (making it more robust etc).

Back in December, you said you were hoping that some cycles would be spent on npm@3 performance within the following month or so. Are there any updates on that?

@othiym23

This comment has been minimized.

Show comment
Hide comment
@othiym23

othiym23 Mar 7, 2016

Contributor

This is why I typically avoid making date-based estimates. ;)

The CLI team has been making substantial progress in rewriting npm's test suite to make it less hairy, and to pass on Windows. The latter, especially, is an important quality milestone for the project, which is why the team is focused on test work for the time being.

After that, the team will be tackling the list of significant known issues. After that is when we've scheduled getting the whole team working on improving performance, which will most likely include some architectural work as well.

All that said, that's just what the CLI team itself is focusing on. The team continues to land community PRs as quickly as it can; several recent PRs have resulted in significant performance improvements in some cases.

Thanks to all for their contributions and patience.

Contributor

othiym23 commented Mar 7, 2016

This is why I typically avoid making date-based estimates. ;)

The CLI team has been making substantial progress in rewriting npm's test suite to make it less hairy, and to pass on Windows. The latter, especially, is an important quality milestone for the project, which is why the team is focused on test work for the time being.

After that, the team will be tackling the list of significant known issues. After that is when we've scheduled getting the whole team working on improving performance, which will most likely include some architectural work as well.

All that said, that's just what the CLI team itself is focusing on. The team continues to land community PRs as quickly as it can; several recent PRs have resulted in significant performance improvements in some cases.

Thanks to all for their contributions and patience.

@iamstarkov

This comment has been minimized.

Show comment
Hide comment
@iamstarkov

iamstarkov Sep 6, 2016

@othiym23 several months passed. can you share some kind of progress on this issue?

iamstarkov commented Sep 6, 2016

@othiym23 several months passed. can you share some kind of progress on this issue?

@othiym23

This comment has been minimized.

Show comment
Hide comment
@othiym23

othiym23 Sep 7, 2016

Contributor

The CLI team has been making substantial progress in rewriting npm's test suite to make it less hairy, and to pass on Windows.

This is (mostly) done. There's some wiggliness on AppVeyor when running the CI suite, and it's required some TLC from me that I haven't had time to provide. The test suite, however, is in a much better place than it was in in March.

After that, the team will be tackling the list of significant known issues.

This work is still in progress (see the label, which should give you an idea of how many issues are in that bucket). It's finicky, demanding work, and the team remains small, so progress is moderate, but real.

We've also landed several patches, including the new version of gauges and are-we-there-yet in npmlog, with (almost exclusively positive) performance implications. I believe that overall the performance of the CLI is significantly better, with fewer pathological corner cases, than it was in March. That said, there's still a ways to go before the team can turn its energies towards making more wholesale improvements to the CLI's performance at a deeper, more architectural level. When we do start work on that, we'll say as much here.

Contributor

othiym23 commented Sep 7, 2016

The CLI team has been making substantial progress in rewriting npm's test suite to make it less hairy, and to pass on Windows.

This is (mostly) done. There's some wiggliness on AppVeyor when running the CI suite, and it's required some TLC from me that I haven't had time to provide. The test suite, however, is in a much better place than it was in in March.

After that, the team will be tackling the list of significant known issues.

This work is still in progress (see the label, which should give you an idea of how many issues are in that bucket). It's finicky, demanding work, and the team remains small, so progress is moderate, but real.

We've also landed several patches, including the new version of gauges and are-we-there-yet in npmlog, with (almost exclusively positive) performance implications. I believe that overall the performance of the CLI is significantly better, with fewer pathological corner cases, than it was in March. That said, there's still a ways to go before the team can turn its energies towards making more wholesale improvements to the CLI's performance at a deeper, more architectural level. When we do start work on that, we'll say as much here.

@othiym23

This comment has been minimized.

Show comment
Hide comment
@othiym23

othiym23 Sep 7, 2016

Contributor

Gosh darn it, I keep closing this issue. Freudian close?

Contributor

othiym23 commented Sep 7, 2016

Gosh darn it, I keep closing this issue. Freudian close?

@othiym23 othiym23 reopened this Sep 7, 2016

@othiym23

This comment has been minimized.

Show comment
Hide comment
@othiym23

othiym23 Sep 7, 2016

Contributor

No, wait, it was closed already! ;_; so confused!

Contributor

othiym23 commented Sep 7, 2016

No, wait, it was closed already! ;_; so confused!

@othiym23 othiym23 closed this Sep 7, 2016

@mourner

This comment has been minimized.

Show comment
Hide comment
@mourner

mourner Sep 7, 2016

I believe that overall the performance of the CLI is significantly better, with fewer pathological corner cases, than it was in March.

There has been some progress, but unfortunately not nearly enough to consider this issue closed. NPM 3 still has consistently 2-3x slower install times than NPM 2, and 5-6x slower times than popular, actively developed alternatives like pnpm and ied (which also flatten the dependency tree) — you can confirm this on any popular Node package, and the alternatives (each having half a thousand commits and tens of contributors) wouldn't even exist in the first place if this wasn't a critical problem.

I'm thinking it will take a month or two to start working on performance, not a year.

It has been 9 months since the comment above...

mourner commented Sep 7, 2016

I believe that overall the performance of the CLI is significantly better, with fewer pathological corner cases, than it was in March.

There has been some progress, but unfortunately not nearly enough to consider this issue closed. NPM 3 still has consistently 2-3x slower install times than NPM 2, and 5-6x slower times than popular, actively developed alternatives like pnpm and ied (which also flatten the dependency tree) — you can confirm this on any popular Node package, and the alternatives (each having half a thousand commits and tens of contributors) wouldn't even exist in the first place if this wasn't a critical problem.

I'm thinking it will take a month or two to start working on performance, not a year.

It has been 9 months since the comment above...

@mourner

This comment has been minimized.

Show comment
Hide comment
@mourner

mourner Sep 7, 2016

P.S. I highly appreciate the hard work you're doing and apologize for being annoying. Just trying to keep performance closer on your radar because it significantly affects my life as an open source maintainer.

mourner commented Sep 7, 2016

P.S. I highly appreciate the hard work you're doing and apologize for being annoying. Just trying to keep performance closer on your radar because it significantly affects my life as an open source maintainer.

@vjpr

This comment has been minimized.

Show comment
Hide comment
@vjpr

vjpr Sep 7, 2016

Is there an open issue to track progress with performance?

I recently tried out pnpm and it is incredibly fast. It makes Node fun again. I think the community should invest more in pnpm and ied, than expect npm to improve. I have a feeling the performance issues are too deep, and would require a huge amount of re-architecting.

Npm3 has done great work in pushing the idea of the deduped by default node_modules but the non-determinism seems hacky, when pnpm solves this perfectly.

The Node resolver is over-complicated. The pnpm model of having a central store with all modules installed as module-name@version is simple, elegant, and easy to understand.

What would be best is if when requiring a module, the resolver would read the closest package.json file, and find the best version to use from a central store.

If two require('foo') statements resolved to two different absolute paths, then a warning would be shown to help the user update package version ranges to prevent two modules being evaled. At present npm linking causing huge amounts of duplicate module evals. Most importantly this fixes performance issues with npm linking which involves evaling many copies of the same module/version.

I have implemented all of this in https://www.npmjs.com/package/live-perf and have seen huge speedups. The only downside is the resolver needs to be cached, because the dynamic resolving has an overhead as every require statement needs to find its closest package.json file with syscalls.

I think these changes to the Node resolver would make the ultimate package manager.

The end goal should be not to ever worry about whats in node_modules, and be able to rely on auto-installing ala https://github.com/siddharthkp/auto-install.

Imagine being able to write code without ever having to run npm install or rm -rf node_modules.

vjpr commented Sep 7, 2016

Is there an open issue to track progress with performance?

I recently tried out pnpm and it is incredibly fast. It makes Node fun again. I think the community should invest more in pnpm and ied, than expect npm to improve. I have a feeling the performance issues are too deep, and would require a huge amount of re-architecting.

Npm3 has done great work in pushing the idea of the deduped by default node_modules but the non-determinism seems hacky, when pnpm solves this perfectly.

The Node resolver is over-complicated. The pnpm model of having a central store with all modules installed as module-name@version is simple, elegant, and easy to understand.

What would be best is if when requiring a module, the resolver would read the closest package.json file, and find the best version to use from a central store.

If two require('foo') statements resolved to two different absolute paths, then a warning would be shown to help the user update package version ranges to prevent two modules being evaled. At present npm linking causing huge amounts of duplicate module evals. Most importantly this fixes performance issues with npm linking which involves evaling many copies of the same module/version.

I have implemented all of this in https://www.npmjs.com/package/live-perf and have seen huge speedups. The only downside is the resolver needs to be cached, because the dynamic resolving has an overhead as every require statement needs to find its closest package.json file with syscalls.

I think these changes to the Node resolver would make the ultimate package manager.

The end goal should be not to ever worry about whats in node_modules, and be able to rely on auto-installing ala https://github.com/siddharthkp/auto-install.

Imagine being able to write code without ever having to run npm install or rm -rf node_modules.

@halhenke

This comment has been minimized.

Show comment
Hide comment
@halhenke

halhenke Sep 8, 2016

Contributor

Its been a while since i looked at the internals of npm (did a tiny PR) or pnpm (meant to do a fairly big feature but....never did 😞 ) but i think its fair to point out that npm does deal with a lot more edge cases/supported methods of installation (git repos etc) than these other clients.

OTOH i do think there is a fundamental flaw (at least from a performance perspective) in the way that npm3 tries to build a model of the ideal tree before going about the business of pulling down packages...again the details escape me but - as mentioned before the npm 3 package resolution/install algorithm is very complicated and quite tightly coupled to a lot of other parts. My knowledge is fuzzy now and quite possibly out of date but i do think a more or less ground up re-implementation/rethink of how packages are resolved/fetched would be necessary to see notable performance gains. Just my thoughts/opinions.

Contributor

halhenke commented Sep 8, 2016

Its been a while since i looked at the internals of npm (did a tiny PR) or pnpm (meant to do a fairly big feature but....never did 😞 ) but i think its fair to point out that npm does deal with a lot more edge cases/supported methods of installation (git repos etc) than these other clients.

OTOH i do think there is a fundamental flaw (at least from a performance perspective) in the way that npm3 tries to build a model of the ideal tree before going about the business of pulling down packages...again the details escape me but - as mentioned before the npm 3 package resolution/install algorithm is very complicated and quite tightly coupled to a lot of other parts. My knowledge is fuzzy now and quite possibly out of date but i do think a more or less ground up re-implementation/rethink of how packages are resolved/fetched would be necessary to see notable performance gains. Just my thoughts/opinions.

@pannous

This comment has been minimized.

Show comment
Hide comment
@pannous

pannous Dec 29, 2016

All the euphemisms and kindness aside:

time npm -g ls
real	0m45.109s

Labels: big-bug good
Closed: why?

These suggestions alleviated the situation dramatically:

npm set progress=false
mv /usr/local/lib/node_modules /usr/local/lib/node_modules.TOO_MANY
cp -r /usr/local/lib/node_modules.TOO_MANY/npm /usr/local/lib/node_modules/

Lesson learned: use global modules very sparingly

pannous commented Dec 29, 2016

All the euphemisms and kindness aside:

time npm -g ls
real	0m45.109s

Labels: big-bug good
Closed: why?

These suggestions alleviated the situation dramatically:

npm set progress=false
mv /usr/local/lib/node_modules /usr/local/lib/node_modules.TOO_MANY
cp -r /usr/local/lib/node_modules.TOO_MANY/npm /usr/local/lib/node_modules/

Lesson learned: use global modules very sparingly

@iarna

This comment has been minimized.

Show comment
Hide comment
@iarna

iarna Jan 2, 2017

Member

@pannous npm ls -g is going to take as long as it takes to list all your global modules in logical tree form, but lots of global modules shouldn't have any impact on any other operation. If you are seeing an impact from having a lot of global modules on any operation other than npm ls -g then please open an issue!

PS disabling your progress bar really REALLY shouldn't have any measurable impact on run time. (I've not seen a benchmark yet where it did.) The progress bar is extremely low impact these days.

Member

iarna commented Jan 2, 2017

@pannous npm ls -g is going to take as long as it takes to list all your global modules in logical tree form, but lots of global modules shouldn't have any impact on any other operation. If you are seeing an impact from having a lot of global modules on any operation other than npm ls -g then please open an issue!

PS disabling your progress bar really REALLY shouldn't have any measurable impact on run time. (I've not seen a benchmark yet where it did.) The progress bar is extremely low impact these days.

@MattFoley

This comment has been minimized.

Show comment
Hide comment
@MattFoley

MattFoley Feb 14, 2017

One thing I've noticed is that performance seems to drop considerably when your package.json contains urls to specific commits, rather than npm packages.

MattFoley commented Feb 14, 2017

One thing I've noticed is that performance seems to drop considerably when your package.json contains urls to specific commits, rather than npm packages.

@vvo

This comment has been minimized.

Show comment
Hide comment
@vvo

vvo Feb 14, 2017

@MattFoley I believe this is something to expect, when specifying a git commit you then have to fire up git command, clone, checkout or something like that. Way slower than downloading a tgz

vvo commented Feb 14, 2017

@MattFoley I believe this is something to expect, when specifying a git commit you then have to fire up git command, clone, checkout or something like that. Way slower than downloading a tgz

@MattFoley

This comment has been minimized.

Show comment
Hide comment
@MattFoley

MattFoley Feb 15, 2017

@vvo I think you're right, but maybe there are some improvements that could be made? Maybe a shallow checkout? It may not even be the git checkout that's causing the hang, not sure.

Seems like it hangs on fetchMetadata, although I don't know enough about npm to really say.

MattFoley commented Feb 15, 2017

@vvo I think you're right, but maybe there are some improvements that could be made? Maybe a shallow checkout? It may not even be the git checkout that's causing the hang, not sure.

Seems like it hangs on fetchMetadata, although I don't know enough about npm to really say.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.