New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

NPM Install Speed #553

Closed
RDalziel opened this Issue Sep 17, 2015 · 38 comments

Comments

Projects
None yet
@RDalziel

RDalziel commented Sep 17, 2015

Is there any way with the current build task to increase the speed of NPM installs?

I've seen methods of caching being referenced around the web, but unsure that's possible with the current NPM Build Task setup.

NPM installs are usually the main cause of slow builds on front-end builds at the moment, any way of speeding them up would be a great boost!

@bryanmacfarlane

This comment has been minimized.

Member

bryanmacfarlane commented Sep 17, 2015

We are considering installing npm read through caching servers in the hosted build pool. I assume you are referencing hosted build (and not your own agent).

The other immediate way to speed up npm step considerably is to install your own agent on an azure VM ( http://aka.ms/tfbuild ) - that's a persistent agent so you get incremental. modules would be there from build to build and would only pull new one when you use a new one/version.

@RDalziel

This comment has been minimized.

RDalziel commented Sep 17, 2015

Cheers Bryan, i'm actually On Premise here. Are you basically referring to not doing a clean build when you refer to incremental modules? I'm not a huge fan of moving away from clean builds as i've had problems in the past with lingering modules masking errors. There's also the problem of artifacts lingering between builds.

Have you got a solution worked for these without writing custom steps to clean up folders?

@bryanmacfarlane

This comment has been minimized.

Member

bryanmacfarlane commented Sep 17, 2015

Well, we're just calling npm so it's between your box, your network and the npm services.

That said, there is a solution. You can install a read through npm cache. That will allow you to do clean builds and still get fast npm installs.

Something like this is what we were looking at for our hosted pool:

https://www.npmjs.com/package/sinopia

You could install it right on the agent or on another box local to your network. You would point your npm config on the agent at that read through cache server.

@RDalziel

This comment has been minimized.

RDalziel commented Sep 17, 2015

Cheers Bryan !

@bryanmacfarlane

This comment has been minimized.

Member

bryanmacfarlane commented Sep 17, 2015

I also just found this which will be simpler and alleviate the problem with an on prem agent:

https://www.npmjs.com/package/npm-cache

@RDalziel

This comment has been minimized.

RDalziel commented Sep 17, 2015

Cheers again Bryan, i did have a go with Sinopia but had some trouble getting it working reliably. Their documentation for installing as a windows service was a bit date also.

Will give npm-cache a go!

@RDalziel

This comment has been minimized.

RDalziel commented Sep 20, 2015

FYI Bryan. I had a look at npm-cache - It looks great but it's not a viable option on windows boxes for now. Apparently it's getting worked on though!

@RDalziel

This comment has been minimized.

RDalziel commented Sep 24, 2015

Again - FYI. I've ended up using Sinopia as we wanted to start hosting internal NPM packages. It's taken a 14 min build down to around 11 minutes.

@bryanmacfarlane

This comment has been minimized.

Member

bryanmacfarlane commented Sep 24, 2015

Awesome

@mpseidel

This comment has been minimized.

mpseidel commented Oct 1, 2016

+1 for caching npm modules on hosted build agents. Services like codeship do this and it makes the whole experience more fun :)

@marcwittke

This comment has been minimized.

marcwittke commented Oct 25, 2016

We are considering installing npm read through caching servers in the hosted build pool.

@bryanmacfarlane did you finalize the consideration? This issue was closed without any feedback regarding this. My npm install of a react boilerplate js application takes about four minutes on the hosted build agent :(

@bryanmacfarlane

This comment has been minimized.

Member

bryanmacfarlane commented Oct 25, 2016

assigning to zack for thoughts. Last time we looked at the bottleneck, npm local I/O (updating local cache folders) was dominating the time on hosted agents. Since then though, we've sped up I/O with D2+SSD. In addition, resolving the graph of versions might also dominate the time before it ever gets to pulling them from the cache server. So, it's unclear whether it would help much on perf. Zack drives the packaging team so he can talk to it.

@marcwittke

This comment has been minimized.

marcwittke commented Oct 25, 2016

@zjrunner for what it's worth, here my (shortened) log. And yes, @bryanmacfarlane might be right with his thoughts about the domination of graph resolution...

2016-10-25T22:49:58.5283037Z [command]C:\Program Files\nodejs\npm.cmd install
2016-10-25T22:50:27.1975224Z npm WARN deprecated tough-cookie@2.2.2: ReDoS vulnerability parsing Set-Cookie https://nodesecurity.io/advisories/130
2016-10-25T22:50:47.2245925Z npm WARN deprecated cross-spawn-async@2.2.4: cross-spawn no longer requires a build toolchain, use it instead!
2016-10-25T22:52:30.6282644Z npm WARN prefer global colorguard@1.2.0 should be installed with -g
2016-10-25T22:52:36.8852554Z 
2016-10-25T22:52:36.8852554Z > pre-commit@1.1.3 install C:\a\1\s\mep-frontend\node_modules\pre-commit
2016-10-25T22:52:36.8852554Z > node install.js
2016-10-25T22:52:36.8852554Z 
2016-10-25T22:52:40.8275357Z 
2016-10-25T22:52:40.8275357Z > spawn-sync@1.0.13 postinstall C:\a\1\s\mep-frontend\node_modules\spawn-sync
2016-10-25T22:52:40.8275357Z > node postinstall
2016-10-25T22:52:40.8275357Z 
2016-10-25T22:52:41.4805353Z 
2016-10-25T22:52:41.4805353Z > ngrok@2.2.2 postinstall C:\a\1\s\mep-frontend\node_modules\ngrok
2016-10-25T22:52:41.4805353Z > node ./postinstall.js
2016-10-25T22:52:41.4805353Z 
2016-10-25T22:52:42.2975440Z ngrok - downloading binary https://bin.equinox.io/a/chRZWPptE7w/ngrok-2.1.1-windows-amd64.zip ...
2016-10-25T22:52:42.6163623Z ngrok - binary downloaded...
2016-10-25T22:52:42.8123618Z ngrok - binary unpacked.
2016-10-25T22:52:42.8943627Z 
2016-10-25T22:52:42.8943627Z > lint-staged@3.0.2 postinstall C:\a\1\s\mep-frontend\node_modules\lint-staged
2016-10-25T22:52:42.8943627Z > echo "lint-staged installed! Do not forget to configure it. See https://github.com/okonet/lint-staged/blob/master/README.md" && exit 0
2016-10-25T22:52:42.8953623Z 
2016-10-25T22:52:42.9013611Z "lint-staged installed! Do not forget to configure it. See https://github.com/okonet/lint-staged/blob/master/README.md" 
2016-10-25T22:52:42.9733639Z 
2016-10-25T22:52:42.9733639Z > pngquant-bin@3.1.1 postinstall C:\a\1\s\mep-frontend\node_modules\pngquant-bin
2016-10-25T22:52:42.9733639Z > node lib/install.js
2016-10-25T22:52:42.9733639Z 
2016-10-25T22:52:44.1292582Z   √ pngquant pre-build test passed successfully
2016-10-25T22:52:44.1412610Z 
2016-10-25T22:52:44.1422585Z > optipng-bin@3.1.2 postinstall C:\a\1\s\mep-frontend\node_modules\optipng-bin
2016-10-25T22:52:44.1422585Z > node lib/install.js
2016-10-25T22:52:44.1422585Z 
2016-10-25T22:52:45.0444527Z   √ optipng pre-build test passed successfully
2016-10-25T22:52:45.0564530Z 
2016-10-25T22:52:45.0564530Z > jpegtran-bin@3.1.0 postinstall C:\a\1\s\mep-frontend\node_modules\jpegtran-bin
2016-10-25T22:52:45.0564530Z > node lib/install.js
2016-10-25T22:52:45.0564530Z 
2016-10-25T22:52:46.0140709Z   √ jpegtran pre-build test passed successfully
2016-10-25T22:52:46.0270705Z 
2016-10-25T22:52:46.0270705Z > gifsicle@3.0.4 postinstall C:\a\1\s\mep-frontend\node_modules\gifsicle
2016-10-25T22:52:46.0270705Z > node lib/install.js
2016-10-25T22:52:46.0270705Z 
2016-10-25T22:52:46.9020711Z   √ gifsicle pre-build test passed successfully
2016-10-25T22:52:47.0660710Z 
2016-10-25T22:52:47.0660710Z > react-boilerplate@3.2.1 preinstall C:\a\1\s\mep-frontend
2016-10-25T22:52:47.0660710Z > npm run npmcheckversion
2016-10-25T22:52:47.0660710Z 
2016-10-25T22:52:48.7030713Z 
2016-10-25T22:52:48.7040709Z > react-boilerplate@3.2.1 npmcheckversion C:\a\1\s\mep-frontend
2016-10-25T22:52:48.7040709Z > node ./internals/scripts/npmcheckversion.js
2016-10-25T22:52:48.7040709Z 
2016-10-25T22:52:50.1260706Z 
2016-10-25T22:52:50.1260706Z > react-boilerplate@3.2.1 postinstall C:\a\1\s\mep-frontend
2016-10-25T22:52:50.1260706Z > npm run build:dll
2016-10-25T22:52:50.1260706Z 
2016-10-25T22:52:51.6550710Z 
2016-10-25T22:52:51.6550710Z > react-boilerplate@3.2.1 build:dll C:\a\1\s\mep-frontend
2016-10-25T22:52:51.6560711Z > node ./internals/scripts/dependencies.js
2016-10-25T22:52:51.6560711Z 
2016-10-25T22:52:52.0310711Z Building the Webpack DLL...
2016-10-25T22:53:06.5460714Z chunk    {�[1m�[33m0�[39m�[22m} �[1m�[32mreactBoilerplateDeps.dll.js�[39m�[22m (reactBoilerplateDeps) 3.55 MB�[1m�[33m [entry]�[39m�[22m�[1m�[32m [rendered]�[39m�[22m
2016-10-25T22:53:06.5460714Z  [1181] �[1mdll reactBoilerplateDeps�[39m�[22m 12 bytes {�[1m�[33m0�[39m�[22m}�[1m�[32m [built]�[39m�[22m
2016-10-25T22:53:06.5460714Z      + 1181 hidden modules
2016-10-25T22:53:14.6156065Z react-boilerplate@3.2.1 C:\a\1\s\mep-frontend
2016-10-25T22:53:14.6156065Z react-boilerplate@3.2.1 C:\a\1\s\mep-frontend
2016-10-25T22:53:14.6156065Z +-- babel-cli@6.14.0 
2016-10-25T22:53:14.6156065Z | +-- babel-register@6.18.0 
2016-10-25T22:53:14.6156065Z | | +-- babel-core@6.18.0 
2016-10-25T22:53:14.6156065Z | | | `-- json5@0.5.0 
2016-10-25T22:53:14.6156065Z | | +-- home-or-tmp@2.0.0 
2016-10-25T22:53:14.6156065Z | | `-- source-map-support@0.4.5 

... skipping ~1500 lines of an impressive npm module graph ...

2016-10-25T22:53:14.7056050Z | `-- querystring@0.2.0 
2016-10-25T22:53:14.7056050Z `-- whatwg-fetch@1.0.0 
2016-10-25T22:53:14.7066049Z 
2016-10-25T22:53:14.7066049Z npm WARN optional Skipping failed optional dependency /chokidar/fsevents:
2016-10-25T22:53:14.7066049Z npm WARN notsup Not compatible with your operating system or architecture: fsevents@1.0.14
2016-10-25T22:53:14.7556061Z ##[section]Finishing: npm install
@zjrunner

This comment has been minimized.

Member

zjrunner commented Nov 3, 2016

We're just getting into the npm build-task space on my team so we have a bit of ramping up to do on costs and complications to optimize away some of the common pain points. One point I'd make about a comment earlier in this thread -- there was a question about custom agents and losing clean builds. From what I've seen your npm cache is maintained in the user profile so you can still do your clean builds and they basically come down to local unpack/copy (so you get to re-layout and stay clean). %appdata%\npm-cache on a windows machine.

If the network was your issue that would be closer than a remote cache. If the module graph is your problem, probably not a lot of help. We've begun looking into yarn as well, not sure if it makes any optimizations here or this is purely a function of the number of tiny files to move around on disk.

@damienpontifex

This comment has been minimized.

damienpontifex commented Dec 12, 2016

I have seen in the past about the progress display severely affecting performance. Is this also the case with npm install on VSTS?

Seems like npm set progress=false resolves some of these and has good performance improvements without any drastic changes (see http://biercoff.com/how-to-crazy-easily-speed-up-your-npm-install-speed/). Could this be integrated/tested with the VSTS npm task?

@jstawski

This comment has been minimized.

jstawski commented Feb 6, 2017

We are also looking into speeding this up on hosted VSTS agent. Using an internal cache sounds like a good idea.

@patrickmichalina

This comment has been minimized.

patrickmichalina commented Mar 8, 2017

Please! Heroku Speeds should be your target. They are orders of magnitude faster.

@andershermansen

This comment has been minimized.

andershermansen commented Mar 14, 2017

@bryanmacfarlane bryanmacfarlane modified the milestone: TFS.vNext (onPrem) Mar 29, 2017

@teemuy

This comment has been minimized.

teemuy commented Apr 4, 2017

The fastest implementation would be to reuse the resulting node_modules directory from a previous build if package.json has not changed (like commonly done using Docker). This is a much needed improvement!

@RDalziel

This comment has been minimized.

RDalziel commented Apr 4, 2017

I'd have to disagree @teemuy. That'd have to go on the assumption there's no min/major version matching going on. I find the majority of problems i get from Node based builds is minor increments breaking the codebase when they shouldn't!

@teemuy

This comment has been minimized.

teemuy commented Apr 5, 2017

I'm not sure what you are disagreeing about. My comment was about maximizing install performance, which is a pain point in our system where the install takes several minutes in each build.
The npm dependency version locking is a separate problem as I see it, and we will likely use npm shrinkwrap to solve that. However, that does nothing to improve the performance of course.

@patrickmichalina

This comment has been minimized.

patrickmichalina commented Apr 7, 2017

Abandoning builds on Azure for NPM projects was the way to go for me!

@mpseidel

This comment has been minimized.

mpseidel commented Apr 7, 2017

@zjrunner yarn definitely made a difference for us - I'd recommend using a linux agent though

@devdigital

This comment has been minimized.

devdigital commented Apr 7, 2017

We also had success with using a yarn offline mirror committed to the VCS, using the yarn install --ignore-engines option to solve some dependency errors.

@Saturate

This comment has been minimized.

Saturate commented Apr 19, 2017

Improving the speed would be a great plus for many. Right now for a simple solution it takes 3 minutes alone to install all the npm dependencies.

I wonder if using https://www.visualstudio.com/da-dk/docs/package/overview has any impact in the performance?

@teemuy

This comment has been minimized.

teemuy commented Apr 20, 2017

@Saturate Probably not a big help, unless you have slow network and have disabled caching. For us it's the npm install processing that takes time. We are using a local npm repository (Artifactory) but still our local vsts takes minutes to run the npm install stage at the start of each build.

@bryanmacfarlane bryanmacfarlane removed this from the TFS.vNext (onPrem) milestone Aug 17, 2017

@keithrob

This comment has been minimized.

Contributor

keithrob commented Aug 22, 2017

@RDalziel,
Are you still experiencing slowness with your NPM builds? I'm curious if you're still using Sinopia and whether or not you've tried using shrinkwrap or newer versions of NPM.
Keith

@RDalziel

This comment has been minimized.

RDalziel commented Aug 24, 2017

Reading back i mentioned installs taking around 10-15 minutes.

These days - it's around 2-4 minutes so i really haven't felt the need to improve on this further. Fairly happy with that speed at the moment. Although - SPA builds do generally feel like they take much longer than .NET builds - but i doubt that'll ever change due to the much larger number of files that are transferred during NPM Installs in relation to NuGet Restores

@keithrob

This comment has been minimized.

Contributor

keithrob commented Aug 24, 2017

@RDalziel ,
Thanks for the update. Closing this issue.
Keith

@keithrob keithrob closed this Aug 24, 2017

@jimmyheaddon

This comment has been minimized.

jimmyheaddon commented Oct 13, 2017

Hello, any update on changes in this area? This task alone accounts for ~80% of our build time, across several web sites using hosted VSTS agents. Typically our builds take between 8 and 10 minutes (different sites), so it's a substantial amount of time for CI builds :(

@keithrob

This comment has been minimized.

Contributor

keithrob commented Oct 13, 2017

@jimmyheaddon,
There are many things that can affect your build time so I would need to know more about which step in your build process is taking time. Are you using yarn or are you using npm v5+? Both of these do a better job of fetching packages in parallel.

Are you certain that one of your packages isn't doing something long-running in post-install script. Have you tried 'npm install --ignore-scripts".

@jimmyheaddon

This comment has been minimized.

jimmyheaddon commented Oct 13, 2017

Some more detail would have been helpful yes, sorry @keithrob!

We're using the latest LTS versions of Node and npm (6.11.4 (includes npm 3.10.10)), so that might explain a few things.

We're hosting our own npm packages using the VSTS package manager, so potentially that proxying behaviour is to blame, too?

I've given npm install --ignore-scripts a whirl, but that makes very little difference in our case. We're simply running npm install in a single npm build step, so it's isolated to just that case (barring pre and post steps, both removed and skipped).

I'll try a new version of npm to see if that helps, but we're a bit stuck with the LTS version :(

Thanks!

@keithrob

This comment has been minimized.

Contributor

keithrob commented Oct 13, 2017

@jimmyheaddon,
If you want to try a specific version of NPM, use the Node Tool Installer task and pick the version you want. Insert that task ahead of the "npm install" task. The tool installer will put the version of NPM you want on the head of $env:PATH and subsequent tasks will get it for free.

@jimmyheaddon

This comment has been minimized.

jimmyheaddon commented Oct 16, 2017

@keithrob that's worked brilliantly! We now install the current release of Node, run npm install, and then revert back to our LTS version of Node in just 2 minutes :)

@Jonatthu

This comment has been minimized.

Jonatthu commented Nov 28, 2018

So Azure will or not fix this problem?
Cache is really needed here

@phil-lgr

This comment has been minimized.

phil-lgr commented Dec 3, 2018

npm ci on my build is 50% of the total build time, a cache would be nice

@mitchdenny

This comment has been minimized.

Member

mitchdenny commented Dec 7, 2018

Hi folks - we are actively looking into the caching issue and hope to start work first up in the new year.

Package restore scenarios is right up there (probably #1) scenario that we want to solve for.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment