NPM Install Speed

[RDalziel]

Is there any way with the current build task to increase the speed of NPM installs?

I've seen methods of caching being referenced around the web, but unsure that's possible with the current NPM Build Task setup.

NPM installs are usually the main cause of slow builds on front-end builds at the moment, any way of speeding them up would be a great boost!

[bryanmacfarlane]

We are considering installing npm read through caching servers in the hosted build pool. I assume you are referencing hosted build (and not your own agent).

The other immediate way to speed up npm step considerably is to install your own agent on an azure VM ( http://aka.ms/tfbuild ) - that's a persistent agent so you get incremental. modules would be there from build to build and would only pull new one when you use a new one/version.

[RDalziel]

Cheers Bryan, i'm actually On Premise here. Are you basically referring to not doing a clean build when you refer to incremental modules? I'm not a huge fan of moving away from clean builds as i've had problems in the past with lingering modules masking errors. There's also the problem of artifacts lingering between builds.

Have you got a solution worked for these without writing custom steps to clean up folders?

[bryanmacfarlane]

Well, we're just calling npm so it's between your box, your network and the npm services.

That said, there is a solution. You can install a read through npm cache. That will allow you to do clean builds and still get fast npm installs.

Something like this is what we were looking at for our hosted pool:

https://www.npmjs.com/package/sinopia

You could install it right on the agent or on another box local to your network. You would point your npm config on the agent at that read through cache server.

[RDalziel]

Cheers Bryan !

[bryanmacfarlane]

I also just found this which will be simpler and alleviate the problem with an on prem agent:

https://www.npmjs.com/package/npm-cache

[RDalziel]

Cheers again Bryan, i did have a go with Sinopia but had some trouble getting it working reliably. Their documentation for installing as a windows service was a bit date also.

Will give npm-cache a go!

[RDalziel]

FYI Bryan. I had a look at npm-cache - It looks great but it's not a viable option on windows boxes for now. Apparently it's getting worked on though!

[RDalziel]

Again - FYI. I've ended up using Sinopia as we wanted to start hosting internal NPM packages. It's taken a 14 min build down to around 11 minutes.

[bryanmacfarlane]

Awesome

[mpseidel]

+1 for caching npm modules on hosted build agents. Services like codeship do this and it makes the whole experience more fun :)

[marcwittke]

We are considering installing npm read through caching servers in the hosted build pool.

@bryanmacfarlane did you finalize the consideration? This issue was closed without any feedback regarding this. My npm install of a react boilerplate js application takes about four minutes on the hosted build agent :(

[bryanmacfarlane]

assigning to zack for thoughts. Last time we looked at the bottleneck, npm local I/O (updating local cache folders) was dominating the time on hosted agents. Since then though, we've sped up I/O with D2+SSD. In addition, resolving the graph of versions might also dominate the time before it ever gets to pulling them from the cache server. So, it's unclear whether it would help much on perf. Zack drives the packaging team so he can talk to it.

[marcwittke]

@zjrunner for what it's worth, here my (shortened) log. And yes, @bryanmacfarlane might be right with his thoughts about the domination of graph resolution...

2016-10-25T22:49:58.5283037Z [command]C:\Program Files\nodejs\npm.cmd install
2016-10-25T22:50:27.1975224Z npm WARN deprecated tough-cookie@2.2.2: ReDoS vulnerability parsing Set-Cookie https://nodesecurity.io/advisories/130
2016-10-25T22:50:47.2245925Z npm WARN deprecated cross-spawn-async@2.2.4: cross-spawn no longer requires a build toolchain, use it instead!
2016-10-25T22:52:30.6282644Z npm WARN prefer global colorguard@1.2.0 should be installed with -g
2016-10-25T22:52:36.8852554Z 
2016-10-25T22:52:36.8852554Z > pre-commit@1.1.3 install C:\a\1\s\mep-frontend\node_modules\pre-commit
2016-10-25T22:52:36.8852554Z > node install.js
2016-10-25T22:52:36.8852554Z 
2016-10-25T22:52:40.8275357Z 
2016-10-25T22:52:40.8275357Z > spawn-sync@1.0.13 postinstall C:\a\1\s\mep-frontend\node_modules\spawn-sync
2016-10-25T22:52:40.8275357Z > node postinstall
2016-10-25T22:52:40.8275357Z 
2016-10-25T22:52:41.4805353Z 
2016-10-25T22:52:41.4805353Z > ngrok@2.2.2 postinstall C:\a\1\s\mep-frontend\node_modules\ngrok
2016-10-25T22:52:41.4805353Z > node ./postinstall.js
2016-10-25T22:52:41.4805353Z 
2016-10-25T22:52:42.2975440Z ngrok - downloading binary https://bin.equinox.io/a/chRZWPptE7w/ngrok-2.1.1-windows-amd64.zip ...
2016-10-25T22:52:42.6163623Z ngrok - binary downloaded...
2016-10-25T22:52:42.8123618Z ngrok - binary unpacked.
2016-10-25T22:52:42.8943627Z 
2016-10-25T22:52:42.8943627Z > lint-staged@3.0.2 postinstall C:\a\1\s\mep-frontend\node_modules\lint-staged
2016-10-25T22:52:42.8943627Z > echo "lint-staged installed! Do not forget to configure it. See https://github.com/okonet/lint-staged/blob/master/README.md" && exit 0
2016-10-25T22:52:42.8953623Z 
2016-10-25T22:52:42.9013611Z "lint-staged installed! Do not forget to configure it. See https://github.com/okonet/lint-staged/blob/master/README.md" 
2016-10-25T22:52:42.9733639Z 
2016-10-25T22:52:42.9733639Z > pngquant-bin@3.1.1 postinstall C:\a\1\s\mep-frontend\node_modules\pngquant-bin
2016-10-25T22:52:42.9733639Z > node lib/install.js
2016-10-25T22:52:42.9733639Z 
2016-10-25T22:52:44.1292582Z   √ pngquant pre-build test passed successfully
2016-10-25T22:52:44.1412610Z 
2016-10-25T22:52:44.1422585Z > optipng-bin@3.1.2 postinstall C:\a\1\s\mep-frontend\node_modules\optipng-bin
2016-10-25T22:52:44.1422585Z > node lib/install.js
2016-10-25T22:52:44.1422585Z 
2016-10-25T22:52:45.0444527Z   √ optipng pre-build test passed successfully
2016-10-25T22:52:45.0564530Z 
2016-10-25T22:52:45.0564530Z > jpegtran-bin@3.1.0 postinstall C:\a\1\s\mep-frontend\node_modules\jpegtran-bin
2016-10-25T22:52:45.0564530Z > node lib/install.js
2016-10-25T22:52:45.0564530Z 
2016-10-25T22:52:46.0140709Z   √ jpegtran pre-build test passed successfully
2016-10-25T22:52:46.0270705Z 
2016-10-25T22:52:46.0270705Z > gifsicle@3.0.4 postinstall C:\a\1\s\mep-frontend\node_modules\gifsicle
2016-10-25T22:52:46.0270705Z > node lib/install.js
2016-10-25T22:52:46.0270705Z 
2016-10-25T22:52:46.9020711Z   √ gifsicle pre-build test passed successfully
2016-10-25T22:52:47.0660710Z 
2016-10-25T22:52:47.0660710Z > react-boilerplate@3.2.1 preinstall C:\a\1\s\mep-frontend
2016-10-25T22:52:47.0660710Z > npm run npmcheckversion
2016-10-25T22:52:47.0660710Z 
2016-10-25T22:52:48.7030713Z 
2016-10-25T22:52:48.7040709Z > react-boilerplate@3.2.1 npmcheckversion C:\a\1\s\mep-frontend
2016-10-25T22:52:48.7040709Z > node ./internals/scripts/npmcheckversion.js
2016-10-25T22:52:48.7040709Z 
2016-10-25T22:52:50.1260706Z 
2016-10-25T22:52:50.1260706Z > react-boilerplate@3.2.1 postinstall C:\a\1\s\mep-frontend
2016-10-25T22:52:50.1260706Z > npm run build:dll
2016-10-25T22:52:50.1260706Z 
2016-10-25T22:52:51.6550710Z 
2016-10-25T22:52:51.6550710Z > react-boilerplate@3.2.1 build:dll C:\a\1\s\mep-frontend
2016-10-25T22:52:51.6560711Z > node ./internals/scripts/dependencies.js
2016-10-25T22:52:51.6560711Z 
2016-10-25T22:52:52.0310711Z Building the Webpack DLL...
2016-10-25T22:53:06.5460714Z chunk    {�[1m�[33m0�[39m�[22m} �[1m�[32mreactBoilerplateDeps.dll.js�[39m�[22m (reactBoilerplateDeps) 3.55 MB�[1m�[33m [entry]�[39m�[22m�[1m�[32m [rendered]�[39m�[22m
2016-10-25T22:53:06.5460714Z  [1181] �[1mdll reactBoilerplateDeps�[39m�[22m 12 bytes {�[1m�[33m0�[39m�[22m}�[1m�[32m [built]�[39m�[22m
2016-10-25T22:53:06.5460714Z      + 1181 hidden modules
2016-10-25T22:53:14.6156065Z react-boilerplate@3.2.1 C:\a\1\s\mep-frontend
2016-10-25T22:53:14.6156065Z react-boilerplate@3.2.1 C:\a\1\s\mep-frontend
2016-10-25T22:53:14.6156065Z +-- babel-cli@6.14.0 
2016-10-25T22:53:14.6156065Z | +-- babel-register@6.18.0 
2016-10-25T22:53:14.6156065Z | | +-- babel-core@6.18.0 
2016-10-25T22:53:14.6156065Z | | | `-- json5@0.5.0 
2016-10-25T22:53:14.6156065Z | | +-- home-or-tmp@2.0.0 
2016-10-25T22:53:14.6156065Z | | `-- source-map-support@0.4.5 

... skipping ~1500 lines of an impressive npm module graph ...

2016-10-25T22:53:14.7056050Z | `-- querystring@0.2.0 
2016-10-25T22:53:14.7056050Z `-- whatwg-fetch@1.0.0 
2016-10-25T22:53:14.7066049Z 
2016-10-25T22:53:14.7066049Z npm WARN optional Skipping failed optional dependency /chokidar/fsevents:
2016-10-25T22:53:14.7066049Z npm WARN notsup Not compatible with your operating system or architecture: fsevents@1.0.14
2016-10-25T22:53:14.7556061Z ##[section]Finishing: npm install

[zjrunner]

We're just getting into the npm build-task space on my team so we have a bit of ramping up to do on costs and complications to optimize away some of the common pain points. One point I'd make about a comment earlier in this thread -- there was a question about custom agents and losing clean builds. From what I've seen your npm cache is maintained in the user profile so you can still do your clean builds and they basically come down to local unpack/copy (so you get to re-layout and stay clean). %appdata%\npm-cache on a windows machine.

If the network was your issue that would be closer than a remote cache. If the module graph is your problem, probably not a lot of help. We've begun looking into yarn as well, not sure if it makes any optimizations here or this is purely a function of the number of tiny files to move around on disk.

[damienpontifex]

I have seen in the past about the progress display severely affecting performance. Is this also the case with npm install on VSTS?

Seems like npm set progress=false resolves some of these and has good performance improvements without any drastic changes (see http://biercoff.com/how-to-crazy-easily-speed-up-your-npm-install-speed/). Could this be integrated/tested with the VSTS npm task?

[jstawski]

We are also looking into speeding this up on hosted VSTS agent. Using an internal cache sounds like a good idea.

[patrickmichalina]

Please! Heroku Speeds should be your target. They are orders of magnitude faster.

[andershermansen]

There is a suggestion on user voice to use a cache to improve install time
https://visualstudio.uservoice.com/forums/330519-team-services/suggestions/11063364--tfsbuild-vnext-add-a-npm-cache-mirror-on-hosted

[teemuy]

The fastest implementation would be to reuse the resulting node_modules directory from a previous build if package.json has not changed (like commonly done using Docker). This is a much needed improvement!

[RDalziel]

I'd have to disagree @teemuy. That'd have to go on the assumption there's no min/major version matching going on. I find the majority of problems i get from Node based builds is minor increments breaking the codebase when they shouldn't!

[teemuy]

I'm not sure what you are disagreeing about. My comment was about maximizing install performance, which is a pain point in our system where the install takes several minutes in each build.
The npm dependency version locking is a separate problem as I see it, and we will likely use npm shrinkwrap to solve that. However, that does nothing to improve the performance of course.

[patrickmichalina]

Abandoning builds on Azure for NPM projects was the way to go for me!

[mpseidel]

@zjrunner yarn definitely made a difference for us - I'd recommend using a linux agent though

[devdigital]

We also had success with using a yarn offline mirror committed to the VCS, using the yarn install --ignore-engines option to solve some dependency errors.

[Saturate]

Improving the speed would be a great plus for many. Right now for a simple solution it takes 3 minutes alone to install all the npm dependencies.

I wonder if using https://www.visualstudio.com/da-dk/docs/package/overview has any impact in the performance?

[teemuy]

@Saturate Probably not a big help, unless you have slow network and have disabled caching. For us it's the npm install processing that takes time. We are using a local npm repository (Artifactory) but still our local vsts takes minutes to run the npm install stage at the start of each build.

[keithrob]

@RDalziel,
Are you still experiencing slowness with your NPM builds? I'm curious if you're still using Sinopia and whether or not you've tried using shrinkwrap or newer versions of NPM.
Keith

[RDalziel]

Reading back i mentioned installs taking around 10-15 minutes.

These days - it's around 2-4 minutes so i really haven't felt the need to improve on this further. Fairly happy with that speed at the moment. Although - SPA builds do generally feel like they take much longer than .NET builds - but i doubt that'll ever change due to the much larger number of files that are transferred during NPM Installs in relation to NuGet Restores

[keithrob]

@RDalziel ,
Thanks for the update. Closing this issue.
Keith

[jimmyheaddon]

Hello, any update on changes in this area? This task alone accounts for ~80% of our build time, across several web sites using hosted VSTS agents. Typically our builds take between 8 and 10 minutes (different sites), so it's a substantial amount of time for CI builds :(

[keithrob]

@jimmyheaddon,
There are many things that can affect your build time so I would need to know more about which step in your build process is taking time. Are you using yarn or are you using npm v5+? Both of these do a better job of fetching packages in parallel.

Are you certain that one of your packages isn't doing something long-running in post-install script. Have you tried 'npm install --ignore-scripts".

[jimmyheaddon]

Some more detail would have been helpful yes, sorry @keithrob!

We're using the latest LTS versions of Node and npm (6.11.4 (includes npm 3.10.10)), so that might explain a few things.

We're hosting our own npm packages using the VSTS package manager, so potentially that proxying behaviour is to blame, too?

I've given npm install --ignore-scripts a whirl, but that makes very little difference in our case. We're simply running npm install in a single npm build step, so it's isolated to just that case (barring pre and post steps, both removed and skipped).

I'll try a new version of npm to see if that helps, but we're a bit stuck with the LTS version :(

Thanks!

[keithrob]

@jimmyheaddon,
If you want to try a specific version of NPM, use the Node Tool Installer task and pick the version you want. Insert that task ahead of the "npm install" task. The tool installer will put the version of NPM you want on the head of $env:PATH and subsequent tasks will get it for free.

[jimmyheaddon]

@keithrob that's worked brilliantly! We now install the current release of Node, run npm install, and then revert back to our LTS version of Node in just 2 minutes :)

[Jonatthu]

So Azure will or not fix this problem?
Cache is really needed here

[phil-lgr]

npm ci on my build is 50% of the total build time, a cache would be nice

[mitchdenny]

Hi folks - we are actively looking into the caching issue and hope to start work first up in the new year.

Package restore scenarios is right up there (probably #1) scenario that we want to solve for.

[phil-lgr]

I guess this is the ticket we can follow right https://developercommunity.visualstudio.com/idea/365799/improve-hosted-build-agent-performance-with-build.html?utm_source=t.co&utm_medium=referral