New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve performance of js-config-webpack-plugin #24

Closed
ernscht opened this Issue Dec 12, 2018 · 6 comments

Comments

Projects
None yet
2 participants
@ernscht
Copy link
Member

ernscht commented Dec 12, 2018

I'm submitting a...


[x] Performance improvement

Current & Expected behavior

Without the thread-loader, compilation takes three to four times less time on changes. We could remove it.

What is the motivation / use case for changing the behavior?

At least in the nitro example it works much faster without the thread-loader.

The "thread-loader" could be useful for a lot of files or with a different configuration (prewarming). Maybe even a configuration option makes sense?

@ernscht ernscht added the enhancement label Dec 12, 2018

@jantimon

This comment has been minimized.

Copy link
Contributor

jantimon commented Dec 17, 2018

I wrote a small node file to measure the behaviour.

The most important part are incremental build times.
The tool does the following:

measure-flow

Result:

current = ts-loader
latest = cache-loader + thread-loader + ts-loader

⏰ DevServer Current { 
  lowest: 0.383,
  highest: 0.728,
  average: 0.47200000000000003,
  median: 0.478 
} 
⏰ DevServer Latest { 
  lowest: 1.1280000000000001,
  highest: 1.46,
  average: 1.2519310344827583,
  median: 1.232 
}

The second test is running multiple full builds without dev server on the same machine:

current = ts-loader
latest = cache-loader + thread-loader + ts-loader

⏰ Build Current - with file cache { 
  lowest: 4.66,
  highest: 6.116,
  average: 5.0339,
  median: 4.8629999999999995 
} 
⏰ Build Latest - with file cache { 
  lowest: 3.545,
  highest: 4.473,
  average: 3.8223000000000007,
  median: 3.8135 
} 

The third test is running multiple full builds on a clean system (similar to a CI build):

current = ts-loader
latest = cache-loader + thread-loader + ts-loader

⏰ Build Current - clean build { 
  lowest: 11.971,
  highest: 13.212,
  average: 12.3424,
  median: 12.3425 
} 
⏰ Build Latest - clean build { 
  lowest: 12.226,
  highest: 16.078,
  average: 14.0427,
  median: 14.6725
} 

Results for 1000 files instead of 100:

current = ts-loader
latest = cache-loader + thread-loader + ts-loader

DevServer Current {
  lowest: 0.996,
  highest: 2.259,
  average: 1.3956551724137927,
  median: 1.318
} 

 DevServer Latest { 
  lowest: 3.144,
  highest: 4.788,
  average: 3.522103448275862,
  median: 3.372 
} 
@jantimon

This comment has been minimized.

Copy link
Contributor

jantimon commented Dec 18, 2018

Just found out the reason for the performance delay:

from: https://github.com/webpack-contrib/thread-loader

// timeout for killing the worker processes when idle
// defaults to 500 (ms)
 // can be set to Infinity for watching builds to keep workers alive
poolTimeout: 2000,

So by default after 500ms all thread-loaders will be killed and have to be restarted for the next build.

Recompile times for 100 and 1000 components in seconds:

Thread loader with default poolTimeout

  [ 1.313, 1.314, 1.297, 1.182, 1.202, 1.14, 1.095, 1.132, 1.166 ]
  [ 4.595, 3.19, 3.433, 3.03, 3.085, 2.851, 2.901, 2.983, 2.657 ]

No thread loader

  [ 0.542, 0.482, 0.498, 0.479, 0.463, 0.441, 0.365, 0.473, 0.473 ]
  [ 2.192,  3.283, 1.77, 1.358, 1.299,  1.067,  1.09, 1.52, 1.391 ]

Thread loader with high poolTimeout

 [ 0.476, 0.485, 0.508, 0.477, 0.467, 0.423, 0.441, 0.454, 0.455 ]
 [ 1.546, 1.312, 1.544, 1.411, 1.255, 1.4, 1.396, 1.216, 1.28 ]

So it seems like no-thread loader is almost as fast as high poolTimeout (we would have to investigate that in detail)

@jantimon

This comment has been minimized.

Copy link
Contributor

jantimon commented Dec 18, 2018

Same result on travis:

Removing thread-loader improved the build speed by 65% for ts-loader and 65% for babel-loader:
https://travis-ci.org/namics/webpack-config-plugins/builds/469655578

babel 100 components

  0.587s faster than the latest released npm version
 35.04% of the latest released npm version build duration

typescript 100 components

 0.776s faster than the latest released npm version
 35.87% of the latest released npm version build duration

babel 1000 components

  1.058s faster than the latest released npm version
  46.54% of the latest released npm version build duration

typescript 1000 component

  1.719s faster than the latest released npm version
  43.93% of the latest released npm version build duration
@jantimon

This comment has been minimized.

Copy link
Contributor

jantimon commented Dec 21, 2018

I could fix the thread-loader 🎉

The results look promising (comparing version thread-loader 1.2.0 and thread-loader 2.1.0):

Time difference for 100 components:

https://travis-ci.org/namics/webpack-config-plugins/jobs/470988476 (typescript)
https://travis-ci.org/namics/webpack-config-plugins/jobs/470988477 (babel)

1.171 s -> 378ms

average: '0.79s faster than the latest released npm version',
averageRelative: '32% of the latest released npm version build duration',
median: '0.79s faster than the latest released npm version',
medianRelative: '32% of the latest released npm version build duration'

Time difference for 1000 components:

https://travis-ci.org/namics/webpack-config-plugins/jobs/470988478 (typescript)
https://travis-ci.org/namics/webpack-config-plugins/jobs/470988479 (babel)

3.071s -> 1.34s

average: '1.72s faster than the latest released npm version',
averageRelative: '44% of the latest released npm version build duration',
median: '1.74s faster than the latest released npm version',
medianRelative: '42% of the latest released npm version build duration'

@jantimon jantimon referenced this issue Dec 21, 2018

Merged

Fix/thread loader respawn #33

1 of 1 task complete
@jantimon

This comment has been minimized.

Copy link
Contributor

jantimon commented Dec 21, 2018

Fixed by #33

@jantimon

This comment has been minimized.

Copy link
Contributor

jantimon commented Dec 21, 2018

Released in 1.2.0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment