worker optimization #93
Conversation
| @@ -19,6 +19,7 @@ const fs = require('fs'); | |||
| const path = require('path'); | |||
|
|
|||
| const availableCpus = require('os').cpus().length - 1; | |||
| const CHUNK_SIZE = 50; | |||
There was a problem hiding this comment.
Should I make this a config option?
There was a problem hiding this comment.
I'm fine with this being a constant for the time being.
|
Sweet, thank you! Will merge your PRs tomorrow.... maybe this also makes it a bit easier to solve #87. |
|
Not sure why the build failed, tests pass locally. Any way to kick off another build? |
|
btw. I'd recommend switching to node-worker-farm: https://github.com/rvagg/node-worker-farm |
|
I just restarted it. Sometimes testing the Babel option fails, no idea why. |
|
Thanks! @cpojer Is it OK if we merge this PR now and we take on the work to use node-worker-farm later? I'm willing to investigate that option (looks really promising) but meanwhile I need some moar perf! :) |
|
Released as v0.3.14. Thank you! |
Change the worker management to be dynamic rather than static. Instead of dividing all files among the available number of processes, we now send chunks of work to each free process, and when a process finishes its chunk of work it gets another chunk. This leads to overall much better perf for a large number of files with non-uniform processing times.