Parallel module install #3907
Parallel module install #3907
Conversation
- use binding where possible, ensures no changes and save allocations - fix indentation on protect block / localization dynvar blocks - remove unnecessary intermediate variables - rename $lock to $repo-lock, to prevent confusion with $!lock - remove unnecessary "is copy" in signatures - remove unneccssary .IO on already IO::Path objects - only sort %provides once, work using keys from there on - replace .values[0] with .values.head to prevent unnecessary caching - some code re-ordering to prevent unnecessary work if next fires
Basically run the precomp in a start block, collect the promises and await them. Results for installation of core modules is meagre. Installing something like App::Mi6 goes from 18 -> 15 seconds wallclock. Which is... disappointing. I wonder why this doesn't improve more.
| ?? $repo-prefix ~ $source.relative($.prefix) | ||
| !! $source; | ||
|
|
||
| @promises.push: start { |
There was a problem hiding this comment.
Spawning a process for every module listed under provides is going to reach diminishing returns pretty quickly.
|
Spawning a process for every module listed under provides is going to reach
diminishing returns pretty quickly.
The problem is that we don't know the dependencies between the modules in a
dist. Ordinarily you'd walk the tree bottom up and keep a couple processes
busy that way, but we don't know how the nodes connect to a tree. Doesn't help
if you spawn X processes but they all are waiting for the same dependency,
while there may be other leaf modules available.
So the next step seems to be a large and hard one involving those precomp
processes communicating their status (compiling/waiting for another precomp)
to the main process so the latter can throttle accordingly. And even then you
need even more communication as finishing one dependency may suddenly unblock
a whole horde of precomp processes.
|
|
I did not follow the whole story of parallelizing precomps and may miss a thing. But how hard would it be to make it a task queue based processing? So that limited number of worker threads are pre-spawned and each would pick a task from a channel? |
Precompilation currently spawns a new process and thus requires IPC |
|
Closing this now, as it appears we should be able to solve slow installation better with making sure we only precomp a module once in the installation process. |
An attempt at parallelizing module installation, now that a blocking race condition has been fixed by nine.