[IMP] Add multiprocessing option for aggregation #16
Conversation
* Move repo aggregation into a helper method * Add option for the amount of processes to use when aggregating * Use multiprocessing pool for aggregation if aforementioned option is enabled
|
I also think it would be interesting to have this option, as maintaining a large amount of repositories highly increases the execution time (even though some of them are only a few KBs). |
|
I'm personally fine with the principle of this PR. One must examine why travis is red though. |
|
I've submitted some changes to the main PR branch (LasLabs#1), feel free to comment or suggest any change. Cheers! |
|
Done! No more Travis errors 👌 |
1 similar comment
|
We would love to see this feature being added to the main branch, would you mind if I open a new PR including our latest changes? |
|
Hi, I had not noticed it's green now. I'm personally fine with this approach. Feel free to open another PR. cc/ @lmignon |
|
BTW, I just notice LasLabs#1 is much more complex than this one. What was not working with this PR? |
|
That PR includes tests and avoids the working directory lock -which prevented using a multithreading approach instead of a multiprocessing one-, among other minor fixes. I will open a proper PR when I'm at home, thank you! |
|
Superseded by #21 |
4 participants
This PR adds a multiprocessing pool for the repo aggregation if desired.
I am still testing this, but wanted to submit the PR early to make sure there's not somewhere else you'd rather I put this. The reasoning I'm thinking that might be the case is because it seems the method I modified is untested (at least directly), so I'm thinking it's not important.