Updated Scheme and Sequence type to summary and optimized MLST to use thread pools #85
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
This is awesome @jennifertran, thanks. I'll try to find some time to look over this soon. |
apetkau
requested changes
Jul 19, 2019
There was a problem hiding this comment.
This code is great @jennifertran, thanks so much for taking a look into improving the scheduling of mlst 👍.
There's a few comments I have for you listed below, but overall this is great.
apetkau
requested changes
Jul 29, 2019
There was a problem hiding this comment.
Thanks @jennifertran for making the changes. I just have a few more comments.
apetkau
approved these changes
Jul 30, 2019
|
This is awesome. Thanks so much @jennifertran. It works great now, and is much much faster. 👍 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Based on Issue #81 and #79
Problem
Based on the issue #81 , since the
mlst --threadscommand wasn't doing what we expected to do, there was still a huge bottleneck within the code since each file has to go through mlst one at a time, it wasn't utilizing it resources enough.Solution
Partition the files evenly based on the number of threads available, instantiate the threads in a thread pool and assign an mlst instance to a thread.
Implementation
The implementation is pretty similar to how blast does it. It uses the
ThreadPoolExecutorclass to assign the thread to a process.Testing
Been checking the
timesystem instaramrto see if there was a significant performance change given 50 files and use thetopcommand to see if the CPU's were being utilized.