Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Ground truth computation is too slow for realistic data set sizes #30

Closed
3 tasks done
dkoslicki opened this issue Apr 6, 2020 · 1 comment
Closed
3 tasks done

Ground truth computation is too slow for realistic data set sizes #30

dkoslicki opened this issue Apr 6, 2020 · 1 comment
Assignees
@dkoslicki
Labels
priority

Comments

@dkoslicki
Copy link
Owner

dkoslicki commented Apr 6, 2020 •
edited
Loading

In the current CMash/GroundTruth.py, since it's using a naive python set calculation, this is much too slow to get results in a reasonable amount of time on larger data sets. Will need to switch to using KMC many times over to calculate the actual ground truth.

Work being done on groundtruth branch.

  • Use KMC to count k-mers of all training database genomes for all k-mer sizes (and return total number of distinct k-mers)
  • Use KMC to count k-mer of the query genome for all k-mer sizes
  • Calculate containment index via kmc_tools intersect and divide by total number of distinct training database k-mers
@dkoslicki dkoslicki self-assigned this Apr 6, 2020
dkoslicki added a commit that referenced this issue Apr 6, 2020
@dkoslicki
change to smaller default sizes for faster local testing #30
2816825
dkoslicki added a commit that referenced this issue Apr 6, 2020
@dkoslicki
copy and rename class to start working on KMC implementation #30
23271a4
dkoslicki added a commit that referenced this issue Apr 6, 2020
@dkoslicki
add method to count kmers with KMC #30
678022c
dkoslicki added a commit that referenced this issue Apr 6, 2020
@dkoslicki
add method to count distinct number of unique k-mers #30
f3a2650
dkoslicki added a commit that referenced this issue Apr 6, 2020
@dkoslicki
return kmc intersection count #30
256147d
dkoslicki added a commit that referenced this issue Apr 6, 2020
@dkoslicki
well that took too long to figure out: KMC doesn't play well with par…
3d2946a 
…allelization since it doesn't give its temp files unique names. Got around this by using RAM only mode. Added ability to compute all training kmers via KMC. #30
dkoslicki added a commit that referenced this issue Apr 6, 2020
@dkoslicki
added TODO's for reminder, as need to move on to other things atm #30
61d9c04
dkoslicki added a commit that referenced this issue Apr 7, 2020
@dkoslicki
completed implementation of KMC version of ground truth, but differen…
592e85f 
…ces observed between it and the pure python version. Investigating now #30
dkoslicki added a commit that referenced this issue Apr 7, 2020
@dkoslicki
found the problem: was in the old solution, not in the new one :P, re…
6025e38 
…sults agree now. Refactoring now #30
dkoslicki added a commit that referenced this issue Apr 7, 2020
@dkoslicki
tested, refactored, and implemented. #30
a72b337
dkoslicki added a commit that referenced this issue Apr 7, 2020
@dkoslicki
Merge remote-tracking branch 'origin/groundtruth' into tests #30
493c880
dkoslicki added a commit that referenced this issue Apr 7, 2020
@dkoslicki
Merge remote-tracking branch 'origin/tests' #30
e40d3b9
dkoslicki added a commit that referenced this issue Apr 7, 2020
@dkoslicki
Merge remote-tracking branch 'origin/tests' #30
c9f99b1
@dkoslicki
Copy link
Owner Author

merged into master and completed. Closing. @ShaopengLiu1 should be ready for you to use

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@dkoslicki dkoslicki

Labels
priority
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@dkoslicki