[MRG] add lca DBs as inputs to 'sourmash search' and 'gather' #533

ctb · 2018-08-18T02:07:39Z

This lets 'search' and 'gather' take LCA DBs as well as SBTs and signatures as input; adds 'multigather' command.

multigather is tested
search and gather on LCA DBs is tested
old style LCA DBs fail with good error messages
performance check of gather on genbank databases
rebuilt LCA databases are available ;)
documentation updated with latest LCA databases

Note that results on SBT and LCA databases for the shakya unassigned contigs are nearly but not completely identical. Trying to decide if this needs to be tracked down...

For merge purposes:

Is it mergeable?
make test Did it pass the tests?
make coverage Is the new code covered?
Did it change the command-line interface? Only additions are allowed
without a major version increment. Changing file formats also requires a
major version number increment.
Was a spellchecker run on the source code and documentation after
changes were made?

Command used to construct LCA DBs from SBT:

sourmash lca index genbank-genomes-taxonomy.2017.05.29.csv genbank-k31.lca.json.gz -k 31 --scaled=10000 -f --traverse-directory .sbt.genbank-k31 --split-identifiers

New LCA DB size:

-rw-r--r-- 1 ctb ged-lab 109M Dec 17 03:51 genbank-k21.lca.json.gz
-rw-r--r-- 1 ctb ged-lab 120M Dec 17 03:56 genbank-k31.lca.json.gz
-rw-r--r-- 1 ctb ged-lab 125M Dec 17 04:00 genbank-k51.lca.json.gz

Benchmarking:

        Command being timed: "sourmash gather shakya-unaligned-contigs.sig genbank-k31.lca.json.gz"
        User time (seconds): 111.97
        System time (seconds): 5.46
        Percent of CPU this job got: 99%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 1:57.87
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 5118232
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 0
        Minor (reclaiming a frame) page faults: 4105295
        Voluntary context switches: 1373
        Involuntary context switches: 133
        Swaps: 0
        File system inputs: 0
        File system outputs: 0
        Socket messages sent: 0
        Socket messages received: 0
        Signals delivered: 0
        Page size (bytes): 4096
        Exit status: 0

codecov-io · 2018-08-18T02:11:52Z

Codecov Report

Merging #533 into master will decrease coverage by 0.02%.
The diff coverage is 91.35%.

@@            Coverage Diff             @@
##           master     #533      +/-   ##
==========================================
- Coverage   88.56%   88.54%   -0.03%     
==========================================
  Files          25       25              
  Lines        3543     3754     +211     
  Branches       37       37              
==========================================
+ Hits         3138     3324     +186     
- Misses        405      430      +25

Impacted Files	Coverage Δ
sourmash/__main__.py	`95.83% <ø> (ø)`	⬆️
sourmash/search.py	`93.61% <100%> (+0.7%)`	⬆️
sourmash/sourmash_args.py	`95.63% <100%> (+0.31%)`	⬆️
sourmash/lca/command_rankinfo.py	`89.79% <100%> (+0.43%)`	⬆️
sourmash/lca/command_gather.py	`83.53% <100%> (-0.4%)`	⬇️
sourmash/commands.py	`89.22% <79.59%> (-1.3%)`	⬇️
sourmash/lca/command_index.py	`90.26% <92.22%> (+0.82%)`	⬆️
sourmash/lca/lca_utils.py	`94.67% <96.66%> (+0.19%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 21b400d...e708d3a. Read the comment docs.

* a trial refactoring of the lca db * save and load seem to basically work * got search & gather on LCA working in new framework * allow key error * keep identities with no lineage assigment * add multigather * Hyperlink DOIs to preferred resolver (#562) * fix divide by zero issue in MinHash.contained_by (#572) * fix divide by zero issue in contained_by * remove unused lineages and identifiers * update report output * majority of lca tests passing now! * fix gather? * ...all tests pass?

…gather_lca_db

luizirber · 2018-12-15T00:24:01Z

about the test coverage: we can add a .codecov.yml file to configure a bit better how codecov measures coverage.

…gather_lca_db

…e on lca index

luizirber · 2018-12-15T21:11:38Z

tests/test_lca.py

+        assert status != 0
+
+
+@utils.in_tempdir


hmm, this looks like tmpdir in pytest... https://github.com/dib-lab/sourmash/pull/557/files#diff-9402af88a46ea03a8087cfebdbdb762bR16

and RunnerContext looks pretty close to a pytext fixture too https://docs.pytest.org/en/latest/fixture.html#fixture

I figured they would ;). But I'm happy with the functionality for now and not inclined to fix it; can be adjusted in the future, yah?

yup, the tmpdir example I linked was a trial on using it instead of our current approach, good to see another option (and eventually refactor it out =P)

ctb · 2018-12-15T21:47:21Z

I'm working on regenerating databases. Once that's done and I do some performance benchmarking, I think I'd like to suggest merging this (since it's becoming big).

luizirber · 2018-12-15T22:25:32Z

sourmash/commands.py

+            query.minhash = query.minhash.downsample_scaled(args.scaled)
+
+        # empty?
+        if not query.minhash.get_mins():


Suggested change

if not query.minhash.get_mins():

if not len(query.minhash):

luizirber · 2018-12-15T22:26:12Z

I'll review in detail on Tue, but agree on merging after perf bench

ctb · 2018-12-17T14:25:22Z

Ready for review & merge! @luizirber the LCA DBs are here on the HPC:

[ctb@dev-intel14-phi sourmash-lca]$ pwd
/mnt/home/ctb/research/sourmash-lca
[ctb@dev-intel14-phi sourmash-lca]$ ls *.lca.json.gz
genbank-k21.lca.json.gz  genbank-k31.lca.json.gz  genbank-k51.lca.json.gz

ctb · 2018-12-17T16:30:53Z

Latest databases available here for download: https://osf.io/vk4fa/files/. Will update documentation appropriately.

sourmash/commands.py

sourmash/lca/command_gather.py

sourmash/lca/command_index.py

luizirber · 2018-12-19T01:16:23Z

sourmash/lca/command_index.py

+            try:
+                record_remnants.remove(ident)
+            except KeyError:
+                # @CTB


Hmm, is this a new pattern? If you pass an exception, you need to sign it to be made responsible at some point in the future? =]

luizirber

Some minor comments, but merge at will!

add lca DBs to 'sourmash search' and 'gather'

94f49dc

extract signatures once, at the cost of more memory

c2ea3eb

ctb mentioned this pull request Aug 23, 2018

[WIP] Refactor LCA database to contain all the information. #537

Merged

5 tasks

luizirber and others added 7 commits October 15, 2018 23:06

Merge branch 'master' into add/search_gather_lca_db

f7a2a54

Merge branch 'master' of github.com:dib-lab/sourmash into add/search_…

ac0ad9e

…gather_lca_db

flag older LCA DBs

8e39362

some cleanup

2224e95

restore py27 compat

af4b538

minor cleanup and commenting

b5ed476

ctb mentioned this pull request Dec 13, 2018

Suggestion to speed up gather for metagenomic samples #431

Closed

ctb added 4 commits December 13, 2018 09:40

add test for old-format LCA DB

16adb2d

bump lca_db version to 2

ae5d025

fix situation where json provides a list rather than a dictionary

a16c776

add test coverage

dd93341

ctb added 5 commits December 15, 2018 10:11

provide some more pythonic & succinct test code; add some more lca tests

12d0a5c

Merge branch 'master' of github.com:dib-lab/sourmash into add/search_…

69486a2

…gather_lca_db

some tests of sourmash search and gather on LCA dbs

560015f

produce a useful error message for -traverse-directory with no --forc…

2f185aa

…e on lca index

fix py27

eb51991

luizirber reviewed Dec 15, 2018

View reviewed changes

add a notify at beginning of load SBTs

6e668d6

use a faster method for checking for empty minhashes

944a0ed

ctb mentioned this pull request Dec 17, 2018

[MRG] Create an Index abstract base class #556

Merged

11 tasks

update links to latest databases

e708d3a

luizirber reviewed Dec 19, 2018

View reviewed changes

sourmash/commands.py Show resolved Hide resolved

luizirber reviewed Dec 19, 2018

View reviewed changes

sourmash/lca/command_gather.py Show resolved Hide resolved

luizirber reviewed Dec 19, 2018

View reviewed changes

sourmash/lca/command_index.py Show resolved Hide resolved

luizirber reviewed Dec 19, 2018

View reviewed changes

luizirber approved these changes Dec 19, 2018

View reviewed changes

luizirber changed the title ~~[WIP] add lca DBs as inputs to 'sourmash search' and 'gather'~~ add lca DBs as inputs to 'sourmash search' and 'gather' Dec 19, 2018

ctb mentioned this pull request Dec 19, 2018

check on lca gather performance #579

Closed

ctb changed the title ~~add lca DBs as inputs to 'sourmash search' and 'gather'~~ [MRG] add lca DBs as inputs to 'sourmash search' and 'gather' Dec 19, 2018

ctb merged commit 954d729 into master Dec 19, 2018

ctb deleted the add/search_gather_lca_db branch December 19, 2018 04:23

This was referenced Dec 19, 2018

Implement more/better revindex functionality on top of LCA databases. #581

Closed

Layer phylogenetic options on to sourmash lca gather? #583

Closed

Minor fix to #533 #584

Merged

luizirber mentioned this pull request Jan 4, 2019

[WIP] add a 'lca search' command #377

Closed

5 tasks

ctb mentioned this pull request Jan 5, 2019

[WIP] explore reverse indexing in LCA v2 databases #604

Closed

9 tasks

ctb mentioned this pull request Jan 17, 2019

discuss (lack of) prepared protein search database #637

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[MRG] add lca DBs as inputs to 'sourmash search' and 'gather' #533

[MRG] add lca DBs as inputs to 'sourmash search' and 'gather' #533

ctb commented Aug 18, 2018 •

edited

codecov-io commented Aug 18, 2018 •

edited by codecov bot

luizirber commented Dec 15, 2018

luizirber Dec 15, 2018

ctb Dec 15, 2018

luizirber Dec 15, 2018

ctb commented Dec 15, 2018

luizirber Dec 15, 2018

luizirber commented Dec 15, 2018

ctb commented Dec 17, 2018

ctb commented Dec 17, 2018

luizirber Dec 19, 2018

luizirber left a comment

[MRG] add lca DBs as inputs to 'sourmash search' and 'gather' #533

[MRG] add lca DBs as inputs to 'sourmash search' and 'gather' #533

Conversation

ctb commented Aug 18, 2018 • edited

Command used to construct LCA DBs from SBT:

New LCA DB size:

Benchmarking:

codecov-io commented Aug 18, 2018 • edited by codecov bot

Codecov Report

luizirber commented Dec 15, 2018

luizirber Dec 15, 2018

Choose a reason for hiding this comment

ctb Dec 15, 2018

Choose a reason for hiding this comment

luizirber Dec 15, 2018

Choose a reason for hiding this comment

ctb commented Dec 15, 2018

luizirber Dec 15, 2018

Choose a reason for hiding this comment

luizirber commented Dec 15, 2018

ctb commented Dec 17, 2018

ctb commented Dec 17, 2018

luizirber Dec 19, 2018

Choose a reason for hiding this comment

luizirber left a comment

Choose a reason for hiding this comment

ctb commented Aug 18, 2018 •

edited

codecov-io commented Aug 18, 2018 •

edited by codecov bot