[MRG] replace khmer.Nodegraph with rust nodegraph

[luizirber]

Main benefits:

• Read/write Nodegraph to/from buffers (no intermediate temporary files)
• Remove dep on khmer (but can still convert this Nodegraph into a khmer.Nodegraph)

Checklist

• Is it mergeable?
• make test Did it pass the tests?
• make coverage Is the new code covered?
• Did it change the command-line interface? Only additions are allowed
  without a major version increment. Changing file formats also requires a
  major version number increment.
• Was a spellchecker run on the source code and documentation after
  changes were made?

[codecov]

Codecov Report

Merging #799 into master will decrease coverage by 11.87%.
The diff coverage is 84.21%.

@@             Coverage Diff             @@
##           master     #799       +/-   ##
===========================================
- Coverage   91.61%   79.74%   -11.88%     
===========================================
  Files          71       97       +26     
  Lines        5071     7706     +2635     
===========================================
+ Hits         4646     6145     +1499     
- Misses        425     1561     +1136     

Flag	Coverage Δ
#rusttests	53.79% <77.20%> (?)

Impacted Files	Coverage Δ
sourmash/commands.py	86.23% <ø> (+1.59%)	⬆️
src/core/src/ffi/nodegraph.rs	0.00% <0.00%> (ø)
src/core/src/ffi/signature.rs	0.00% <0.00%> (ø)
src/core/src/index/sbt/mod.rs	74.51% <ø> (ø)
sourmash/nodegraph.py	93.75% <93.75%> (ø)
src/core/src/sketch/nodegraph.rs	84.74% <93.78%> (ø)
sourmash/sbtmh.py	92.72% <95.00%> (-0.13%)	⬇️
sourmash/cli/info.py	100.00% <100.00%> (ø)
sourmash/sbt.py	87.80% <100.00%> (+1.97%)	⬆️
sourmash/sbt_storage.py	95.69% <100.00%> (+0.19%)	⬆️
... and 35 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a42508f...eeed865. Read the comment docs.

[olgabot]

So much force-pushing ... ;) exciting though! Any word on how this decreases the footprint of SBTs?

[luizirber]

So much force-pushing ... ;) exciting though! Any word on how this decreases the footprint of SBTs?

Hmm, this doesn't change much in that front... Main issue with SBTs is keeping internal nodes in memory, and sometimes you want to keep them, sometimes you don't. For search you can unload it once it's processed, and #784 does that.

For gather it is more complicated, because you will probably check the top nodes in the tree again, but probably not so much of the ones near the leaves. We don't have a good way to deal with that yet...

Note that calling node.unload() should be safe, because the node knows how to reload the data from the storage. It will take longer (because you need to read the data again into memory), but it is a way of saving memory.

[luizirber]

What this PR avoids is writing/reading the data to a temporary file, because the Rust Nodegraph can be read from a memory buffer, which khmer didn't support. It used to be

• Read from storage into memory buffer
• Write memory buffer to temp file
• load nodegraph from temp file

now it is

• Read from storage into memory buffer
• load nodegraph from memory buffer

[ctb]

Side note, at one point David Koslicki was interested in reusing our Bloom filter implementation. Might see if this fits his needs.

[luizirber]

Side note, at one point David Koslicki was interested in reusing our Bloom filter implementation. Might see if this fits his needs.

This is not a full reimplementation of the khmer.Nodegraph, and I think he was using other features that sourmash doesn't use (so I didn't implement them).

And I think CMash is also using kmc nowadays.

[ctb]

On Tue, Apr 21, 2020 at 10:04:48AM -0700, Luiz Irber wrote: (and please don't update the PR with a `master` merge if it's not right before merging the PR, it messes rebasing =])

REBASING fine

[ctb]

How does writing compressed nodegraphs work, then? I'm guessing we would need to add that as a feature to SBTs?

[luizirber]

How does writing compressed nodegraphs work, then? I'm guessing we would need to add that as a feature to SBTs?

SBTs are currently v4; saving compressed nodes imply a v5 (because if we keep the same version number, older versions of sourmash can't load a v4 anymore if it is compressed...)

There is already experimental support for loading v5 in #694, but we still save as v4 by default. Since we don't save as v5 yet, there is no officially supported v5 SBTs in existence, so it's OK to update the format (probably also adding some field for the LocalizedSBT in #925?)

(Currently the difference from v4 to v5 is splitting leaves and internal nodes, which allows saving LinearIndex in the same format. Also, only the leaves are required, and maybe start calling them datasets or signatures is also good)

I suggest merging this PR with support for loading compressed nodegraphs, and implement v5 and other changes in #648

[ctb]

+1