Optimise mem usage, disallowing random ref order #10
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Currently, pycoverm is not particularly memory efficient. It keeps coverages in an name=>Vec IndexMap, then creates a matrix and fills the matrxi from the map.
This allows the order of reference sequences in the BAM files to be in different order, but it consumes a lot of memory. Further, it's not very common to have a situation where BAM files are mapped to the exact same set of references, but in a different order, as BAM reference order is deterministic, and reflects the order of the reference FASTA file.
This commit checks that all BAM files have the same headers in the same order. It then computes an old_refindex => new_refindex map, which can be stored as a single Vec. The coverage matrix can then be preallocated and filled in. This approach is nearly perfectly memory efficient.
The only remaining issue is that the headers might take up a lot of memory, which is unnecessary in the case that the users already know the headers. This might seem trivial, but if there are 10 million sequences that each has to be heap allocated, it can take some time. A solution to this, which I employ in Vamb, is to pass a hash of the headers to the coverage computation. The headers can then be verified without consuming any memory.