Use locality-sensitive hashing to reduce space of candidate probes #13
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This works with any given LSH family.
Like the duplicate filter, this is not necessary, but running it before the set cover filter can significantly lower runtime and reduce memory usage by decreasing the input size (number of candidate probes) to the set cover filter. This detects and collapses near-duplicate probes using locality-sensitive hashing with a family of hash functions that works with Hamming distance. An option to use this filter (in place of the duplicate filter) is provided in the arguments for design.py. There are two caveats explained in the argument's help message: (1) this filter may lead to a slightly less-than-optimal solution (i.e., more probes than needed) because it may remove "better" candidate probes that are near-duplicates of ones that are kept, and (2) this filter may (depending on the provided distance threshold) lead to less than desired coverage of the target genomes because the universe to cover is constructed from the set of candidate probes, which are collapsed by this filter.
…r neighbor lookup
This adds another family of hash functions to use for LSH. In this context, each sequence can be thought of as a set of k-mers; hash functions from this family will hash similar sequences to the same value based on their shared k-mers.
This filter is like the one using LSH based on Hamming distance, except this uses a family that constructs MinHash signatures. This commit adds an option to design.py for using this filter. The main advantage of this filter compared to the Hamming distance filter is that it can be more sensitive in detecting near-duplicate probes (e.g., ones that are shifted relative to each other). The main downside is that the distance threshold is based on Jaccard distance and it is less intuitive to understand how these values translate to probe hybridization compared to Hamming distance.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
A user can invoke LSH as one of two optional arguments provided to the executable. This can provide significant reductions in runtime and memory requirements, particularly when the input consists of a large number of highly similar sequences.