datasketch: Big Data Looks Small
datasketch gives you probabilistic data structures that can process and search very large amount of data super fast, with little loss of accuracy.
This package contains the following data sketches:
|MinHash||estimate Jaccard similarity and cardinality|
|Weighted MinHash||estimate weighted Jaccard similarity|
The following indexes for data sketches are provided to support sub-linear query time:
|Index||For Data Sketch||Supported Query Type|
|MinHash LSH||MinHash, Weighted MinHash||Jaccard Threshold|
|MinHash LSH Forest||MinHash, Weighted MinHash||Jaccard Top-K|
|MinHash LSH Ensemble||MinHash||Containment Threshold|
datasketch must be used with Python 2.7 or above and NumPy 1.11 or above. Scipy is optional, but with it the LSH initialization can be much faster.
Note that MinHash LSH also supports a Redis storage layer and have experimental asynchronous module with asynchronous Redis and MongoDB storages.
To install datasketch using
pip install datasketch -U
This will also install NumPy as dependency.