[QST] Efficient minhashing with cuDF

[ayushdg]

What is your question?
Minhashing or locality sensitive hashing is a popular technique used to group similar documents based on the similarity b/w words/shingles in those documents.

Right now I have an approach that computes the minhashes for documents as follows:
Series of documents -> n-grams(shingles)-> hash_values -> groupby+min

---

Documents

0            doc 1
1    this is doc 2
2            doc 3
Name: doc_text, dtype: object

exploded n-grams

0     doc 
0     oc 1
1     this
1     his 
1     is i
1     s is
1     is 
1     is d
1     s do
1     doc
1     doc 
1     oc 2
3     doc 
3     oc 3
Name: doc_text, dtype: object

hash_values -> groupby+min

2    2005254014
0     216665549
1    1081029400
dtype: uint32

The need to explode each document into a bunch of tokens increase the memory consumption per document and adds the need to groupby document_id at the end

I was wondering if some sort of UDF or custom kernel might be more efficient for this task which can parallelize the sliding window hashing + storing minhash that would avoid this extra memory usage and the need for a groupby at the end.

cc: @davidwendt @wence-

[ayushdg]

Edit: Also it is common in practice to compute multiple minhashes k, either by using k different hash function, 1 hash function and k different permutations of that hash, or 1 hash function with k different seeds.

Taking the last case as an example we would compute 1 minhash for each seed per document, and end up with k such minhashes.