Add nvtext::jaccard_index API for strings columns #13669
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
davidwendt
added
feature request
…ngram-chars-hashed
davidwendt
added
3 - Ready for Review
|
We are seeing a 6x+speedup 🚀 with this PR on our internal workflows. Thanks for the great work on this @davidwendt . Let's try to get this one in soon. |
davidwendt
changed the title
davidwendt
changed the title
bdice
reviewed
Jul 18, 2023
davidwendt
changed the title
bdice
approved these changes
Jul 19, 2023
vyasr
requested changes
Jul 19, 2023
There was a problem hiding this comment.
This is a slick implementation, nice work! I have some queries, but most are either not terribly substantial or could be punted on if needed.
Do we care about the potential for hash collisions, or is the approximation we get from the hashed values good enough?
vyasr
approved these changes
Jul 19, 2023
|
/merge |
etseidl
pushed a commit
to etseidl/cudf
that referenced
this pull request
Jul 21, 2023
Adds the `nvtext::jaccard_index()` API to cudf. ``` std::unique_ptr<cudf::column> jaccard_index( cudf::strings_column_view const& input1, cudf::strings_column_view const& input2, cudf::size_type width, rmm::mr::device_memory_resource* mr); ``` The Jaccard Index is described here for computing the distance between two arrays of elements: https://en.wikipedia.org/wiki/Jaccard_index ``` Formula is J = |A ∩ B| ------- |A ∪ B| where |A ∩ B| is number of common values between A and B and |x| is the number of unique values in x. ``` The computation here compares strings columns by treating each string as text (i.e. sentences, paragraphs, articles) instead of individual words or tokens to be compared directly. The algorithm applies a sliding window (size specified by the `width` parameter) to each string to form the set of tokens to compare within each row of the two input columns. These substrings are essentially character ngrams and each substring is part of the union and intersect calculations for that row. For efficiency, the substrings are hashed using the default MurmurHash32 to identify uniqueness within each row. Once the union and intersect sizes for the row are resolved, the Jaccard index is computed using the above formula and returned as a float32 value. The two input columns must be the same size and will match the output column size. Authors: - David Wendt (https://github.com/davidwendt) - Vyas Ramasubramani (https://github.com/vyasr) Approvers: - Bradley Dice (https://github.com/bdice) - Vyas Ramasubramani (https://github.com/vyasr) URL: rapidsai#13669
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Adds the
nvtext::jaccard_index()API to cudf.The Jaccard Index is described here for computing the distance between two arrays of elements: https://en.wikipedia.org/wiki/Jaccard_index
The computation here compares strings columns by treating each string as text (i.e. sentences, paragraphs, articles) instead of individual words or tokens to be compared directly. The algorithm applies a sliding window (size specified by the
widthparameter) to each string to form the set of tokens to compare within each row of the two input columns.These substrings are essentially character ngrams and each substring is part of the union and intersect calculations for that row. For efficiency, the substrings are hashed using the default MurmurHash32 to identify uniqueness within each row.
Once the union and intersect sizes for the row are resolved, the Jaccard index is computed using the above formula and returned as a float32 value.
The two input columns must be the same size and will match the output column size.
Checklist