Reduce allocations in JaccardSimilarity and JaroWinklerSimilarity#757
Closed
nishantmehta wants to merge 2 commits into
Closed
Reduce allocations in JaccardSimilarity and JaroWinklerSimilarity#757nishantmehta wants to merge 2 commits into
nishantmehta wants to merge 2 commits into
Conversation
apply() built three HashSets per call: the distinct elements of each input, plus a union set (a copy of the left set with all right elements added) used only to obtain the union size. The union size is determined by the inclusion-exclusion identity |union| = |left| + |right| - |intersection|, so the third set is unnecessary. Count the intersection by probing the smaller distinct-element set against the larger, then derive the union size arithmetically. The two remaining sets are inherent (the distinct elements of each input) and the sets are pre-sized to the input length to avoid resizing. Measured with a ThreadMXBean allocation driver (200k warmed ops, two 19-character inputs): 2408 B/op -> 1448 B/op (-40%). JaccardSimilarityTest and JaccardDistanceTest pass unchanged. Signed-off-by: Nishant Mehta <nishantmehta.n@gmail.com>
…Similarity matches() materialized two Object[] arrays (the matched characters of each input) solely to count transpositions by comparing them position by position, and tracked matched positions of the shorter input in an int[] when only a boolean per position is used. Count transpositions by walking the matched positions of both inputs in lockstep instead, comparing characters directly, and use a boolean[] for the shorter input's match marks. This removes the two Object[] allocations and shrinks the index array; behavior is unchanged. Measured with a ThreadMXBean allocation driver (200k warmed ops, two 19-character inputs): 376 B/op -> 144 B/op (-62%). JaroWinklerSimilarityTest and JaroWinklerDistanceTest pass unchanged (20 tests). Signed-off-by: Nishant Mehta <nishantmehta.n@gmail.com>
Member
|
Closing. No one's asking for this. Fix bugs in Jira if want to help improve this things. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two independent allocation reductions in the
similaritypackage:1.
JaccardSimilarity.applybuilt threeHashSets per call: the distinct elements of each input, plus a union set (a copy of the left set with all right elements added) used only to obtain the union size. By the inclusion–exclusion identity|A∪B| = |A| + |B| − |A∩B|, the union set is unnecessary — count the intersection by probing the smaller distinct-element set against the larger and derive the union size arithmetically. The two remaining sets are inherent (the distinct elements of each input) and are now pre-sized to the input length.2.
JaroWinklerSimilarity(matches) materialized twoObject[]arrays (the matched characters of each input) solely to count transpositions position-by-position, and tracked matched positions of the shorter input in anint[]where abooleanper position suffices. Count transpositions by walking the matched positions of both inputs in lockstep, comparing characters directly, and use aboolean[]for the match marks.Behavior is unchanged in both.
Measurement
ThreadMXBean allocation driver, 200k warmed ops, two 19-character inputs:
JaccardSimilarity.applyJaroWinklerSimilarity(matches)Testing
JaccardSimilarityTest,JaccardDistanceTest,JaroWinklerSimilarityTestandJaroWinklerDistanceTestpass unchanged (36 tests).