Skip to content

Reduce allocations in JaccardSimilarity and JaroWinklerSimilarity#757

Closed
nishantmehta wants to merge 2 commits into
apache:masterfrom
nishantmehta:pr/similarity-allocs
Closed

Reduce allocations in JaccardSimilarity and JaroWinklerSimilarity#757
nishantmehta wants to merge 2 commits into
apache:masterfrom
nishantmehta:pr/similarity-allocs

Conversation

@nishantmehta

Copy link
Copy Markdown

Summary

Two independent allocation reductions in the similarity package:

1. JaccardSimilarity.apply built three HashSets per call: the distinct elements of each input, plus a union set (a copy of the left set with all right elements added) used only to obtain the union size. By the inclusion–exclusion identity |A∪B| = |A| + |B| − |A∩B|, the union set is unnecessary — count the intersection by probing the smaller distinct-element set against the larger and derive the union size arithmetically. The two remaining sets are inherent (the distinct elements of each input) and are now pre-sized to the input length.

2. JaroWinklerSimilarity (matches) materialized two Object[] arrays (the matched characters of each input) solely to count transpositions position-by-position, and tracked matched positions of the shorter input in an int[] where a boolean per position suffices. Count transpositions by walking the matched positions of both inputs in lockstep, comparing characters directly, and use a boolean[] for the match marks.

Behavior is unchanged in both.

Measurement

ThreadMXBean allocation driver, 200k warmed ops, two 19-character inputs:

method before after
JaccardSimilarity.apply 2408 B/op 1448 B/op (−40%)
JaroWinklerSimilarity (matches) 376 B/op 144 B/op (−62%)

Testing

JaccardSimilarityTest, JaccardDistanceTest, JaroWinklerSimilarityTest and JaroWinklerDistanceTest pass unchanged (36 tests).

apply() built three HashSets per call: the distinct elements of each input,
plus a union set (a copy of the left set with all right elements added) used
only to obtain the union size.

The union size is determined by the inclusion-exclusion identity
|union| = |left| + |right| - |intersection|, so the third set is unnecessary.
Count the intersection by probing the smaller distinct-element set against the
larger, then derive the union size arithmetically. The two remaining sets are
inherent (the distinct elements of each input) and the sets are pre-sized to
the input length to avoid resizing.

Measured with a ThreadMXBean allocation driver (200k warmed ops, two
19-character inputs): 2408 B/op -> 1448 B/op (-40%). JaccardSimilarityTest and
JaccardDistanceTest pass unchanged.

Signed-off-by: Nishant Mehta <nishantmehta.n@gmail.com>
…Similarity

matches() materialized two Object[] arrays (the matched characters of each
input) solely to count transpositions by comparing them position by position,
and tracked matched positions of the shorter input in an int[] when only a
boolean per position is used.

Count transpositions by walking the matched positions of both inputs in
lockstep instead, comparing characters directly, and use a boolean[] for the
shorter input's match marks. This removes the two Object[] allocations and
shrinks the index array; behavior is unchanged.

Measured with a ThreadMXBean allocation driver (200k warmed ops, two
19-character inputs): 376 B/op -> 144 B/op (-62%). JaroWinklerSimilarityTest
and JaroWinklerDistanceTest pass unchanged (20 tests).

Signed-off-by: Nishant Mehta <nishantmehta.n@gmail.com>
@garydgregory

Copy link
Copy Markdown
Member

Closing. No one's asking for this. Fix bugs in Jira if want to help improve this things.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants