Fix comparing strings with multibyte chars #3
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Using `x.len()` on a str returns the number of bytes, which is not
always the same as the number of characters. For example, the characters
'ö' or '香' are representing using more than one byte. This leads
to strange behaviour (e.g. `levenshtein("a", "ä")` would return 0).
Replace calls to `str.len()` with `str.chars().count()`, which returns
the correct number of characters in a string.
|
Thanks for this! I just published it to Cargo as v0.4.1. |
|
Thanks for the quick response :) |
jnqnfe
added a commit
to jnqnfe/strsim-rs
that referenced
this pull request
Nov 4, 2018
The new 'normalised' form does char counting on top of the standard algorithm it calls. This change avoids unnecessary recalculation by moving the main algorithm to a private function which takes `Option` wrapped counts, allowing the normalised form to pass in the values it calculates, thus letting this be done only once.
jnqnfe
added a commit
to jnqnfe/strsim-rs
that referenced
this pull request
Nov 4, 2018
as per levenshtein optimisation rapidfuzz#2
jnqnfe
added a commit
to jnqnfe/strsim-rs
that referenced
this pull request
Nov 4, 2018
avoid clone() on a usize, it's copy-able so let's just copy it
jnqnfe
added a commit
to jnqnfe/strsim-rs
that referenced
this pull request
Nov 4, 2018
more simple, and construction with vec! here should be more optimal over a push() loop (see the implementation which uses the private extend_with() function)
jnqnfe
added a commit
to jnqnfe/strsim-rs
that referenced
this pull request
Nov 4, 2018
use vec! for `curr_distances` construction, as with jaro optimisation rapidfuzz#3
jnqnfe
added a commit
to jnqnfe/strsim-rs
that referenced
this pull request
Nov 4, 2018
same as with levenstein optimisation rapidfuzz#3 - avoid unnecessary repeated char counting with 'normalised' form.
jnqnfe
added a commit
to jnqnfe/strsim-rs
that referenced
this pull request
Nov 8, 2018
Taking the j-w optimisations further, this makes use of the prefix splitting helper within the inner Jaro algorithm. The function has been modified such that instead of taking a char-count of the size of the common prefix removed from the pair of strings, it now optionally takes a pointer to return the count, obtaining it within the function through use of the helper internally. Using the prefix splitting helper within the function means that we avoid doing a `.chars().count()` iteration over the prefix twice, once going over `a` and once going over `b`. It also then allows the main part of the algorithm to completely avoid processing the common prefix portion of the strings.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Using
x.len()on a str returns the number of bytes, which is notalways the same as the number of characters. For example, the characters
'ö' or '香' are representing using more than one byte. This leads
to strange behaviour (e.g.
levenshtein("a", "ä")would return 0).Replace calls to
str.len()withstr.chars().count(), which returnsthe correct number of characters in a string.