Depend on Erlang for computing grapheme clusters #11024
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Erlang ships with its own embedding of the Unicode Codebase for a couple releases and this release changes Elixir to depend on it in order to compute grapheme clusters. The Erlang implementation was up to 2x faster in low codepoints (such as latin1) while the Elixir one could be faster up to 3x in high codepoints (such as emoji) so at the end the performance results are roughly the same. As a benefit, we no longer need to ship our copy of the grapheme cluster algorithm, which would take up to 250kB in disk and more than 15 seconds to compile. Note we still keep our own String downcase and upcase algorithms, as our version is considerably more efficient on all cases since it works exclusively with binaries (more than 5x faster).
josevalim
commented
May 31, 2021
josevalim
commented
May 31, 2021
| # 4. Update String.Unicode.version/0 and on String module docs (version and link) | ||
| # 5. make unicode | ||
|
|
||
| data_path = Path.join(__DIR__, "UnicodeData.txt") |
There was a problem hiding this comment.
I have moved properties.ex to this file and renamed String.Unicode to String.Casing. Other than that, the contents are strictly the same.
There was a problem hiding this comment.
Correct, renamed Casing to Unicode.
fertapric
reviewed
May 31, 2021
fertapric
reviewed
May 31, 2021
lib/elixir/lib/string.ex
Outdated
| defp do_last([gc | rest], _), do: do_last(:unicode_util.gc(rest), gc) | ||
| defp do_last([], acc) when is_binary(acc), do: acc | ||
| defp do_last([], acc), do: :unicode.characters_to_binary([acc]) | ||
| defp do_last({:error, <<byte, rest::bitstring>>}, _), do: do_last(rest, <<byte>>) |
There was a problem hiding this comment.
Suggested change
| defp do_last({:error, <<byte, rest::bitstring>>}, _), do: do_last(rest, <<byte>>) | |
| defp do_last({:error, <<byte, rest::bitstring>>}, _), do: do_last(:unicode_util.gc(rest), <<byte>>) |
For example, String.last(<<?a, 200, ?a>>) #=> "a"
There was a problem hiding this comment.
Awesome find. Fixed and regression test added.
fertapric
approved these changes
Jun 1, 2021
|
💚 💙 💜 💛 ❤️ |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Erlang ships with its own embedding of the Unicode
Codebase for a couple releases and this release changes
Elixir to depend on it in order to compute grapheme
clusters.
The Erlang implementation was up to 2x faster in low
codepoints (such as latin1) while the Elixir one could
be faster up to 3x in high codepoints (such as emoji)
so at the end the performance results are roughly the
same. As a benefit, we no longer need to ship our copy
of the grapheme cluster algorithm, which would take up
to 250kB in disk and more than 15 seconds to compile.
Note we still keep our own String downcase and upcase
algorithms, as our version is considerably more efficient
on all cases since it works exclusively with binaries
(more than 5x faster).