Skip to content

Fix nfd normalization bug #4606

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
josevalim merged 2 commits into from
May 13, 2016
Merged

Fix nfd normalization bug #4606

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
ee60849
Fix nfd normalization bug
May 10, 2016
dba3438
Change String.normalize return type to binary
May 11, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions lib/elixir/lib/string.ex
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -505,13 +505,13 @@ defmodule String do
## Examples

iex> String.normalize("yêṩ", :nfd)
"yêṩ"
"yêṩ"

iex> String.normalize("leña", :nfc)
"leña"

"""
@spec normalize(t, atom) :: boolean
@spec normalize(t, atom) :: t
Copy link
Contributor Author

@gringocl gringocl May 11, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💁 Don't know if this is the right place to change this but it seems incorrect.

defdelegate normalize(string, form), to: String.Normalizer

@doc """
Expand Down
2 changes: 1 addition & 1 deletion lib/elixir/test/elixir/string_test.exs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -378,7 +378,7 @@ defmodule StringTest do

test "normalize" do
assert String.normalize("ŝ", :nfd) == "ŝ"
assert String.normalize("ḇravô", :nfd) == "ḇravô"
assert String.normalize("ḇravô", :nfd) == "ḇravô"
Copy link
Contributor Author

@gringocl gringocl May 11, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't know if there is a way to view these differently in the console. Visually I don't see anything different. Its not until you look at the binary representation or the codepoints that an assessment can be made as by they are failing.

test normalize (StringTest)
     lib/elixir/test/elixir/string_test.exs:379
     Assertion with == failed
     code: String.normalize("ḇravô", :nfd) == "ḇravô"
     lhs:  "ḇravô"
     rhs:  "ḇravô"
     diff: "ḇravôô"
     stacktrace:
       lib/elixir/test/elixir/string_test.exs:382: (test)

Console output

<<"ḇravô">> <> <<0>> = <<98, 204, 177, 114, 97, 118, 111, 204, 130, 0>>  # lhs
<<"ḇravô">> <> <<0>> = <<98, 204, 177, 114, 97, 118, 195, 180, 0>>  # rhs

This does make it difficult to diagnose a test failure.

assert String.normalize("ṩierra", :nfd) == "ṩierra"
assert String.normalize("뢴", :nfd) == "뢴"
assert String.normalize("êchǭ", :nfc) == "êchǭ"
Expand Down
2 changes: 1 addition & 1 deletion lib/elixir/unicode/unicode.ex
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -505,7 +505,7 @@ defmodule String.Normalizer do
{n, rest} = String.Unicode.next_grapheme_size(binary)
part = :binary.part(binary, 0, n)
case n do
1 -> normalize_nfc(rest, acc <> part)
1 -> normalize_nfd(rest, acc <> part)
_ -> normalize_nfd(rest, acc <> canonical_order(part, []))
end
end
Expand Down