String.codepoints not returning multiple codepoints

[gringocl]

Environment

• Elixir version (elixir -v): 1.2.5
• Operating system: osx 10.11.3

Current behavior

v 1.2.5

["e", "́"] = "é" |> String.normalize(:nfd) |> String.codepoints
["e", "é"] = "eé" |> String.normalize(:nfd) |> String.codepoints

Expected behavior

v 1.2.3

["e", "́"] = "é" |> String.normalize(:nfd) |> String.codepoints
["e", "e", "́"] = "eé" |> String.normalize(:nfd) |> String.codepoints

This seems to have changed. Was this expected?

[josevalim]

Thank you for the report. It looks like a bug in our nfd code.

[gringocl]

This seems to be the problem. Changing this back to normalize_nfd solves my issue however there are two failing specs.

  defp normalize_nfd(binary, acc) do
    {n, rest} = String.Unicode.next_grapheme_size(binary)
    part = :binary.part(binary, 0, n)
    case n do
      1 -> normalize_nfc(rest, acc <> part)
      _ -> normalize_nfd(rest, acc <> canonical_order(part, []))
    end
  end

I will hopefully have a pr soon!

[josevalim]

Great catch. Such a silly mistake. A PR with tests will be really appreciated. :)

[gringocl]

❓ Lurking around in the String module trying to understand, how this code works is the spec for String.normalize correct?

  @spec normalize(t, atom) :: boolean
  defdelegate normalize(string, form), to: String.Normalizer

Does this return a boolean or is this supposed to be a String.t? These functions return the acc, which seem to be a bitstring.