String.split / Regex.split trim behavior with empty patterns

[bruce]

Both String.split and Regex.split (which the former relies on) have the following behavior when provided an empty pattern:

iex> String.split("abc", %r//)
["a", "b", "c", ""]
iex> Regex.split(%r//, "abc")
["a", "b", "c", ""]

While I understand we can provide the :trim option and that this behavior matches Erlang's re module, namely:

[This] leaves an empty rest which is also returned. This behaviour differs from the default behaviour of the split function in Perl, where empty strings at the end are by default removed.

I thought it might make sense to discuss whether Elixir's more friendly (dare I say "modern") approach might provide an opening to change this to the more conventional behavior that I think most would expect. (I still don't understand why Erlang doesn't also return an empty value at the beginning of the result list.)

If the decision is made not to change this behavior, perhaps this behavior should be clearly stated in the Elixir docs, with doctest-able examples included.

Thoughts?

[josevalim]

Hey Bruce!

I would argue that this behaviour is consistent with the other values we can pass to split:

iex> String.split "abc", "c"
["ab", ""]
iex> String.split "abc", %r"c"
["ab", ""]
iex> String.split "", %r""
[""]

Unless I misunderstood you and you simply want to make :trim true by default under all circumstances?

[bruce]

While it may be consistent, I don't think the result has any value to the user. For instance, in this example:

iex> String.split "abc", %r"c"
["ab", ""]

The empty has value because it shows a match that may not occur, eg, compare to:

iex> String.split "abc", %r"d"
["abc"]

The empty value returned in the original example is significant. Since matches on empty values always return an empty end, I believe there'd never be a reason to not pass :trim.

Now, if the argument is that the empty is significant, I'd argue that an empty at the beginning of the result would be equally significant, eg, this would be a more consistent approach

iex> String.split("abc", %r"") # doesn't work
["", "a", "b", "c", ""]
# Consistent with the following:
iex> String.split("cabc", %r"c")
["", "ab", ""]

So, :trim by default for everything? No, for empty patterns? Yes (or some other path to the same result).

But, if you don't agree 😀 maybe we should add docs -- it was certainly a surprise to me (even having used Erlang, I'd never run across this interesting behavior).

[josevalim]

Now, if the argument is that the empty is significant, I'd argue that an empty at the beginning of the result would be equally significant, eg, this would be a more consistent approach.

A split only happens after a character is consumed. That's why it doesn't make sense to have an empty one at the beginning, before the match. :)

But, if you don't agree 😀 maybe we should add docs

A pull request for the docs is very welcome then! :) And, after all, if you want to explicitly split on the empty pattern, functions like :binary.bin_to_list, String.codepoints and String.graphemes would be clearer and do a better job too. :)

[bruce]

A pull request for the docs is very welcome then!

I'm happy to do that! 😄

If you want to explicitly split on the empty pattern, functions like :binary.bin_to_list, String.codepoints and String.graphemes would be clearer and do a better job too.

I definitely think you're right there; they have the benefit of being more explicit as well. I just wanted to point out the unique behavior.

One more thing I noticed:

iex> String.split("abc", "")
** (ArgumentError) argument error
    binary.erl:242: :binary.split/3
    /private/tmp/elixir-sdVh/lib/elixir/lib/string.ex:201: String.split/3
    erl_eval.erl:569: :erl_eval.do_apply/6
    src/elixir.erl:138: :elixir.eval_forms/3

Since it's certainly possible for someone to be splitting by a dynamic value, it seems like we should allow an empty string (eg, this feels like an implementation concern of which underlying Erlang function to use). Thoughts?

[josevalim]

Good catch! An empty string should probably default to String.graphemes
with the added "" :P

José Valim
www.plataformatec.com.br
Skype: jv.ptec
Founder and Lead Developer

[bruce]

Ok, I'll put in a PR for that as well. Thanks!