Unicode split

[devinus]

No description provided.

[yrashk]

I strongly suggest implementing http://www.unicode.org/reports/tr29/ instead as it a more correct way to split in Unicode. @josevalim ?

[devinus]

@yrashk That report deals with grouping words into word boundaries, not splitting on whitespace. It's not that it's more "correct," it's entirely different functionality. Like text_segments or something. If we have a traditional split on whitespace function I'd prefer it to use the same codepoints stripped by strip as well.

[yrashk]

I understand your point. I am just trying to figure out what's the best thing to do. Isn't splitting by whitespace a subset of text segmentation? Text segmentation allows to truly break text into words, accounting for punctuation, parens and such...

[devinus]

@yrashk Yep, and I think it's awesome and should be added. But this is merely a split on whitespace function that should treat the same codepoints as whitespace that strip et al do too.

[devinus]

String.split "  foo  bar  baz  " #=> ["foo", "bar", "baz"]

[devinus]

String.text_segments

[josevalim]

@devinus could you please update the docs too to make it clear that it will split any unicode whitespace (and not word boudaries) and add an unicode example? Thanks!

[devinus]

@josevalim Done.

[yrashk]

bump?