Parse non-breaking-space

[ericmj]

No description provided.

[ericmj]

@josevalim String.split/1 does not split on nbsp because it is in the bidirectional category "Common Number Separator". String.split/1 splits on the categories: "Paragraph Separator", "Segment Separator" and "Whitespace".

[josevalim]

You said erlang handles it but erlang accepts all codepoints between 120 and 160, so we need to look at other places. I tried in ruby and it does not accept it as any other whitespace.

$ ruby -e "1 + 2"
-e:1: syntax error, unexpected tIDENTIFIER, expecting end-of-input

Other places we can look at?

[ericmj]

ruby accepts it in the shell (the space expands to "\U+FFC2\U+FFA0" on input):

irb(main):001:0> 1\U+FFC2\U+FFA0+2
=> 3

python does not:

>>> 1 +2
  File "<stdin>", line 1
    1 +2
     ^
SyntaxError: invalid syntax

ghci does:

Prelude> 1 +2
3

[josevalim]

@ericmj I am closing this one due to the lack of consistency in other languages. I think the unicode consortium has a document about how whitespace, characters and so on must be treated by programming languages. Someone could read it with more details if they are interested. For now, we can wait until R18 is out, which is when we will add full unicode support to the language (in atoms, identifiers and so on) and when we will definitely need to look into this.