Tokenizer not rejecting unsupported newlines in strings and other binary fragments

[lukaszsamson]

Elixir and Erlang/OTP versions

Erlang/OTP 28 [erts-16.2] [source] [64-bit] [smp:12:12] [ds:12:12:10] [async-threads:1] [jit]

Elixir 1.20.0-rc.4 (603602e) (compiled with Erlang/OTP 28)

Operating system

any

Current behavior

ASCII fast path in

elixir/lib/elixir/src/elixir_interpolation.erl

Line 83 in 603602e

extract([Char1, Char2 | Rest], Buffer, Output, Line, Column, Scope, Interpol, Last)

does not reject unsupported newlines
Compare that to correct code in

elixir/lib/elixir/src/elixir_interpolation.erl

Line 92 in 603602e

[Char | _] when ?bidi(Char); ?break(Char) ->

Expected behavior

Unsupported newlines rejected

[lukaszsamson]

There is another problem - raw CR is also accepted both in strings and comments. It's not included in ?break macro. I can raise a follow up PR

[josevalim]

@lukaszsamson I need to find the previous discussions but we have not added \r on purpose. IIRC, opposite to the other characters, it is handled properly by the huge majority of editors and tools (instead of being hidden characters).

[josevalim]

It is also worth saying that we forbid \r in code (except as \r\n).

[lukaszsamson]

Yes, editors (and LSP) handle raw CR but elixir does not consider it a line ending. The obvious consequence is line numbering diverges and errors/warnings get confusing positions on code with raw CRs.
In code it is treated a tokenizer error. In string fragments it is rather harmless. I'm concerned about comments. Consider this code:

foo = 1 # some comment <CR>foo = 2

will be displayed as

foo = 1 # some comment 
foo = 2

and elixir formatter will change it to a form (still with )

# some comment <CR> foo = 2
foo = 1

[josevalim]

Got it, please do send a PR!