Make internal encoding of locations aware of unicode #6073
Conversation
When some unicode characters are present on a line, the existing encoding of positions, based on number of bytes since line start, is incorrect. This can be seen in e.g. error messages picked up in the editor (or on the command-line). This PR takes unicode into account. Even thought the ocaml locations are byte-based, one can trick the system by encoding as pos_cnum: (number of bytes from file start to line start) + (number of utf16 code units since line start) Since the compiler's printer performs a subtraction, the utf16 character position is shown. Notice that editors, vscode in particular, show you something in "Col", but its internal commands expect correct utf16 character which is different.
cristianoc
changed the title
| @@ -0,0 +1,12 @@ | |||
|
|
|||
| [1;31mWe've found a bug for you
|
I think this might fix a long standing issue with semantic highlighting and my native tounge (Swedish) where using letters like åäö offsets the coloring so it ends up wrong after those letters. |
Definitely. If you have an example at hand at some point, it would be handy to make sure it does. |
Out traveling, but try for example: let str = `hej på dig ${123->Int.toString}`That should color the interpolated part wrong. |
|
Found one independently: let foo = x => if ( x == "Den här positionen är avstängd, väldigt avstängd") {<div />} else { <div />} |
|
@cristianoc love the Swedish! |
|
This does fix the issue, once the relevant parser files in the editor extension are updated. |
This fixes highlighting, and other things, in presence of unicode strings.
When some unicode characters are present on a line, the existing encoding of positions, based on number of bytes since line start, is incorrect. This can be seen in e.g. error messages picked up in the editor (or on the command-line).
This PR takes unicode into account. Even though the ocaml locations are byte-based, one can trick the system by encoding as
pos_cnum:(number of bytes from file start to line start) + (number of utf16 code units since line start)Since the compiler's printer performs a subtraction, the utf16 character position is shown. Notice that editors, vscode in particular, show you something in "Col", but its internal commands expect correct utf16 character which is different.