Unicode issue for Supplementary Planes

[eoftedal]

If you enter a unicode character in the Supplementary Planes like this http://www.fileformat.info/info/unicode/char/1D306/index.htm or http://www.fileformat.info/info/unicode/char/1F4A9/index.htm , it's counted as two characters. You can actually move the cursor to the middle of the character, and mark half a character.

Note: If you move in between the two halves, and insert a new character in the middle, the whole line disappears. If you save and reopen, it's no longer interpreted as UTF-8 when you reopen. If you save with UTF-8 BOM, reopening makes a little bit more sense.

[FichteFoll]

Same as #663?

[mathiasbynens]

@FichteFoll That’s a separate issue. This issue is about astral / supplementary Unicode symbols and how they behave in the editor; #663 is purely a visual/rendering issue with BMP symbols.

[FichteFoll]

Okay, after testing I can see that this is an entirely different issue.

[Torvin]

This breaks API badly. For example, if we have a view with 'a😘b' text, then:

• view.size() returns 4
• len(view.substr(sublime.Region(0, view.size()))) returns 3

So, any positional operations (like view.replace) produce wrong results. E.g.

# replacing 'b' with 'c'
text = self.view.substr(sublime.Region(0, self.view.size()))
index = text.find('b')
self.view.replace(edit, sublime.Region(index, index + 1), 'c')

absolutely breaks current view (even makes text disappear). I suggest changing severity of this bug to 'major'.

Also, is there a workaround? I cannot think of a reliable way to work with such texts via API.

[vovkkk]

@Torvin search in view:

>>> view.find('b', 0)
(3, 4)

r = self.view.find('b', 0)
self.view.replace(edit, r, 'c')

---

BTW, in ST2 len(view.substr(sublime.Region(0, view.size()))) returns 4 — another example that py2>py3.

[FichteFoll]

Appears to be fixed with 3098.