Backspacing over emojis causes them to be half erased

[ElvenSpellmaker]

Setting up Noto emoji using Emojis=noto in the config file (it doesn't allow me to select in the UI, it only ever shows None which might be another bug).

When entering emoji and then backspacing it only deletes half the emoji:

[mintty]

The same happens with double-width CJK characters. The problem is that the shell's line editing features are not fully aware of non-single-width glyphs and position the cursor just one position back in this case.

[ElvenSpellmaker]

I can backspace fine over a double width such as は, is it some kind of broken mapping where the shell hasn't been updated to understand emojis?

[mintty]

Ah right, this didn't work yet not too long ago. So maybe the shell uses out-of-date width information (a.k.a. locale).

[ElvenSpellmaker]

The files referenced by this?

$ locale
LANG=en_GB.UTF-8
LC_CTYPE="en_GB.utf8"
LC_NUMERIC="en_GB.utf8"
LC_TIME="en_GB.utf8"
LC_COLLATE="en_GB.utf8"
LC_MONETARY="en_GB.utf8"
LC_MESSAGES="en_GB.utf8"
LC_ALL=en_GB.utf8

[mintty]

For me, bash and mintty behave alike for characters 啕, ✅, 😀.
Which mintty version do you have and which character do you try?

[ElvenSpellmaker]

3.1.8

啕 erases fine
✅ erases fine
😀 does it in two parts

[mintty]

Forgot to ask which shell you use. Anyway, with bash I get this:

and after 3s:

[mintty]

For the records:
Under the assumption that the shell assumes the smiley to be only 1 cell wide (as explained above), the screen shots show correct behaviour. Not a terminal issue.

[scandum]

Is there a table somewhere with all the correct character widths?

Mintty has a multitude of issues with character width, but there's no point reporting them without knowing whether mintty is getting it wrong, the font is getting it wrong, or whether it's like the wild west and everyone does whatever they want.

[BrianInglis]

@mintty builds the project using tables from the latest Unicode releases which include recommended monospace character spacing for Asian and other glyphs, also takes into account text direction and other info, depending on locale and font settings.
You may see slightly different outputs depending on your locale settings (Asian/Western) and of course fonts.

[mintty]

By default, mintty applies character width strictly following the "locale" set before starting mintty. Thus character width properties are determined by the system, not mintty. There is no multitude of mintty issues in this mode of operation, although the locale mechanism has its own issues, generic to all terminals. There is also no font dependency in default mode of operation.
Matters change with options Locale, Charset, Charwidth, PrintableControls, as well as with some escape sequences dynamically. See the manual page and the wiki page on control sequences. I will try to add a complete overview to the wiki Tips page.

[scandum]

I had the impression indicwide.t is used, among other things, to determine width?

[mintty]

Only if enabled by the respective escape sequence, see https://github.com/mintty/mintty/wiki/CtrlSeqs#wide-characters.

[scandum]

Interesting, too bad it's not supported by xterm as well. Any hints as to how to pull the "locale" information for characters?

[ElvenSpellmaker]

@mintty This backspace bug seems to be fixed, except now emojis are double width but count as single width when cursoring over them to the right, so you have to right arrow over them twice to get to the previous letter.

Also if you just have an emoji then the same applies to left arrow too. If you type a character mid-emoji you end up with a weird prompt:

EDIT: if you use back buffer it looks like this where the glyph has been split in two:

Also, the cursor flashes if you're over an emoji in the terminal.

Should I open two new bugs for these, or keep it all on this?

(My emoji is noto)

[mintty]

To assess this issue and whether it's a bug at all (see comments above), please describe your environment:
which system version (uname -a), which shell and version (e.g. echo $BASH_VERSION) and which character you display.

[ElvenSpellmaker]

$ uname -a
CYGWIN_NT-10.0 AML0147 3.3.3(0.341/5/3) 2021-12-03 16:35 x86_64 Cygwin

$ echo $BASH_VERSION
4.4.12(3)-release

Any emoji causes the above behaviour

[mintty]

You must have configured something unusual or you must be doing something strange. Which you didn't describe, by the way, so a reproducible test case is missing. Look at this:

After one cursor-left:

After another cursor-left:

Back to the end, then Backspace:

[ElvenSpellmaker]

You can see the right right arrow never goes past the emoji (it takes two presses to go over the emoji) and typing a letter ( in this case l) splits the emoji glyph in pieces.

If I now hit enter I get:

$  l▒
-bash: $'\360\237\215l\225': command not found

[ElvenSpellmaker]

This one shows the other behaviour I mentioned whereby the emoji and cursor flashes violently while over an emoji:

[mintty]

Maybe we should also check your .inputrc and your locale, also your libreadline version.
If the interacting application does not handle double-width properly, as e.g. cat does, you can indeed navigate half-way into an emoji and typing a character will then split up the emoji. That's explained above and not a bug.
The cause of your shell behaving this old way is still mysterious.

[mintty]

I wasn't sure about the precise key sequence that you're entering. This shows that exact descriptions are highly preferable over images or videos.
I can now in fact reproduce the issue above, when entering a smiley on the command line and navigating it like this:
🙂 Backspace x cursor-right y
with the following display
🙂 Backspace -> 🙂 highlighted
x -> x🙂, smiley highlighted
cursor-right -> x🙂, smiley highlighted (no change)
y -> x y�, � highlighted

When logging actual character output (e.g. mintty -l .log), final output is:
🙂^H^Hx🙂^H^H^[[Cy�
where ^H is a Backspace character, ^[ is an Escape character, ^[[C is the cursor-right terminal sequence, � is a 0x82 byte.
I don't know why shell interaction produces this sequence of output characters but as explained above, mintty rendering response to it is correct as the application places the cursor inside a double-width character and even outputs an illegal character code, both of which it should never do.
Things get much worse with emoji sequences but I'm not going to demonstrate that now. Not a mintty bug.