Cursor never moves forward when mixing autocomplete and emoji

[refi64]

Yes, this is a weird-as-crud bug. I found it while working on my terminal emulator.

Basically:

• Make sure you're using a shell that tries to autocomplete your history by showing a shadowed version right after the normal text. This includes stuff like Xonsh or zsh + oh my zsh. If that doesn't make sense, here's a screenshot:
  

  Notice the shadowed autocomplete text.

• Open up a normal terminal emulator (or pretty much any one that supports copy/paste) and paste the following line:

  echo 🤔
  

• Now that it's been saved to your history, open up gtktsm or any other terminal emulator that uses libtsm. When you enter the letter e and the autocomplete fills in the rest of the line, the cursor will never move forward. Hit backspace, e, backspace, e, over and over again, and you can literally erase the entire line.

This is weird...

[Aetf]

Hmm, another weird bug with emoji.

I don't have zsh installed, but I'm using fish, which has autocomplete builtin. Anyway, I tested in kmscon, gtktsm, and also in non-libtsm-based konsole, just to see how does it behave.

Despite some font and other cursor placement issues, I can't reproduce the exact issue you described.

Probably due to the way zsh implementing autocomplete is different from fish?

[refi64]

Just tried, and I can't reproduce it in fish either.

Xonsh is another shell where this problem comes up.

[Aetf]

I currently have few spare time to investigate this. Pull requests are welcomed.

[refi64]

Hmm, looks like the problem is basically with mk_wcwidth: the library is old, and the Unicode standards it's based on haven't been updated in years. Therefore, is basically thinks that most characters added after 2007 (including emoji) are only 1 column wide, which screws up all the calculations.

Out of curiosity: is there a reason libtsm doesn't just use POSIX's wcwidth? For all practical purposes, on Unix platforms, it's usually UTF-32/UCS4 anyway. There's also a wcwidth implementation that Termux uses...

[Aetf]

It's on the original author's to-do list for a while...

Anyway, I tried the wcwidth implementation from Termux, which seems to be the most up to date one in C (usually POSIX's wcwidth in libc is several Unicode standards behind).
Now the emoji is displayed full width, but breaking line width calculation in fish. But xonsh is working fine now...

I pushed the code to wcwidth branch, if you want to play with it.

[refi64]

@Aetf I believe this is this Fish bug, as I can reproduce it in Sakura and URxvt, too.

One of the comments seems to suggest that this happens when the terminal emulator's wcwidth disagrees with Fish's. In this case, Fish thinks the emoji is width 1, but now libtsm is (correctly) seeing it as width 2, which breaks everything.

Maybe there could be a runtime setting for this? Some kind of global function (tsm_set_internal_wcwidth?) that sets whether or not to globally use the internal wcwidth or the system one. That way, when on shells like Fish that glitch out, you could override the setting.

[Aetf]

Yeah, I saw that bug a while ago when I was messing around with emojis in Konsole.

Having a switch for wcwidth isn't going to help here, as fish also uses its internal copy of wcwidth, unless otherwise compiled.

Another way is to force fish emoji width to 2 using set fish_emoji_width 2. I tried that with gtktsm (using updated wcwidth), and it works, at least the cursor position is correct.

Does the wcwidth branch fixes the original problem you posted? If so I'm going to merge that.

[refi64]

Yup, it fixes the original issue. I guess Xonsh works now since Python probably has its own wcwidth.

[Aetf]

Xonsh works with new wcwidth. I've tried that. Fix merged in #8