Emoji generates rendering problem

[guillaumebreton]

Summary

When an emoji is rendered on the screen, other characters are not rendered at their correct places.

How to reproduce

I've built a test program in Go which displays an emoji and waits for an input. As soon as the input is renderer, an invalid space is placed in front of the line, and if a backspace is used, the complete input is mangled. If the line is selected using the mouse, the complete screen is badly rendered.

Here is a video of the problem : https://youtu.be/R1GpJX7uEdw

This video use the following configuration : https://github.com/guillaumebreton/dotfiles/blob/master/tmux/tmux.conf.symlink

The bug is not present outside TMUX (ie. inside iterm2)

Environment

Platform : Darwin i386 with Iterm2 (Build 3.0.14)
Tmux version : tmux 2.3
$TERM outside TMUX : xterm-256color
$TERM inside TMUX : screen-256color

Logs

Logs of a session with no configuration (as found in CONTRIBUTING.md)
logs.zip

[nicm]

Does this still happen with master? And with a tmux built with --enable-utf8proc? On 23 Mar 2017 9:52 pm, "Guillaume Breton" <notifications@github.com> wrote: Summary When an emoji is rendered on the screen, other characters are not rendered at their correct places. How to reproduce I've build a test program <https://github.com/guillaumebreton/tmux-emoji> in Go which displays an emoji and waits for an input. As soon as the input is renderer, a invalid space is placed in front of the line, and if a backspace is used, the complete input is mangled. If the line is selected using the mouse, the complete screen is badly rendered. Here is a video of the problem : https://youtu.be/R1GpJX7uEdw This video use the following configuration : https://github.com/ guillaumebreton/dotfiles/blob/master/tmux/tmux.conf.symlink The bug is not present outside TMUX (ie. inside iterm2) Environment Platform : Darwin i386 with Iterm2 (Build 3.0.14) Tmux version : tmux 2.3 $TERM outside TMUX : xterm-256color $TERM inside TMUX : screen-256color Logs Logs of a session with no configuration (as found in CONTRIBUTING.md) logs.zip <https://github.com/tmux/tmux/files/866256/logs.zip> — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#836>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AASkcyXzasdwnA6iGzqFA0L29802aj8fks5roummgaJpZM4Mnab7> .

[guillaumebreton]

Let me try to build master with --enable-utf8proc 👍

[guillaumebreton]

The bug is still present with master without --enable-utf8proc.

I tried ./configure --enable-utf8proc but it does not seem to work (configure: error: "utf8proc not found"). How can I use this flag ?

[nicm]

You need to install libutf8proc first. On 23 Mar 2017 10:20 pm, "Guillaume Breton" <notifications@github.com> wrote: The bug is still present with master without --enable-utf8proc. I tried ./configure --enable-utf8proc but it does not seem to work (configure: error: "utf8proc not found"). How can I use this flag ? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#836 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AASkc9ybyeS5rXGbMGM4U3FhAw3XRAZnks5rovAUgaJpZM4Mnab7> .

[guillaumebreton]

@nicm thanks 👍 Made the test with master and --enabled-utf8proc but the problem persist. ( I didn't test the mouse selection though as I launched Tmux with mo configuration)

Here is a video of the steps I followed : https://youtu.be/Iv53xnAB-CE

[nicm]

Please run tmux built from master with -vvvv then print one problem symbol, then exit tmux and show me the tmux-server-*.log file.

…

On Fri, Mar 24, 2017 at 03:00:58AM -0700, Guillaume Breton wrote: @nicm thanks d-*** Made the test with master and --enabled-utf8proc but the problem persist. ( I didn't test the mouse selection though as I launched Tmux with mo configuration) Here is a video of the steps I followed : https://youtu.be/Iv53xnAB-CE - You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread. Link: #836 (comment)

[guillaumebreton]

Here are the logs :)
logs_simple_test.zip

[nicm]

Your libc (or utf8proc if you have built with it) is failing to return a width for this symbol:

1490304541.683582 input_utf8_open 3
1490304541.683594 input_utf8_add
1490304541.683602 Unicode 26a1, wcwidth() -1
1490304541.683608 input_utf8_close 3 '\342\232\241' (width 1)

So tmux has no real choice but to assume it is width 1.

This is a common issue on OS X because the locale support is poor. But I'm surprised utf8proc doesn't have it either. Are you sure you restarted tmux completely after building with utf8proc?

[nicm]

On Linux, libc tells me this symbol should be width 1. So possibly in this case libc is correct and your terminal has the wrong idea of the width (or they are using different Unicode versions and the width changed).

[guillaumebreton]

I ran a tmux kill-server before every test to be sure and the install the utf8proc with brew install utf8proc (which points to http://julialang.org/utf8proc/ version 2.1). In my case, my terminal is rendering everything correctly outside tmux :/

Is there anything I could do to gather more information about this issue ?

[nicm]

Right but tmux and the terminal have to agree on the widths of characters, just because it works outside doesn't mean it is getting it right.

[guillaumebreton]

Ok, it makes sense :). Just to be sure, I tested with Iterm2 and the default OSX terminal and end up with the same result.

[nicm]

The log you sent me is not from master, can you try again and make sure you have built from the latest Git master?

I think the problem is from the second symbol not the first, which appears to be \357\270\217 which is U+FE0F http://graphemica.com/FE0F which is zero width combining mark, but your system is saying it is width 1.

[guillaumebreton]

tmux-server-48424.txt
Sorry 🤦‍♂️ Here is the correct log, I checked the version this time in the log. It may come from the way I output the emoji in the go test program. I'll try to replace it with the unicode value, and keep you updated.

[nicm]

OK this log is different:

1490353192.403073 input_utf8_close 3 '\342\232\241' (width 2)
1490353192.403097 input_utf8_close 3 '\357\270\217' (width 0)

I think your terminal thinks that \342\232\241 should be width 1 not width 2.

[nicm]

What does a screenshot of printing these symbols look like inside and outside tmux?

[guillaumebreton]

While doing screenshots, I found a way to reproduce the bad printing. It seems that it comes from the terminal, as the \342\232\241 character with the \357\270\217 character should be of length 2 not 1. This behaviour can be reproduced inside tmux, outside tmux (in both iterm2 and osx terminal). What do you think ?

[nicm]

In all these screenshots, the symbol is two cells wide, but the terminal is only moving the cursor by one cell. I think \357\270\217 is irrelevant because it is zero width.

[guillaumebreton]

If I don't output the \uFE0F character, the character is not "painted" in osx terminal, and it's 1 cell wide.

[nicm]

OK well in your last log tmux thinks the character is 2 cells wide, so that is what the terminal needs to do too or they will not display it properly when used together.

[guillaumebreton]

Ok, thank a lot for you help 👍 I close the issue.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.