Buffer: unicode-width and emojis #75
Comments
|
To complicate matters further, not every terminal emulator handles emojis the same way. For instance, Windows Terminal is decent for the most part, but renders the "woman scientist" emoji from the
|
|
Has this resulted in a bug or particular issue during your use ? |
It has for me. Because
Edit: Wezterm seems to render my example correctly... |
|
As the author of that issue I dug I ti that a bunch. I didn't have the bandwidth to finish my fix in helix. There are a bunch of issues here. Firstly unicode width does not respect text representation/always uses a fixed unicode version. The problem is that while unicode 9 support (where character width changed last Tims) can be assumed these days, unicode 15 support (supourt for text representation) is different between terminals. Even many terminals that support unicode 14 purposfully don't support text representation(alacritty for example). I have written a custom crate to efficiently compute unicode width that optionally respects text representation. The problem is that there is no way to know what the current emulator expects. If I ever upstream this into helix I would have to detect the emulator by name and offer a setting for the user to overwrite said detection (not as good solution). The second (much bigger) issue in tui-rs is (and helix by extension) is the use of unicode segmentation. No terminal emulator except wezterm (and wezterm is backwards compatible) supports unicode segmentation (altough almost all perform unicode diagraph normalization... With the exception of alacritty... Sigh). The right way to segment text into terminal cells is to associate a single cell with a single unicode-width unit (really what Unicode-width reports is the number of cells). A single cell always contains at most a single non-zero width character (which is currently not the case in tui-rs since it contains a grapheme which can contain multiple non-zero width characters). I do want to get around fixing that in our own fork of tui-rs eventually but unicode-segmentstion js sadly used much more heavily there so its gonna be harder to rip out. |
|
Thanks for this info - can you link to your fork? I noticed a while back that Wez commented about their approach to this at unicode-rs/unicode-width#4 (comment):
/// Returns the number of cells visually occupied by a sequence
/// of graphemes
pub fn unicode_column_width(s: &str) -> usize {
use unicode_segmentation::UnicodeSegmentation;
s.graphemes(true).map(grapheme_column_width).sum()
}
/// Returns the number of cells visually occupied by a grapheme.
/// The input string must be a single grapheme.
pub fn grapheme_column_width(s: &str) -> usize {
// Due to this issue:
// https://github.com/unicode-rs/unicode-width/issues/4
// we cannot simply use the unicode-width crate to compute
// the desired value.
// Let's check for emoji-ness for ourselves first
use xi_unicode::EmojiExt;
for c in s.chars() {
if c.is_emoji_modifier_base() || c.is_emoji_modifier() {
// treat modifier sequences as double wide
return 2;
}
}
UnicodeWidthStr::width(s)
} |
|
Yes, you actually need segmentation to present emoji widths properly. Mentioned by both the quoted comment and my previous comment in this thread. I do not believe that "ripping out unicode-segmentation" is a good idea. In fact, it is a step backward. Terminal cells are represented by a grapheme cluster, not whatever |
|
I actually implemented more or less the thing you proposed above into helix and it completely breaks every single terminal emulator but wezterm. If you pack multiple non-zero characters in a single cell that would take up a single cell in the emulator you end up with a situation where the internal cell coordinates don't line up with those used in the emulator and the diffing mechanism breaks down (like ghost characters and highlights everywhere once such a character is on screen). If you only care about wezterm and consider it a "bug" in the emulator that it doesn't support Unicode segmentation then you can certainly go that route. Most emulators don't actually consider this a bug (try opening an issue with alacrity about this). I also kind of agree with this stance since languages that operate on multicharacter graphemes don't really make sense in a monospaced terminal anyway (unicode-rs/unicode-width#27 (comment)). This is not my project so I can not say what the goals here are but for helix only supporting wezterm would be absolutely unacceptable. You could probably find some workaround where you do unicode segmentation and then discard the second non-zero width character but that would incur a bunch of overhead and I don't really see the point. Emojis being displayed as individual codepoints is how every tui not written in rust works so at least for helix I am perfectly happy if we match that behavior (for example try looking at some multi codepoint emojis in vim/nvim). I stopped digging here at some point but what I observed was that grapheme width emoji representation like However other tui applications (like vim) don't have these issues. I am pretty sure my investigation back then showed me that grapheme segmentation was really the issue that caused vim to at least avoid significant visual artifacts. Even if it's not the fact that emojis show up as partially black boxes is definitely caused by that. Considering that grapheme segmentation has quite a bit of additional overhead (while widths need to be computed either way) my conclusion was to rip that out of the helix rendering backend when I get around to that. |
|
As a Windows Terminal user, I can tell you that the people working on that emulator concur that it is a "bug" that it doesn't support Unicode segmentation (or at least ZWJ emojis -- I can't speak for all uses of Unicode segmentation, which includes a lot of languages that require it). For instance, microsoft/terminal#8000 includes a test case for a character (single codepoint) which is intended to be rendered with 12 cells. UAX#11 (and by extension There are other gems in their issue tracker:
Supporting broken emulators because that's what they offer is a poor choice. You might instead opt to intentionally not support grapheme clusters that are not already widely supported in emulators, for some heuristic of support. Just return an error or panic when you encounter grapheme clusters that are not known to be compatible with today's state of terminal emulators. This would be the conservative approach. But if you want to do the right thing, you need to skate to where the puck is going. At least Windows Terminal is keen to fix this bug, as they have signaled their intent as far as I can tell. I'm surprised to hear that others are less interested. |
|
I don't follow windows terminal but on the unix side of things no terminal really supports this (wezterm somewhat but it's character width is basically implementation defined if you do.segment). There is not even an agreed upon width for characters and in gernal very little agrement about text width https://gitlab.freedesktop.org/terminal-wg/specifications/-/merge_requests/8. Really the only agreement seems to ge: Once you start taking unicode seriously the width of a grapheme depends on the font. A tui application has no idea about the font so it's hard to define a common notion of width. So every terminal that does support that is doing it's own thing right now with absolutely no standards. So you are going to aim at a fully moving target (this is btw also the reason why unicode-width doesn't support emoji presentation selectors, their stance is similar: beyond the single codepoint width definition in unicode there is no notion of cannonical unicode width) . What I consider important is that there are no visual artifacts in the rest of the UI. If some emoji doesnt look as nice that is secondary compared to having visual aetifacts all over the screen. Since every emulator is reasonably backwards compatible with applications that don't unicode segment (and likely will stay that way for a looong time) using the tradition approach is much more stable. You could put segmentation behind a feature flag or something. This is not my crate so I don't care either way but for helix I plan to remove unicode segmentation from the lower level rendering code |
|
It seems to me that rendering the TUI when it contains "unknown" grapheme clusters by discarding non-zero-width characters (or even just using the Unicode fallback character) is the least disruptive, even if you disagree with the overhead and don't see the point. Just because some editors render ZWJ emojis with multiple codepoints doesn't mean it's a pleasant UX. See https://lord.io/text-editing-hates-you-too/#emoji-modifiers for some particular egregious examples of this problem in action! |
|
@pascalkuthe wrote:
If we were to include switch from unicode-width to this in ratatui, it sounds like there would be a need to select between Unicode9 or Unicode14 mode in end users apps in order to render correctly for kitty and windows terminals. Have you got any ideas on config / application patterns that would to make that work well? Edit: looking at the following suggests that this might still not be quite what we need to render correctly. Why is 5/6 reasonable here as opposed to 2? #[test]
fn emoji_representation() {
// its annoying but we don't grapheme segment so each emoji must be calcultade indivudlaly
assert_eq!(str_width("👩❤️👨", Unicode9), 5);
assert_eq!(str_width("👩❤️👨", Unicode14), 6);
assert_eq!(str_width("✔️", Unicode9), 1);
assert_eq!(str_width("✔️", Unicode14), 2);
} |
e As I said in the discussion above, unicode grapheme width is not really standardized (and for example whether this compared emoji has width 1, 2, or 3 is not really defined by any standard, really the width depends on the font). Width of 5/6 really is the correct thing from the perspective of every terminal emulator (for backward compatibility reasons). The reason emojis show up as partially black boxes right now is because you actually overlap them with empty cells (created with A short example, this code https://github.com/helix-editor/helix/blob/40d7e6c9c85d4f1ce2345f6e9d59fc091243124d/helix-tui/src/buffer.rs#L412 (you have almost the same code in this repo) needs to be turned into the following: let mut chars = string.chars().peekable();
while let Some(c) = chars.next() {
let width = c.width().unwrap_or(0);
if width == 0 {
continue;
}
// `x_offset + width > max_offset` could be integer overflow on 32-bit machines if we
// change dimensions to usize or u32 and someone resizes the terminal to 1x2^32.
if width > max_x_offset.saturating_sub(x_offset) {
break;
}
let mut s = c.to_string();
while let Some(&c) = chars.peek() {
if c.width().unwrap_or(0) != 0 {
break;
}
s.push(c);
chars.next();
}
self.content[index].set_symbol(&s);
self.content[index].set_style(style);
// Reset following cells if multi-width (they would be hidden by the grapheme),
for i in index + 1..index + width {
self.content[i].reset();
}
index += width;
x_offset += width;
}helix before that change (with the char above):
and after that change:
This really relates to what I mentioned above. Unicode segmentation in a tui doesn't really make sense when most emulators don't support it (and for example kitty and wezterm don't agree on the width of some emoji. If you really do want to support that usecase then you would need to cutoff every grapheme at the second non-zero width char (but in that case my crate would also always yield a width <= 2). Like I said unicode-width does the correct thing from the perspective of the unicode standard and almost all terminal emulators. My crate just (optionally) supports respecting the grapheme representation character (which is also part of the unicode standard just some terminals support it and some don't). To specify a bit further, if you look at kitty with that patch you actually get a correct emoji that is just too wide:
You can't actually do better here. If you decrease the width to two this code will be buggy and lead to visual artifacts even on kitty despite the fact that it supports displaying these emojis. The segmentation is simply handled at the font level (but has no effect on cell width). Just to demonstrate this further if you actually paste that emoji into your shell and print it with echo it will also still have a width of 6 on kitty:
wezterm sadly actually changes cell width. I think that likely needs to be considered a bug tough (only emulator in existence that behaves that way and will break a ton of existing programs, for example kakoune is broken by this) Detection is a bit of a pain, you can do it using |
|
Iterm seems to get this right: Wezterm seems to also: |
|
I guess I don't conaider this "right". I would consider this a bug in those emulator since it breaks existing applicationa (like kakune). Maybe its possible to work around this in the rendering code somehow but I don't think so. Seems like a huge pain. I actually searched trough the wezterm repo but there doesn't seem to be any past discussion about this. I might look into filing one to move to the kitty behavior since it still looks the better than alacritty I guess while not actually breaking anything. There is no standardised notion of grapheme width. The unicode standard says "you need to look at the font to determine graphe width, we only define width for codepoints". So if I implement and emulator that uses width 3 here it would be just as correct. You could agree that emojis always have width 2 but what about other scripts? Some graphemes can get really wide or even takeup multiple lines. What size do those have? Once there is an agrees upon standard what width a grapheme actually has in a terminal that makes sense to support that can be used but until then it's essentially implementation defined. Like I said an alternative could be to discard the rest of a grapheme starting at the second non-zero-width char but that's kind if hacky (different emojis would show up as the same thing) but that is probably the most portable approach. You can experiment with what you think is the right approach for your crate regarding segmentation. I would advise against breaking all the traditional emulators (alacrity, zellij, termux. vim builtin terminal, GNOME terminal,whatever the kde terminal is called, xterm, ...) and instead would hope to see wezterm and iterm become more compatible (considering wez past stance om backwards comparability he may be open to that, wezterm used to default to unicode14 now it default to unicode9 for backwards compatability). Either way the crate K linked provides an optionally unicode14 compatible chard width calculation that is exactly identical to what wezterm/yermwiz does if you unicode segment first, calculate the width of he grapheme and then cap the result to 2. |
I don't think understand. I think you're saying almost everything is wrong, so it's better to be consistently wrong than to do the right thing? |
|
That's my takeaway, unfortunately. I went through some of the kakoune issues, and found e.g. mawww/kakoune#1447 where the maintainers indicate they aspire to add segmentation support. If the goal is to "make it work right now at all costs" then claiming 👩❤️👨 occupies 5 or 6 cells is not only logically incorrect, but also doesn't align with the intentions of developers of the applications cited as motivation for doing this in the first place.
IMHO, this is your best option and fits neatly with the "at all costs" predicate. It is also a compromise that I conceded in a comment above. Your rendering will not be perfect, but it will be both backward and forward compatible. The latter is something I believe you are dismissing unfairly. |
I am saying the approach currently used by most tui apps and emulators js based on the unicode standard. (Which is what unicore--wifth and my crate implement) Therefore everyone can agree on it (mosly except for mismatches in unicode version). Kitty does the right thing by adding support for improved rendering while keeping cell width unchanged and hence not breaking this standardized notion of width. The approach used by wezterm and Iterm is not based on any standardized notion of width. They simply chose to limit the width of a unicoee grapheme (calculated the same as other emulators do) to two. That's limit is arbitrarly chosen by the authors of those terminals (I guess because it looks good for emojis but for other graphemes this width is not correct at all). Nowhere in the unicode standard will you find any definition that would suggest thjs width. This change to the standard definition of width in a terminal breaks existing applications (since tui and emulators must agree on it) and so I consider that behavior wrong. I consider it more importent that terminal and emulator agree on a single notion of width than what that notion of width actually is. If they don't agree every single character in that line will be prone to rendering articats (wrong char in the wrong place, ghost colors,...) which is a much bigger problem then a bit of empty space behind an emoji |
As an maintainer of a text editor (which is quite similar to kakune) you miss understand that issue. They are talking about keeping the gelaheme as the smallest unit of editing. This is absolutely the right thing to do and what helix does too but has nothing to do with the way the rendering works. I pinged wez about this and he was interested about discussing it on GH so perhaohs there will be some movement on the emulator aide. |
|
Let me be clear, I don't misunderstand the issue. I have been saying since the beginning of the thread that there is a missing piece of the puzzle, and you seem to agree with as much:
Grapheme clusters are only half of the solution, but they are a necessary half. Grapheme clusters are not only the smallest unit for editing but also the smallest unit which can populate a terminal cell. This is the "M" in the "M:N" terminology. The missing half is, for any given grapheme cluster, declaring how many cells it occupies. This is the "N". While this piece is not currently standardized (and I don't know what the current state of affairs looks like) it isn't always going to be the case. In the meantime, handling the least common denominator is, as described multiple times already, either dropping additional non-zero-width characters from the grapheme cluster or replacing the entire cluster with the fallback character. Regardless, ripping out segmentation is absolutely the wrong thing to do.
The fact that none of them currently agree means that the only way to get portable output across all disagreeing implementations is to just not output the grapheme clusters in the first place. Which is what "removing additional non-zero-width characters" is all about. |
|
It does sound like @parasyte and @pascalkuthe both have some expertise beyond mine in this area. Theres' a lot of depth in this issue about what we should do to fix these issues, but I'm not sure that I can really discern an obvious next step to changing unicode rendering in a way that satisfies:
Do you have some concrete suggestions that you can agree on what would be the way to fix this?
Rule 1.... If a .... |
|
For more context for future readers, from this article https://hsivonen.fi/string-length/:
It seems like @pascalkuthe is suggesting keeping the smallest unit of editing (e.g. what happens when you press the right arrow key in the terminal) a grapheme, i.e. using unicode-segmentation but using unicode-width for display? I have to think about that more. I think the only thing that is clear to me is that we have to use this something like this for emojis: Do we know for a fact that all emojis will be displayed as 2 wide in a terminal? Will that work in all terminals? |
|
https://pkg.go.dev/github.com/rivo/uniseg#hdr-Monospace_Width is worth a look |
|
I just noticed a new crate (released 5 days ago): https://crates.io/crates/unicode-display-width by @jameslanska which might be a good "someone else's problem" type solution to the issue at hand. |
|
I thought this was relevant and interesting: |
From the README:
As far as I know, the only approach to this problem that's portable across a wide range of terminal emulators is to use the |
I'm not a user of helix or tui. But I found this helix issue and thought I'd mention it here:
helix-editor/helix#6012
Helix uses a fork of a subset of tui. Notably the core of Buffer is basically the same (AFAIK).
I'm pretty sure that this bug is shared by everything using the tui-style Buffer.
AFAIK, the reason is that unicode-width doesn't (or can't?) report the column width for emojis. See readme and issues:
https://github.com/unicode-rs/unicode-width
I won't have anything to contribute here, or any personal interest, really.
Perhaps you should get into contact with the helix team for this, as this effects both.
The text was updated successfully, but these errors were encountered: