TUI Unicode Discussion

[tig]

Unicode and TUI is a very interesting topic. Terminal.Gui apps want to use a broad range of Unicode features. This thread is a master discussion thread for Terminal.Gui unicode capabilities. Ideas, issues, designs, etc...

We're talking about features such as:

• Emoji
• Combining marks
• Grapheme
• Glyphs that are wider than one column
• etc...

Terminal.Gui v2 has much better support for these things than v1, but there is still a long way to go. Related issues:

Specs

• http://unicode.org/reports/tr11/

Relevant Terms/Definitions

• String length- https://hsivonen.fi/string-length/ - "String length is about as meaningful a measurement as string height”

• Column width

  • We currently use (in v2): this wcwidth package: https://github.com/spectreconsole/wcwidth
  • https://man7.org/linux/man-pages/man3/wcswidth.3.html
  • https://pkg.go.dev/github.com/rivo/uniseg#hdr-Monospace_Width - there is no standard for the calculation of such widths, and this package differs from wcswidth() in a number of ways, presumably to generate more visually pleasing results.

• Rendering v Editing - The requirements for editing vs. just displaying text are different. For example, if a user wants to enter and edit a grapheme consisting of a and two accute accents (/u0301/u0301) that is 3 "keystrokes". But what is displayed is a single-width glyph:
  

  • If they backspace, it should take 3 backspace to remove the entire glyph.
  • However, in displaying such a glyph, it should take only a single column and should NOT cause glpyhs to the right to be pushed to the right (as Windows Terminal currently does).

• Unicode segmentation -

• Combining Marks -

• Grapheme Cluster - one user-perceived character

  • There is no standardised notion of grapheme width. The unicode standard says "you need to look at the font to determine graphe width, we only define width for codepoints". So if I implement and emulator that uses width 3 here it would be just as correct. You could agree that emojis always have width 2 but what about other scripts? Some graphemes can get really wide or even takeup multiple lines. What size do those have? Once there is an agrees upon standard what width a grapheme actually has in a terminal that makes sense to support that can be used but until then it's essentially implementation-defined.

• Emoji -

• 'COOKED_READ` in Windows Terminal - The input mode that CMD and legacy versions of PowerShell (without PSReadline) use. It's the mode there the console host handles producing a single line of input for the client application and only returns it when the user mashes Enter.

[tig]

Here are some proposed tenets for Terminal.Gui related to Unicode support:

Terminal.Gui Unicode Handling Tenets (Unless you know better ones):

• Unicode Doesn't Corrupt - If some glyph, emoji, combining mark, or other Unicode thing doesnt look as nice that is secondary to having visual artifacts that corrupt the rest of the TUI experience.
• Skate to Where the Puck is Going - Few terminal emulators do Unicode right today. There are few standards. Terminal.Gui has an opportunity to help lead given it enables cross-platform TUI apps. Our API design is biased towards supporting the ideal and using our ConsoleDriver abstraction layer to enable apps to work well cross-platform.
• Most Common Denominator - Related to the above, Terminal.Gui apps support the richest level of Unicode support possible on whatever terminal emulator is the gold standard for Unicode behavior. (Right now it's not clear what terminal is the gold standard; it seems the Windows Terminal team has a great approach, but the implementation lags).

[tig]

Great Windows Terminal issues/discussions:

• The text layout code is expecting the quantity of glyphs is equal to the cells allocated for each line
• Funadmental Flaws inherent in the design of AtlasEngine microsoft/terminal#16132
• [Epic] Text Buffer rewrite  microsoft/terminal#8000
• Add emoji support to Windows Console
• Feature Request: Finish full unicode support (M:N cell rendering, ZWJ?) microsoft/terminal#1472
• Give API to measure the space that a string occupies
• ReadConsoleOutputW doesn't work with surrogate pairs (the buffer layers thread) microsoft/terminal#10810

[tig]

Linux Terminal issues/discussions:

• Terminal WG
  • https://gitlab.freedesktop.org/terminal-wg/specifications/-/merge_requests/8 - The only agreement seems to ge: Once you start taking unicode seriously the width of a grapheme depends on the font. A tui application has no idea about the font so it's hard to define a common notion of width.
• alacrity, zellij, termux. vim builtin terminal, GNOME terminal,whatever the kde terminal is called, xterm,
• wezterm
  • sadly actually changes cell width. I think that likely needs to be considered a bug tough (only emulator in existence that behaves that way and will break a ton of existing programs, for example kakoune is broken by this)
• Kitty -
  • "Kitty does the right thing by adding support for improved rendering while keeping cell width unchanged and hence not breaking this standardized notion of width."

[tig]

What other libraires are thinking:

• Ratatui is a crate for cooking up terminal user interfaces in rust. It is a lightweight library that provides a set of widgets and utilities to build complex rust TUIs. Ratatui was forked from the Tui-rs crate in 2023 in order to continue its development.

  • Buffer: unicode-width and emojis ratatui-org/ratatui#75

• The Julia Programming Language

  • https://github.com/JuliaStrings/utf8proc

• Go

  • https://pkg.go.dev/github.com/rivo/uniseg#section-readme

• Rust

  • https://crates.io/crates/unicode-display-width

[tig]

Terminal.Gui Text Handling Requirements

• Word Boundaries - Word boundaries are used in several different contexts. The most familiar ones are selection (double-click mouse selection), cursor movement ("move to next word" control-arrow keys), and the dialog option "Whole Word Search" for search and replace. They are also used in database queries, to determine whether elements are within a certain number of words of one another. Searching may also use word boundaries in determining matching items. Terminal.Gui needs APIs to determine word boundaries within strings.

• Sentence Boundaries - Sentence boundaries are often used for triple-click or some other method of selecting or iterating through blocks of text that are larger than single words. They are also used to determine whether words occur within the same sentence in database queries. Terminal.Gui needs APIs to determine sentence boundaries within strings.

• Line Breaking - Line breaking, also known as word wrapping, is the process of breaking a section of text into lines such that it will fit in the available width of a page, window or other display area. Terminal.Gui needs APIs to determine where a string may or may not be broken and where it must be broken (for example after newline characters).

• Glyph Width - Terminals use a fixed width for each glyph displayed. Some glyphs such as emojis or glyphs found in Asian and other languages may take up more than one terminal cell. Terminal.Gui needs APIs to determine the number of cells a string will take up when displayed.