Incorrect rendering of (some?) unicode characters

[krishnakumarg1984]

Describe the bug

There is a problem in rendering (some?) Unicode characters. This felt (perhaps) dangerous to use Wezterm as the daily driver yet, because of the circumstances under which I discovered it. Let me explain.

I was editing a C++ scientific code with wezterm just now (this file was last edited a month back using iterm2), when I noticed some really strange _ in a formula (under comments). I tend to use the native Unicode for formula involving greek characters etc because they are so concise and by looking at the comment line (in a long C++ code), I can have a quick idea of what the programmer intended to implement (atleast the intention). As I said, I had last edited the file a month ago in iterm2 and had put the Unicode comments. Now reopening the file in wezterm, when my eyes reached the equation line, I couldn't exactly remember the formula, but saw a bunch of spurious underscores. And I started deleting them! After deleting a few, I realised that something must have gone wrong, before stopping myself and realising that I am now using a new terminal emulator.

The screenshots below will tell the story better. They are open in the same tmux session live and the screenshots are taken at the same moment. Notice the commented line at the top.

 WezTerm rendering

iterm2 rendering

Environment

• OS: macOS catalina 10.15.5
• Version: wezterm 20200608-110940-3fb3a61

To Reproduce

Here are the relevant lines of code (not valid code, but just for testing purposes)

    // A*((b^2 −2bx + x^2)􏰁ω x cos(ωt) +􏰀((b^2 −2bx + x^2)Σa x +
    // 2D(2b−3x))􏰁sin(ωt))
    return intensity *
           ((b * b - 2. * b * x + x * x) * frequency * x *
              std::cos(frequency * time) +
            ((b * b - 2. * b * x + x * x) * absorption_cross_section * x +
             2. * diffusion_coefficient * (2. * b - 3. * x)) *
              std::sin(frequency * time));

[wez]

Did you configure a specific font?
What is your LANG environment set to?

[wez]

This is what I see when I run wezterm -n (with runs with the default config), open vim and paste your sample text.

I think we're missing something about your environment.

https://wezfurlong.org/wezterm/faq.html has some relevant information. tmux can complicate things; I'd suggest first verifying that things work as expected without adding another layer of emulation so that there are fewer variables to isolate.

[krishnakumarg1984]

Hmm, I didn't know that was relevant. But perhaps it really is. My font is Caskaydia Cove Nerd Font. My $LANG is en_GB.UTF-8.

local wezterm = require 'wezterm';
return {
    color_scheme = "Gruvbox Dark",
    -- font = wezterm.font("JetBrains Mono"),
    font = wezterm.font_with_fallback({
        "CaskaydiaCove Nerd Font Mono",
        -- This is searched for glyphs that aren't found in the first one
        "JetBrains Mono",
        -- This is searched for glyphs that aren't found in either of
        -- the first two
        -- "My third best font",
    }),
    font_rules = {
        -- Define a rule that matches when italic text is shown
        {
            -- If specified, this rule matches when a cell's italic value exactly
            -- matches this.  If unspecified, the attribute value is irrelevant
            -- with respect to matching.
            italic = true,

            -- Match based on intensity: "Bold", "Normal" and "Half" are supported
            -- intensity = "Normal",

            -- Match based on underline: "None", "Single", and "Double" are supported
            -- underline = "None",

            -- Match based on the blink attribute: "None", "Slow", "Rapid"
            -- blink = "None",

            -- Match based on reverse video
            -- reverse = false,

            -- Match based on strikethrough
            -- strikethrough = false,

            -- Match based on the invisible attribute
            -- invisible = false,

            -- When the above attributes match, apply this font styling
            font = wezterm.font("CaskaydiaCove Nerd Font Mono", {italic=true}),
            font = wezterm.font("JetBrains Mono", {italic=true}),
        }
    },
    -- The font size, measured in points
    font_size = 15.0,
    -- How many lines of scrollback you want to retain per tab
    scrollback_lines = 3500,

    -- Enable the scrollbar.  This is currently disabled by default.
    -- It will occupy the right window padding space.
    -- If right padding is set to 0 then it will be increased
    -- to a single cell width
    enable_scroll_bar = true,

    -- What to set the TERM variable to
    term = "xterm-256color",

    -- Constrains the rate at which output from a child command is
    -- processed and applied to the terminal model.
    -- This acts as a brake in the case of a command spewing a
    -- ton of output and allows for the UI to remain responsive
    -- so that you can hit CTRL-C to interrupt it if desired.
    -- The default value is 400,000 bytes/s.
    ratelimit_output_bytes_per_second = 400000,

    -- Constrains the rate at which the multiplexer server will
    -- unilaterally push data to the client.
    -- This helps to avoid saturating the link between the client
    -- and server.
    -- Each time the screen is updated as a result of the child
    -- command outputting data (rather than in response to input
    -- from the client), the server considers whether to push
    -- the result to the client.
    -- That decision is throttled by this configuration value
    -- which has a default value of 10/s
    ratelimit_mux_output_pushes_per_second = 10,

    -- Constrain how often the mux server scans the terminal
    -- model to compute a diff to send to the mux client.
    -- The default value is 100/s
    ratelimit_mux_output_scans_per_second = 100,

    -- If false, do not try to use a Wayland protocol connection
    -- when starting the gui frontend, and instead use X11.
    -- This option is only considered on X11/Wayland systems and
    -- has no effect on macOS or Windows.
    -- The default is true.
    enable_wayland = true,

    -- Specifies how often a blinking cursor transitions between visible
    -- and invisible, expressed in milliseconds.
    -- Setting this to 0 disables blinking.
    -- Note that this value is approximate due to the way that the system
    -- event loop schedulers manage timers; non-zero values will be at
    -- least the interval specified with some degree of slop.
    -- It is recommended to avoid blinking cursors when on battery power,
    -- as it is relatively costly to keep re-rendering for the blink!
    cursor_blink_rate = 800,

    -- Specifies the default cursor style.  various escape sequences
    -- can override the default style in different situations (eg:
    -- an editor can change it depending on the mode), but this value
    -- controls how the cursor appears when it is reset to default.
    -- The default is `SteadyBlock`.
    -- Acceptable values are `SteadyBlock`, `BlinkingBlock`,
    -- `SteadyUnderline`, `BlinkingUnderline`, `SteadyBar`,
    -- and `BlinkingBar`.
    default_cursor_style = "SteadyBlock",

    -- similarly, don't ask the macOS IME/text services to compose input
    use_ime = false,
}

[krishnakumarg1984]

@wez You were right. Indeed, it looks like the problem arises from using tmux. The rendering is fine outside of tmux. Do you know why specifically the rendering becomes difficult with tmux? This could perhaps be a missing terminfo ,termcap or color setting or something? The following is my .tmux.conf

set -g default-terminal "screen-256color"
set-option -sa terminal-overrides ',xterm-256color:RGB'
set-option -g default-shell $SHELL
setw -g xterm-keys on
set-option -g allow-rename off
set -q -g status-utf8 on  # expect UTF-8 (tmux < 2.2)
setw -q -g utf8 on

[wez]

I suspect that your tmux/environment isn't configured correctly for utf8:

https://github.com/tmux/tmux/wiki/FAQ#how-do-i-use-utf-8

If it believes the terminal is not compatible with UTF-8, any UTF-8 characters will be replaced with underscores.

That sounds like it is happening here.

In addition to troubleshooting your tmux config/environment, you probably want this:

set -g default-terminal "tmux-256color"
# workaround issue with rendering ACS in a UTF-8 context,
set-option -ga terminal-overrides ',*:enacs@:smacs@:rmacs@:acsc@'
# and force True color support on.
set-option -ga terminal-overrides ',xterm-256color:Tc'

Some background on this: tmux originally piggy-backed on the terminfo for GNU screen, but as tmux added more features that terminfo no longer really described the feature set.
One of the problematic character attributes is italics which won't work correctly in tmux unless you have a correct terminfo definition that has tmux in the name (it checks for this explicitly).

[wez]

One thing that may be happening: if you didn't have LANG set appropriately when you first started the tmux server, it may be stuck thinking you don't have UTF-8 until you kill off and restart tmux in an environment with LANG set.

[krishnakumarg1984]

Thank you for the explanation. I shall look into my tmux and other things in my environment.

The real question then is: "how is iterm2 working correctly?". iterm2 seriously covers a lot of ground on macOS. But it lacks GPU acceleration and cross-platform applicability, which really attracts me to wezterm.

[wez]

iterm2 has some logic to preset LANG to something reasonable, and I think that goes a long way to avoid some of these pitfalls. I'm looking at adding similar logic to wezterm because it is surprising. It's unfortunate that macOS doesn't set LANG appropriately by default.

[krishnakumarg1984]

The reasonable default LANG setting hasn't helped. I am still having Unicode issues on tmux.

[wez]

This is what I see with wezterm running the default configuration and with my tmux config (as shown in #220 (comment))

We established that this is an issue with your tmux configuration but it's not clear whether you restarted the tmux server or adjusted your tmux configuration since previously.

While I'd like to see you achieve success, I don't have a lot of time to spend on working to extract troubleshooting information; terse one line statements about things not working don't help me to help you; you need to do some legwork and investigation to understand why tmux isn't working for you in wezterm when perhaps it is working in other terminal emulators.
Look at the env output in both places and compare tmux info output. Poke around and see if you can detect differences that might account for the differing behavior on your system, and share what you find!

[erf]

Thank you for the explanation. I shall look into my tmux and other things in my environment.

The real question then is: "how is iterm2 working correctly?". iterm2 seriously covers a lot of ground on macOS. But it lacks GPU acceleration and cross-platform applicability, which really attracts me to wezterm.

iterm2 have GPU aceleration using metal according to this: https://iterm2.com/downloads/stable/iTerm2-3_2_0.changelog

[krishnakumarg1984]

@erf Thanks. I didn't know that iterm2 had introduced GPU acceleration. Nevertheless, wezterm is the only cross-platform GPU accelerated terminal emulator with ligature support. kitty, alacritty and iterm tick 2 out of 3 boxes, but only wezterm has all 3 aspects covered.

@wez, with wezterm 20200620-160318-e00b076c, the Unicode characters render fine within tmux as well. Perhaps we may close this issue.

[wez]

My money is on the tmux server being started with the unset LANG and that sticking around. I'm happy to call this resolved!

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.