► rendering as _ after 0.26.0 update

[kevinkirkup]

Describe the bug
After updating to the latest version (0.26.0), the ► character is no longer being rendered properly in tmux session.
It is showing up as an underscore instead _.
Outside of a tmux session, it appears to work properly.

To Reproduce
Steps to reproduce the behavior:

1. Install 0.26.0
2. start a tmux session
3. insert bullet character

Environment details

kitty 0.26.0 created by Kovid Goyal
Darwin Havok-2.local 21.6.0 Darwin Kernel Version 21.6.0: Wed Aug 10 14:25:27 PDT 2022; root:xnu-8020.141.5~2/RELEASE_X86_64 x86_64
ProductName:	macOS ProductVersion:	12.5.1 BuildVersion:	21G83
Frozen: True
Paths:
  kitty: /Applications/kitty.app/Contents/MacOS/kitty
  base dir: /Applications/kitty.app/Contents/Resources/kitty
  extensions dir: /Applications/kitty.app/Contents/Resources/Python/lib/kitty-extensions
  system shell: /bin/zsh
Loaded config files:
  /Users/kevinkirkup/.config/kitty/kitty.conf

Config options different from defaults:
font_family          PragmataPro Liga
font_size            12.0
Colors:
	color0               #6a6a6a   
	color1               #e64217   
	color10              #b2e267   
	color11              #fce95b   
	color12              #99caff   
	color13              #cd31cc   
	color14              #94edfe   
	color15              #e2e2e2   
	color2               #b2e267   
	color3               #d9c84e   
	color4               #5a7799   
	color5               #ff7df3   
	color6               #78becc   
	color7               #d5d5d4   
	color8               #323131   
	color9               #ff491a   
	cursor               #c7c7c7   
	cursor_text_color    #fffefe   
	foreground           #e1e2e1   
	selection_background #c1ddff   
	url_color            #005bbb   

Important environment variables seen by the kitty process:
	PATH                                /Applications/kitty.app/Contents/MacOS:/usr/bin:/bin:/usr/sbin:/sbin
	LANG                                en_US.UTF-8
	SHELL                               /bin/zsh
	DISPLAY                             /private/tmp/com.apple.launchd.jkK0YMA7c8/org.macosforge.xquartz:0
	USER                                xxxxxxxxxxxx

Additional context
Still seeing the same issue when running without config (kitty --config NONE)

[Liquidmantis]

I'm having the same issue, but it's not limited to that character.

Prompt in Kitty alone:

Prompt in Tmux:

[kovidgoyal]

Renders fine for me both in and out of tmux. Most likely some env var is
different, probably LANG or one of the LC_* ones, which will be causing
whatever is generating that character to generate _ instead.

Fix those and you should be fine.

[kevinkirkup]

This was definitely a change in behavior from the prior release though.
Also, not sure what the fix is here.

[kevinkirkup]

❯ echo $LANG
en_US.UTF-8
❯ echo $LC_TERMINAL
iTerm2

Looks like LC_TERMINAL is being set to iTerm2. Pretty sure I didn't do that.
Any idea what it should be set to?

[kovidgoyal]

On Mon, Aug 29, 2022 at 07:00:55AM -0700, Kevin S Kirkup wrote: This was definitely a change in behavior from the prior release though.

There is no change in rendering of that symbol. As evidenced by the fact that it works outside of tmux. What tmux does to change that symbol to underscore, god knows.

Also, not sure what the fix is here.

Check the env vars inside and outside tmux. Make them the same. Figure out which one is the culprit. Then we can try to see why tmux is not setting it correctly.

[kovidgoyal]

No idea, I dont even know what that is for. I suggest unsetting it.

[seth-schroeder]

Also seeing this bug but with the non-ascii banner of console Spacemacs 28.1. Terminal renders it fine.

[Liquidmantis]

Mine are the same in or out of tmux. Right now the only delta is updating Kitty to 0.26 this morning.

[seth-schroeder]

# MacOS, Kitty 26.0
% echo $LC_TERMINAL

% echo $LANG

% set | grep LC
MAILCHECK=60
NULLCMD=cat
READNULLCMD=more

[kovidgoyal]

There has to be something different in your env. Things don't happen by magic. Find out what it is.

env > outside-tmux.txt
tmux
env > inside-tmux.txt

diff -Naurp outside-tmux.txt inside-tmux.txt

And your LANG should be set to a UTF-8 based locale.

[Liquidmantis]

Looks like Kitty is detecting as UTF-8 somehow:

kitty 0.26.0 created by Kovid Goyal
Darwin MacMantis 21.6.0 Darwin Kernel Version 21.6.0: Sat Jun 18 17:07:22 PDT 2022; root:xnu-8020.140.41~1/RELEASE_ARM64_T6000 arm64
ProductName:	macOS ProductVersion:	12.5 BuildVersion:	21G72
Frozen: True
Paths:
  kitty: /Applications/kitty.app/Contents/MacOS/kitty
  base dir: /Applications/kitty.app/Contents/Resources/kitty
  extensions dir: /Applications/kitty.app/Contents/Resources/Python/lib/kitty-extensions
  system shell: /bin/zsh
Loaded config files:
  /Users/liquid/.config/kitty/kitty.conf

Config options different from defaults:
font_family          FiraCode Nerd Font
font_features:
{'FiraCodeNerdFontCopleteM': ('+zero',)}
font_size            14.0
macos_titlebar_color 1
window_padding_width FloatEdges(left=5.0, top=5.0, right=5.0, bottom=5.0)
Added shortcuts:
	ctrl+, →  send_text all \x1b[44;5u
	ctrl+. →  send_text all \x1b[46;5u
	ctrl+; →  send_text all \x1b[59;5u
	ctrl+i →  send_text all \x1b[105;5u
	shift+enter →  send_text all \x1b[13;2u
Changed shortcuts:
	kitty_mod+h →  send_text all \x1b[72;6u
	kitty_mod+j →  send_text all \x1b[74;6u
	kitty_mod+k →  send_text all \x1b[75;6u
	kitty_mod+l →  send_text all \x1b[76;6u
Colors:
	background           #323d43   
	color0               #3c474d   
	color1               #e68183   
	color10              #a7c080   
	color11              #d9bb80   
	color12              #83b6af   
	color13              #d39bb6   
	color14              #87c095   
	color15              #868d80   
	color2               #a7c080   
	color3               #d9bb80   
	color4               #83b6af   
	color5               #d39bb6   
	color6               #87c095   
	color7               #868d80   
	color8               #868d80   
	color9               #e68183   
	cursor               #d8caac   
	foreground           #d8caac   
	selection_background #505a60   
	selection_foreground #d8caac   

Important environment variables seen by the kitty process:
	PATH                                /Applications/kitty.app/Contents/MacOS:/usr/bin:/bin:/usr/sbin:/sbin
	LANG                                en_US.UTF-8
	SHELL                               /bin/zsh
	DISPLAY                             /private/tmp/com.apple.launchd.hpyc4ppnpF/org.xquartz:0
	USER                                liquid

[kovidgoyal]

yes, kitty sets it correctly automatically from cocoa locale settings. But check what it is inside tmux.

[kovidgoyal]

Oh and for the record, LC_TERMINAL is a hack iterm2 uses because its shell integration scripts break when sshing from another terminal. https://gitlab.com/gnachman/iterm2/-/issues/7154

You should definitely not have it in kitty. I am guessing you have it because you ran kitty from inside iterm2

[kevinkirkup]

I deleted iTerm2 so pretty sure that's not the cause.
I'll try to figure out which one is causing it for and I'll update this thread.

❯ locale
LANG="en_US.UTF"
LC_COLLATE="C"
LC_CTYPE="C"
LC_MESSAGES="C"
LC_MONETARY="C"
LC_NUMERIC="C"
LC_TIME="C"
LC_ALL=

[kovidgoyal]

OK let me know what your findings are. I will re-open since it seems to affect multiple people.

[kovidgoyal]

Aah I think I know the cause, there's a bug in 0.26 that can cause kitty
to not set LANG, which causes tmux to filter out all unicode characters
and replace them with underscores

[seth-schroeder]

That makes sense. When I export LANG or LC_ALL to en_US.UTF-8 I see the behavior from v0.25.1.

I'm going to leave the export in my .zshrc because it seems like an important thing for lots of needs.

[CPWeaver]

I can confirm this -- was about to post:

FWIW, I saw this too and explicitly setting LANG=en_US.UTF-8 before invoking tmux fixes it for me.

With or without explicitly setting the var, kitty reports the same value for LANG (detected automatically) but I see the odd rendering. Is it possible kitty used to set LANG for children and no longer does?

Tmux has logic to detect whether utf-8 is supported so perhaps something is affecting that?

[kevinkirkup]

@CPWeaver Thanks! That fixed it for me.

[seth-schroeder]

❤️ thank you @kovidgoyal

[kovidgoyal]

You are welcome, I hate regressions :)

[page-down]

The reason I didn't catch this issue before the release is that I set the env LANG config option and didn't unset LANG before running kitty --config=NONE.

@kovidgoyal
I noticed that when compiling kitty.app from the source code, LC_CTYPE=UTF-8 appears in show_kitty_env_vars. (kitty is launched by macOS launchd, e.g. Launchpad)

However this environment variable will not appear in the official release.
It looks like the fix only works for frozen builds compiled via bypy.

macOS: Fix the LC_TYPE env var being set to UTF-8 on systems in which
… the language and country code do not form a valid locale

[kovidgoyal]

@page-down: Not sure why that would be, the relevant code is in
launcher/main.c set_kitty_run_data() and that is called in all code
paths. It is used in final_env() in child.py

Both those code paths apply for kitty.app builds. Can you trace it down?
it is difficult for me to run kitty.app builds as I build on a headless
machine.

[page-down]

Can you trace it down?

Oh, I am referring only to show_kitty_env_vars. On the code path, LC_CTYPE has been removed from the child environment, so it does not affect the child process.

Apparently macOS does not set LC_CTYPE when running kitty with launchd. This value is given from Python.
When using Python installed from python.org, or Homebrew, or Conda, removing LC_*, LANG under macOS (replicate the environment running from launchd) will give you LC_CTYPE=UTF-8.

unset LC_ALL LC_CTYPE LANG; python3 -c 'import os;print(os.environ["LC_CTYPE"]);'
# UTF-8

Perhaps the relevant codes in Python are located in Python/pylifecycle.c.
Do you think this environment variable could be removed from os.environ? Maybe it will affect other UTF-8 related Python behavior.
At least make show_kitty_env_vars consistent with the frozen build.

---

@kovidgoyal
While tracking this down, I noticed lc_ctype_set_by_user, so I found another regression.
In version 0.26.1, users can no longer remove the LANG environment variable.

/usr/bin/env -iS SHELL=$SHELL USER=$USER LOGNAME=$LOGNAME HOME=$HOME TMPDIR=$TMPDIR PATH=/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin \
  /Applications/kitty.app/Contents/MacOS/kitty --config=NONE -o 'env LANG' --hold /usr/bin/env

Do you know why the LINES and COLUMNS environment variables are added when you run kitty --hold?
These two do not appear in 0.25.2, nor do they appear when --hold is not used. I can't find the relevant codes.

sh -c 'unset LINES COLUMNS; kitty --config=NONE -o "map f1 show_kitty_env_vars" --hold env'
sh -c 'unset LINES COLUMNS; kitty --config=NONE python -c "import os; print(os.environ); input()"'

[kovidgoyal]

@page-down: these issues should all be fixed now.