Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

wrong locale set by kitty #1233

Closed
cvlmtg opened this issue Dec 13, 2018 · 38 comments
Closed

wrong locale set by kitty #1233

cvlmtg opened this issue Dec 13, 2018 · 38 comments

Comments

@cvlmtg
Copy link

cvlmtg commented Dec 13, 2018

this happens on macOS Mojave with kitty 0.13.1

 ~ kitty --version
[347 13:06:22.614967] Failed to set locale with LANG: en_IT.UTF-8
kitty 0.13.1 created by Kovid Goyal

locale en_IT is wrong and it also it breaks nvim math functions, because e.g. str2float("3.3") returns "3.0"

@kovidgoyal
Copy link
Owner

the locale comes from your OS language settings. Change them and you will be fine.

@cvlmtg
Copy link
Author

cvlmtg commented Dec 13, 2018

@kovidgoyal settings are fine. this is what happens in Terminal.app

 ~ locale
LANG="it_IT.UTF-8"
LC_COLLATE="it_IT.UTF-8"
LC_CTYPE="it_IT.UTF-8"
LC_MESSAGES="it_IT.UTF-8"
LC_MONETARY="it_IT.UTF-8"
LC_NUMERIC="it_IT.UTF-8"
LC_TIME="it_IT.UTF-8"
LC_ALL=

this is what happens in kitty

 ~ locale
LANG=
LC_COLLATE="C"
LC_CTYPE="en_US.UTF-8"
LC_MESSAGES="C"
LC_MONETARY="C"
LC_NUMERIC="C"
LC_TIME="C"
LC_ALL=

@kovidgoyal
Copy link
Owner

that just means terminal.app is using a different technique to deduce the locale, which happens to work on your system.

@kovidgoyal
Copy link
Owner

oh and the relevant function in kitty code is cocoa_get_lang() if you think there is a bug in that, feel free to submit a patch, I can see no bug.

@eelcocramer
Copy link

eelcocramer commented Jun 21, 2019

I also got this issue and fixed it by setting LANG to en_US in my shell configuration (config.fish, .bashrc, etc).

cocoa_get_lang() resolved en_NL.UTF-8 on my system. I also tried setting LANG to en_US.UTF-8 but I got the same results as mentioned in the initial issue.

@tallakt
Copy link

tallakt commented Nov 1, 2019

I'm having this as well. It's pretty annoying, if I'm not able to fix this, I'll have to find another terminal. I did not find a fix for this anywhere also, so perhaps some action should be taken on behaf of kitty devs? iTerm works out of the box, as does the built in terminal app

@wookayin
Copy link

As of 0.16.0, Kitty sets LC_CTYPE='UTF-8' which is wrong. (iTerm2, Terminal do not)

@kovidgoyal
Copy link
Owner

No, it doesn't. grep the codebase for LC_CTYPE

@KimonHoffmann
Copy link

I also experience the invalid setting of LANG (en_DE in my case) and LC_TYPE (UTF-8) with kitty v0.17.4 on macOS. It is worth mentioning that the latter appears to be an implicit consequence of the former and is probably not an error on its own.

Apparently for some applications NSLocale gives strange results and I have not yet been able to pinpoint how and why it happens. In a "Swift Playground" in Xcode I observed the same behavior, while I got the correct values from a minimal Objective-C or Swift CLI application.

I have now helped myself with a dirty workaround, by explicitly setting the locale I want via:

defaults write net.kovidgoyal.kitty OSLocale -string de_DE

After doing that (and restarting kitty) the LANG variable contains the correct information and the stray and ill formed LC_CTYPE variable is gone.

@KimonHoffmann
Copy link

Even though the linked comments refer to iOS instead of macOS, they might still point in the correct direction:

@kovidgoyal
Copy link
Owner

According to those posts currentLocale should return only a locale kitty is localized for. kitty is not localized for de_DE, so I dont see how the issue could be that.

@KimonHoffmann
Copy link

You are right: It shouldn't.

This matches the problem description for me and all of the posters above that it doesn't. It returns en_* (en_DE in my case), which is not a valid locale identifier.

It does only return the correct identifier, when forced to do so via the defaults setting described above.

@kovidgoyal
Copy link
Owner

Well if you have a suggestion that works better, I'm all ears, as I said before, the relevant function is cocoa_get_lang() which is just a wrapper around currentLocale(). A possible alternative is to use preferredLanguages() instead of currentLocale(). Or alternately to validate the locale returned by currentLocale() by using newlocale() to construct a locale based on it.

@KimonHoffmann
Copy link

I already played around with an alternate implementation of cocoa_get_lang() to no avail.

In the "Swift Playground" that exhibits the same behavior as kitty, I also evaluated perferredLanguages, which does contain the desired identifier, but in practice it might be difficult to determine, which array element is the one to use.

What bothers me is that even after a few hours of investigating I don't yet have the slightest clue, what causes macOS to behave that way for certain applications.

@kovidgoyal
Copy link
Owner

kovidgoyal commented Jun 18, 2020 via email

@KimonHoffmann
Copy link

Yes, I am aware of that, but in the case of the affected "Swift Playground" this is not the case.
The list of preferred languages has two entries and the first is en.

In an application that does not exhibit the problematic behavior, on my local system NSLocale produces the following values:

  • NSLocale.autoupdatingCurrent.regionCode: DE
  • NSLocale.autoupdatingCurrent.languageCode: de
  • NSLocale.autoupdatingCurrent.identifier: de_DE
  • NSLocale.preferredLanguages: [ "de-DE" ]

In an application that does exhibit the problematic behaviour, the values are:

  • NSLocale.autoupdatingCurrent.regionCode: DE
  • NSLocale.autoupdatingCurrent.languageCode: en
  • NSLocale.autoupdatingCurrent.identifier: en_DE
  • NSLocale.preferredLanguages: [ "en", "de-DE" ]

This leads me to another suspicion, namely that for applications affected by this behavior the locale information is not somehow restricted - as suspected in the comments linked earlier - but instead that the language code en for some reason gets preferential treatment.

Please note: Observing this behavior is notoriously difficult, because any context, in which the environment contains a properly set LANG variable makes the problem go away. E.g. just starting kitty from the macOS Terminal application (where LANG is set to de_DE.UTF-8) via

$ /Applications/kitty.app/Contents/MacOS/kitty 

Does not exhibit the problem, while starting it via

$ env -i  /Applications/kitty.app/Contents/MacOS/kitty

or via

$ open /Applications/kitty.app

does.

@KimonHoffmann
Copy link

KimonHoffmann commented Jun 18, 2020

Found it!
Removing the following key value pair from Info.plist solves the problem:

<key>CFBundleDevelopmentRegion</key>
<string>English</string>

@kovidgoyal
Copy link
Owner

I think the best solution is to check the validity of the locale in
cocoa_get_lang() and if it is not valid, to return en_US. This can be
done via newlocale()

@KimonHoffmann
Copy link

See my latest comment above.
With this change cocoa_get_locale() works as advertised.

I attached a patch that does not change the behaviour as it is right now, but hopefully makes the function more resilient in general. Feel free to use it if you like.

patch-cocoa_get_locale.txt

@kovidgoyal
Copy link
Owner

kovidgoyal commented Jun 18, 2020 via email

@kovidgoyal
Copy link
Owner

kovidgoyal commented Jun 18, 2020 via email

@KimonHoffmann
Copy link

KimonHoffmann commented Jun 18, 2020

I've also looked at various applications and what is "correct" (for those that have it) is not that clear cut. Some have en some have English could be that different versions of Xcode place different values there.

Removing it should make no difference as this setting corresponds to a setting in Xcode (Edit Scheme -> Run -> Options) where you can set the application locale to explicitly differ from the system locale, mostly for debugging purposes.

Screenshot

@KimonHoffmann
Copy link

apparently either creating an empty file at Resources/empty.lproj
or adding CFBundleAllowMixedLocalizations might also work. Can you
confirm?

I'll check that out and report back.

@KimonHoffmann
Copy link

apparently either creating an empty file at Resources/empty.lproj or adding CFBundleAllowMixedLocalizations might also work. Can you confirm?

Yes, I can confirm that both options also have the desired effect, both independently and combined.

@KimonHoffmann
Copy link

I just investigated a little bit further and realized that one statement I made earlier is not quite correct:

While all the workarounds and solutions discussed fix the incorrect setting of the LANG environment variable, the erroneous LC_CTYPE variable is still present and I have no real clue where it comes from.

@kovidgoyal
Copy link
Owner

That will be coming from the shell, kitty itself does not set LC_CTYPE,
only LANG.

@KimonHoffmann
Copy link

Yes, I didn't want to imply that kitty is explicitly setting it.
But since this variable does not get set, when using other terminal emulators[*] I'm suspecting it is some kind of side effect of some other action being performed at initialization time.

[*] other emulators tested:

  • Terminal.app
  • Alacritty
  • xterm (does not even set LANG)

@kovidgoyal
Copy link
Owner

Like what?? The only way kitty has to communicate with child processes
is environment variables and stdin. With the standard shell startup
stdin is not used. That leaves environment variables, and kitty only
sets LANG that could possibly be relevant to locales, and that too only
on macOS, because of #90

In fact iTerm for example, actually does set LC_CTYPE https://github.com/gnachman/iTerm2/blob/faf3b633cee9d8749ca2f02a5b3226af4f93411f/sources/PTYSession.m#L1656

@KimonHoffmann
Copy link

I followed the LC_CTYPE rabbit hole a bit further and it seems to be a behavior inherent to python 3 on macOS (at least on macOS Catalina).

See:

$ env -i /usr/bin/python3
Python 3.7.3 (default, Apr 24 2020, 18:51:23)
[Clang 11.0.3 (clang-1103.0.32.62)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> os.environ
environ({'__CF_USER_TEXT_ENCODING': '0x1F5:0x0:0x3', 'SDKROOT': '/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk', 'CPATH': '/usr/local/include', 'LIBRARY_PATH': '/usr/local/lib', '__PYVENV_LAUNCHER__': '/Applications/Xcode.app/Contents/Developer/usr/bin/python3', 'LC_CTYPE': 'UTF-8'})

Compared to:

$ env -i /usr/bin/python
Python 2.7.16 (default, Apr 17 2020, 18:29:03)
[GCC 4.2.1 Compatible Apple LLVM 11.0.3 (clang-1103.0.29.20) (-macos10.15-objc- on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> os.environ
{'__CF_USER_TEXT_ENCODING': '0x1F5:0x0:0x3', 'VERSIONER_PYTHON_VERSION': '2.7'}

According to various discussions around this topic macOSs (or python on macOSs) setting of LC_CTYPE=UTF-8 is actually conforming to POSIX, but a lot of tools have trouble with it, because they expect the locale.encoding format.

It might be an option to apply some kind of heuristic to determine whether LC_CTYPE in the environment should be considered an artifact (and not an intentional choice made by the user) that should be remedied by unsetting it.

@kovidgoyal
Copy link
Owner

kovidgoyal commented Jun 24, 2020 via email

@KimonHoffmann
Copy link

Yes, it does:

$ env -i ./kitty.app/Contents/MacOS/kitty +runpy 'import os, pprint; pprint.pprint(dict(os.environ))'
{'LC_CTYPE': 'UTF-8', '__CF_USER_TEXT_ENCODING': '0x1F5:0x0:0x3'}

@kovidgoyal
Copy link
Owner

Hmm not sure what a good solution is here then. Since UTF-8 is a valid
value for LC_CTYPE and we have no way to know if it existed before
python inserted it, there is no real way to filter it out.

@KimonHoffmann
Copy link

Yes, I agree that it's tricky.

One heuristic I could imagine would be (only on macOS) to see, whether

  1. LC_CTYPE is equal to UTF-8 and
  2. LC_ALL is unset (or alternatively no other LC_* variable is set)

kovidgoyal added a commit that referenced this issue Jun 24, 2020
… the language and country code do not form a valid locale

Fixes #1233
@kovidgoyal
Copy link
Owner

kovidgoyal commented Jun 24, 2020 via email

kovidgoyal added a commit that referenced this issue Jun 24, 2020
… the language and country code do not form a valid locale

Fixes #1233
@KimonHoffmann
Copy link

Yes, I can confirm that the erroneous LC_CTYPE setting is gone.

And i really like the way you solved it, because no guessing is involved.

Luflosi added a commit to Luflosi/kitty that referenced this issue May 3, 2021
The spotlight search in the "Help" menu of the menu bar did not appear when the system language is set to a language other than English. macOS magically adds the search box when the text in that menu matches the translation of "Help" in the system language. On my German system for example replacing "Help" with "Hilfe" will make the search box appear again. This is however not a good solution for kitty as it would require implementing localization support for all languages. Removing `CFBundleAllowMixedLocalizations=True` from `setup.py` also brings back the search box but that may break the locale again. The line was added in  8818b5d to fix an issue related to kovidgoyal#1233.
Luckily I found a seemingly undocumented function that helps us out here, which does not rely the text comparison magic.
@kaiz0711
Copy link

kaiz0711 commented Oct 5, 2021

locale
if use zsh
export LANGUAGE=en_US.UTF-8
export LC_ALL=en_US.UTF-8
export LANG=en_US.UTF-8
export LC_TYPE=en_US.UTF-8

@garrettmaring
Copy link

garrettmaring commented Apr 11, 2023

Still seeing non-en_US after setting above locales on kitty 27.1 on M1 mac

Screenshot 2023-04-11 at 1 04 35 PM

@garrettmaring
Copy link

Still seeing non-en_US after setting above locales on kitty 27.1 on M1 mac

Screenshot 2023-04-11 at 1 04 35 PM

the fix by was

symbol_map

# "Nerd Fonts - Pomicons"
symbol_map  U+E000-U+E00D Symbols Nerd Font

# "Nerd Fonts - Powerline"
symbol_map U+e0a0-U+e0a2,U+e0b0-U+e0b3 Symbols Nerd Font

# "Nerd Fonts - Powerline Extra"
symbol_map U+e0a3-U+e0a3,U+e0b4-U+e0c8,U+e0cc-U+e0d2,U+e0d4-U+e0d4 Symbols Nerd Font

# "Nerd Fonts - Symbols original"
symbol_map U+e5fa-U+e62b Symbols Nerd Font

# "Nerd Fonts - Devicons"
symbol_map U+e700-U+e7c5 Symbols Nerd Font

# "Nerd Fonts - Font awesome"
symbol_map U+f000-U+f2e0 Symbols Nerd Font

# "Nerd Fonts - Font awesome extension"
symbol_map U+e200-U+e2a9 Symbols Nerd Font

# "Nerd Fonts - Octicons"
symbol_map U+f400-U+f4a8,U+2665-U+2665,U+26A1-U+26A1,U+f27c-U+f27c Symbols Nerd Font

# "Nerd Fonts - Font Linux"
symbol_map U+F300-U+F313 Symbols Nerd Font

#  Nerd Fonts - Font Power Symbols"
symbol_map U+23fb-U+23fe,U+2b58-U+2b58 Symbols Nerd Font

#  "Nerd Fonts - Material Design Icons"
symbol_map U+f500-U+fd46 Symbols Nerd Font

# "Nerd Fonts - Weather Icons"
symbol_map U+e300-U+e3eb Symbols Nerd Font

# Misc Code Point Fixes
symbol_map U+21B5,U+25B8,U+2605,U+2630,U+2632,U+2714,U+E0A3,U+E615,U+E62B Symbols Nerd Font

which fixes the problem despite showing deprecation warnings for the symbol_map key (latest versions of kitty, macos)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

8 participants