Text-as-code-points enhancement does not work with Cyrillic keyboard layout #6167

unxed · 2023-04-09T11:08:13Z

Describe the bug
If "report associated text" bit (0b10000) is enabled, associated text reported only if English keyboard layout is selected. No associated text is reported for Russian (Cyrillic) keyboard layout.

To Reproduce
Steps to reproduce the behavior:

Enable all enhancements with \x1b[=31u
Select Russian keyboard layout
Press ф key (same as a in English keyboard layout)
Keypress event code will be \x1b[1092::97;u, where 1092 is unicode ф and 97 is corresponding latin a. But no "text-as-code-point" field at all!
Switch to English keyboard
Press a key
Keypress event code will be \x1b[97;;97u so "text-as-code-point" field is on it's place.

Environment details

kitty 0.27.1 () created by Kovid Goyal
Linux rcnb 5.19.0-38-generic #39~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Fri Mar 17 21:16:15 UTC 2 x86_64
Linux Mint 21 Vanessa rcnb /dev/tty

DISTRIB_ID=LinuxMint
DISTRIB_RELEASE=21
DISTRIB_CODENAME=vanessa
DISTRIB_DESCRIPTION="Linux Mint 21 Vanessa"
Running under: X11
Frozen: True
Paths:
  kitty: /home/unxed/.local/kitty.app/bin/kitty
  base dir: /home/unxed/.local/kitty.app/lib/kitty
  extensions dir: /home/unxed/.local/kitty.app/lib/kitty-extensions
  system shell: /bin/bash
Loaded config files:
  /etc/xdg/kitty/kitty.conf
  /home/unxed/.config/kitty/kitty.conf

Config options different from defaults:
update_check_interval 0.0
Removed shortcuts:
	kitty_mod+left →  previous_tab
	kitty_mod+right →  next_tab

Important environment variables seen by the kitty process:
	PATH                                /home/unxed/.local/kitty.app/bin:/home/unxed/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
	LANG                                ru_RU.UTF-8
	SHELL                               /bin/bash
	DISPLAY                             :0
	USER                                unxed
	XDG_CONFIG_DIRS                     /etc/xdg/xdg-cinnamon:/etc/xdg
	XDG_SESSION_PATH                    /org/freedesktop/DisplayManager/Session0
	LC_ADDRESS                          ru_RU.UTF-8
	LC_NAME                             ru_RU.UTF-8
	LC_MONETARY                         ru_RU.UTF-8
	XDG_SEAT                            seat0
	XDG_SESSION_DESKTOP                 cinnamon
	XDG_SESSION_TYPE                    x11
	XDG_GREETER_DATA_DIR                /var/lib/lightdm-data/unxed
	LC_PAPER                            ru_RU.UTF-8
	XDG_CURRENT_DESKTOP                 X-Cinnamon
	XDG_SEAT_PATH                       /org/freedesktop/DisplayManager/Seat0
	XDG_SESSION_CLASS                   user
	LC_IDENTIFICATION                   ru_RU.UTF-8
	LC_TELEPHONE                        ru_RU.UTF-8
	LC_MEASUREMENT                      ru_RU.UTF-8
	XDG_VTNR                            7
	XDG_SESSION_ID                      c2
	XDG_RUNTIME_DIR                     /run/user/1000
	LC_TIME                             ru_RU.UTF-8
	XDG_DATA_DIRS                       /usr/share/cinnamon:/usr/share/gnome:/home/unxed/.local/share/flatpak/exports/share:/var/lib/flatpak/exports/share:/usr/local/share:/usr/share
	LC_NUMERIC                          ru_RU.UTF-8

Additional context
Still reproduces with kitty --config NONE

The text was updated successfully, but these errors were encountered:

kovidgoyal · 2023-04-09T11:30:52Z

Works for me, your version of kitty is extremely old, I suggest updating
from https://sw.kovidgoyal.net/kitty/binary/

unxed · 2023-04-09T11:54:52Z

curl -L https://sw.kovidgoyal.net/kitty/installer.sh | sh /dev/stdin installs kitty 0.27.1 for me. This issue still reproduces with it. It even reproduces on nightly build!

kovidgoyal · 2023-04-09T13:46:18Z

Then I am going to need more details to reproduce with. What layout are you using exactly, and what command did you use to switch to it?

unxed · 2023-04-09T15:36:10Z

Thanks for taking the time to look into this issue!

I have two layouts installed: English (US) and Russian. I use key combination Alt+Shift to switch between them.

kovidgoyal · 2023-04-09T16:28:44Z

That's something distro specific, I need to know what exact layout is
being used, what arguments to give the setxkbmap command to switch to
the layout where you observe the issue.

kovidgoyal · 2023-04-09T16:40:30Z

Running

setxkbmap ru; kitty +kitten show_key -m kitty

shows the following output on pressing the a key, as expected.

Press any keys - Ctrl+C or Ctrl+D will terminate
ф PRESS 
CSI 1092 :  : 97 u
Alternate key: a 

ф RELEASE 
CSI 1092 :  : 97 ; 1 : 3 u
Alternate key: a

kovidgoyal · 2023-04-09T17:16:25Z

Never mind I see what the problem is, I was confused by an unrelated bug in the show_key kitten

unxed · 2023-04-09T18:13:53Z

Thanks a lot! Unfortunately I am unable to build kitty from sources (I get "tools/crypto/crypto.go:9:2: package crypto/ecdh is not in GOROOT (/usr/lib/go-1.18/src/crypto/ecdh)" error and not so familiar with golang to solve it), so I will wait for fresh nightly version to build and recheck.

unxed · 2023-04-09T22:03:39Z

In the nightly build, the error is no longer reproduced, thank you very much!

kovidgoyal · 2023-04-10T05:44:07Z

You need go >= 1.20 for kitty

unxed · 2024-04-05T06:01:57Z

Sill problems. Steps to reproduce:

Install Ubuntu 23.10
Run it (use default Wayland session)
Install two keyboard layouts, English and Russian
Install (by apt install, 0.26.5-3ubuntu2) and run kitty
Enable mode 31 for progressive keyboard enhancement in kitty
Switch to Russian layout
Press some letter keys
Usual unicode text will appear, not escape sequences as it should be with byte 0b1000 (8) Report all keys as escape codes set to 1
Also try pressing keys in Russian layout together with Alt. Normal kitty escape sequences will appear. Still "text as code points" is reported in English, but it should be in Russian instead.

I've recorded a video for you showing this behavior:

simplescreenrecorder-2024-04-05_07.58.19.mp4

kovidgoyal · 2024-04-05T06:13:24Z

I am afraid I have zero interest in debugging issues in ancient versions of kitty.

unxed · 2024-04-05T06:26:45Z

Retested with 0.33.1
Behavior is just the same

kovidgoyal · 2024-04-05T06:32:59Z

The only way that can happen is if your system is not sending actual keypresses but instead just plain text for those key events, some IMEs for instance will do that. Easily verified by running kitty with --debug-input. In which case fix your system to not do that.

unxed · 2024-04-05T06:51:41Z

Here is a keyboard log (with --debug-input) of just the same testing.

log.txt

I know nothing about IME, it's just a clean Ubuntu 23.10 setup with no additional tweaks or settings, just two keyboard layouts enabled. Also other terminals that can do enhanced key reporting, as far2l's internal VT, do report keys as expected.

If protocol is not working out of the box on the one of the most popular operating systems, maybe there is some problem with protocol and/or implementation?

kovidgoyal · 2024-04-05T07:04:39Z

Look at the log, it is full of IME events, as I thought. And I cant see
anything wrong there. All the press and release events are followed by
sent encoded key to child. Rather than using your own code to test
things, I suggest you use

kitten show-key -m kitty

Here is the relevant section of the log, immediately after turning on kitty keyboard mode.
It shows every press and release event being sent to the child encoded.

Set key encoding flags to: 31
on_focus_change: window id: 0x1 focused: 0
on_focus_change: window id: 0x1 focused: 1
on_key_input: glfw key: 0x0 native_code: 0x0 action: PRESS mods: none text: '￑ﾄ' state: 2 committed pre-edit text: ￑ﾄ
Release xkb_keycode: 0x1e clean_sym: Cyrillic_ef mods: none glfw_key: 1092 (￑ﾄ) xkb_key: 1734 (Cyrillic_ef) alternate_key: 97 (aﾄ)
on_key_input: glfw key: 0x0 native_code: 0x0 action: PRESS mods: none text: '￑ﾋ' state: 2 committed pre-edit text: ￑ﾋ
Release xkb_keycode: 0x1f clean_sym: Cyrillic_yeru mods: none glfw_key: 1099 (￑ﾋ) xkb_key: 1753 (Cyrillic_yeru) alternate_key: 115 (sﾋ)
on_key_input: glfw key: 0x0 native_code: 0x0 action: PRESS mods: none text: '￐ﾲ' state: 2 committed pre-edit text: ￐ﾲ
Release xkb_keycode: 0x20 clean_sym: Cyrillic_ve mods: none glfw_key: 1074 (￐ﾲ) xkb_key: 1751 (Cyrillic_ve) alternate_key: 100 (dﾲ)
Press xkb_keycode: 0x38 clean_sym: Alt_L composed_sym: Alt_L mods: none glfw_key: 57443 (LEFT_ALT) xkb_key: 65513 (Alt_L)
on_key_input: glfw key: 0xe063 native_code: 0xffe9 action: PRESS mods: alt text: '' state: 0 sent encoded key to child: ^[ [ 5 7 4 4 3 ; 3 u
Press xkb_keycode: 0x1e clean_sym: Cyrillic_ef composed_sym: Cyrillic_ef active_unknown_mods: Meta mods: alt glfw_key: 1092 (￑ﾄ) xkb_key: 1734 (Cyr
illic_ef) alternate_key: 97 (aﾄ)
on_key_input: glfw key: 0x444 native_code: 0x6c6 action: PRESS mods: alt text: '' state: 0 sent encoded key to child: ^[ [ 1 0 9 2 : : 9 7 ; 3 u
Release xkb_keycode: 0x1e clean_sym: Cyrillic_ef mods: alt glfw_key: 1092 (￑ﾄ) xkb_key: 1734 (Cyrillic_ef) alternate_key: 97 (aﾄ)
on_key_input: glfw key: 0x444 native_code: 0x6c6 action: RELEASE mods: alt text: '' state: 0 sent encoded key to child: ^[ [ 1 0 9 2 : : 9 7 ; 3 :
3 u
Press xkb_keycode: 0x1f clean_sym: Cyrillic_yeru composed_sym: Cyrillic_yeru active_unknown_mods: Meta mods: alt glfw_key: 1099 (￑ﾋ) xkb_key: 1753
(Cyrillic_yeru) alternate_key: 115 (sﾋ)
on_key_input: glfw key: 0x44b native_code: 0x6d9 action: PRESS mods: alt text: '' state: 0 sent encoded key to child: ^[ [ 1 0 9 9 : : 1 1 5 ; 3 u
Release xkb_keycode: 0x1f clean_sym: Cyrillic_yeru mods: alt glfw_key: 1099 (￑ﾋ) xkb_key: 1753 (Cyrillic_yeru) alternate_key: 115 (sﾋ)
on_key_input: glfw key: 0x44b native_code: 0x6d9 action: RELEASE mods: alt text: '' state: 0 sent encoded key to child: ^[ [ 1 0 9 9 : : 1 1 5 ; 3
: 3 u
Press xkb_keycode: 0x20 clean_sym: Cyrillic_ve composed_sym: Cyrillic_ve active_unknown_mods: Meta mods: alt glfw_key: 1074 (￐ﾲ) xkb_key: 1751 (Cyr
illic_ve) alternate_key: 100 (dﾲ)
on_key_input: glfw key: 0x432 native_code: 0x6d7 action: PRESS mods: alt text: '' state: 0 sent encoded key to child: ^[ [ 1 0 7 4 : : 1 0 0 ; 3 u
Release xkb_keycode: 0x20 clean_sym: Cyrillic_ve mods: alt glfw_key: 1074 (￐ﾲ) xkb_key: 1751 (Cyrillic_ve) alternate_key: 100 (dﾲ)
on_key_input: glfw key: 0x432 native_code: 0x6d7 action: RELEASE mods: alt text: '' state: 0 sent encoded key to child: ^[ [ 1 0 7 4 : : 1 0 0 ; 3
: 3 u
Release xkb_keycode: 0x38 clean_sym: Alt_L mods: alt glfw_key: 57443 (LEFT_ALT) xkb_key: 65513 (Alt_L)
on_key_input: glfw key: 0xe063 native_code: 0xffe9 action: RELEASE mods: none text: '' state: 0 sent encoded key to child: ^[ [ 5 7 4 4 3 ; 1 : 3 u

kovidgoyal · 2024-04-05T07:13:24Z

And you can see in the log that your Wayland compositor is converting key press events to key press events with key code zero, for example the line:

on_key_input: glfw key: 0x0 native_code: 0x0 action: PRESS mods: none text: '￑ﾄ' state: 2 committed pre-edit text: ￑ﾄ

As usual, this will be a wayland specific problem and my standard advice is to switch to X11. As for why the ru layout does this in GNOME, I have no clue, it will need investigation.

unxed · 2024-04-05T07:28:02Z

Issue with associated text being reported as English, not current keyboard layout (Russian), is still reproducing under X11.

kovidgoyal · 2024-04-05T07:45:18Z

And I just tested foot under Wayland with the RU layout. It also does
not report press events only release events and no text. So definitely
an issue with whatever IME system GNOME is using for the RU layout.

kovidgoyal · 2024-04-05T08:00:55Z

Issue with associated text being reported as English, not current keyboard layout (Russian), is still reproducing under X11.

What's the issue, exactly? kitty will report whatever text the system sends to it, it does not translate russian to english on the fly. You can see it again with --debug-input

unxed · 2024-04-05T08:10:27Z

It should be documented in protocol definition then. Not "textual representation of pressed keys", but "same data that OS sends to terminal app, unmodified".

Well, such protocol will never become somewhat popular or even multiplatform. Things may change depending on compositor, DE, OS version, whatever.

This is not how standards work.

kovidgoyal · 2024-04-05T08:18:57Z

On Fri, Apr 05, 2024 at 01:10:49AM -0700, unxed wrote: It should be documented in protocol definition then. Not "textual representation of pressed keys", but "whatever OS sends to terminal app, unmodified". Well, such protocol will never become somewhat popular or even multiplatform. Things may change depending on compositor, de, os version, whatever. This is not how standards work.

Sigh. The terminal can only send on what the "system" sends to it. Terminals are not magic. You need to go complain to whatever idiot in GNOME decided to implement the russian layout via IME instead of key events. *No software* that supports IME running under GNOME can report key events for key presses in the russian layout. This is squarely an issue with GNOME.

kovidgoyal · 2024-04-05T09:27:29Z

And here is an option you can use in kitty to disable IME which will allow your use case to work, regardless of GNOME and its shenanigans. 2c4ffba

unxed · 2024-04-09T16:57:31Z

Thank you! At a minimum, this behavior is now documented and the reasons for it are clear. And thanks for the new option too.

unxed added the bug label Apr 9, 2023

kovidgoyal closed this as completed Apr 9, 2023

kovidgoyal reopened this Apr 9, 2023

kovidgoyal closed this as completed in d4c5b8c Apr 9, 2023

unxed mentioned this issue Jan 28, 2025

[RFC] New keyboard reporting protocol #3248

Closed

Uh oh!

Text-as-code-points enhancement does not work with Cyrillic keyboard layout #6167

Text-as-code-points enhancement does not work with Cyrillic keyboard layout #6167

Comments

unxed commented Apr 9, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

kovidgoyal commented Apr 9, 2023

Uh oh!

unxed commented Apr 9, 2023

Uh oh!

kovidgoyal commented Apr 9, 2023

Uh oh!

unxed commented Apr 9, 2023

Uh oh!

kovidgoyal commented Apr 9, 2023

Uh oh!

kovidgoyal commented Apr 9, 2023

Uh oh!

kovidgoyal commented Apr 9, 2023

Uh oh!

unxed commented Apr 9, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

unxed commented Apr 9, 2023

Uh oh!

kovidgoyal commented Apr 10, 2023

Uh oh!

unxed commented Apr 5, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kovidgoyal commented Apr 5, 2024

Uh oh!

unxed commented Apr 5, 2024

Uh oh!

kovidgoyal commented Apr 5, 2024

Uh oh!

unxed commented Apr 5, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kovidgoyal commented Apr 5, 2024

Uh oh!

kovidgoyal commented Apr 5, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

unxed commented Apr 5, 2024

Uh oh!

kovidgoyal commented Apr 5, 2024

Uh oh!

kovidgoyal commented Apr 5, 2024

Uh oh!

unxed commented Apr 5, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kovidgoyal commented Apr 5, 2024 via email

Uh oh!

kovidgoyal commented Apr 5, 2024

Uh oh!

unxed commented Apr 9, 2024

Uh oh!

unxed commented Apr 9, 2023 •

edited

Loading

unxed commented Apr 9, 2023 •

edited

Loading

unxed commented Apr 5, 2024 •

edited

Loading

unxed commented Apr 5, 2024 •

edited

Loading

kovidgoyal commented Apr 5, 2024 •

edited

Loading

unxed commented Apr 5, 2024 •

edited

Loading