Minibuffer input ignored for CJK locales

[xelxebar]

Overview

When LANG is set to an existing CJK, input in the minibuffer is almost completely ignored. Interestingly, the keys that do work are <enter>, arrow navigation, and dash (-).

I have confirmed this with the following locales:

• ja_JP.utf8,
• ko_KR.utf8, and
• zh_TW.utf8.

In addition, I checked various other locales, and those seem to be working just fine.

Note that if the CJK locale doesn't exist on your system, then nyxt correctly falls back to LANG=C.

Reproduction

In my case, nxyt is installed via guix. The following invocation pairs down the a minimal environment capable of demonstrating the issue:

$ env -i XAUTHORITY=$XAUTHORITY DISPLAY=$DISPLAY LANG=ja_JP.utf8 GUIX_LOCPATH=$GUIX_LOCPATH $(command -v nyxt)
Nyxt version 1.5.0-1.9440980
<INFO> [22:27:58] Using data profile "default".
<INFO> [22:27:58] Listening to socket "/home/x/.local/share/nyxt/nyxt.socket".
<INFO> [22:28:01] Deleting socket /home/x/.local/share/nyxt/nyxt.socket

Version Info

$ guix describe
Generation 84	 7月 06 2020 01:18:06	(current)
  guix f3a5e58
    repository URL: https://git.savannah.gnu.org/git/guix.git
    branch: master
    commit: f3a5e5881390be67965ebdd7fd7620e95d771f03

$ readlink -f $(command -v nyxt)
/gnu/store/s6034py1v9ylij9dp2707fr67wakh6a5-nyxt-1.5.0-1.9440980/bin/nyxt

$ lsb_release -a
LSB Version:	1.0
Distributor ID:	VoidLinux
Description:	Void Linux
Release:	rolling
Codename:	void

Running with an unset LANG lets me grab the following from copy-system-information:

Nyxt version: 1.5.0-1.9440980 
Operating system kernel: Linux 5.4.50_1
Lisp implementation: SBCL 2.0.5
Features: (WEBKIT2 WEBKIT2-2.28 WEBKIT2-EMOJI WEBKIT2-MEDIA WEBKIT2-SANDBOXING
           GTK-3-22 GTK-3-20 GTK-3-18 GTK-3-16 GTK-3-14 GTK-3-12 GTK-3-10
           GTK-3-8 GTK-3-6 GTK-3-4 GTK GDK-3-22 GDK-3-20 GDK-3-18 GDK-3-16
           GDK-3-14 GDK-3-12 GDK-3-10 GDK-3-8 GDK-3-6 GDK-3-4 CAIRO-1-10
           CAIRO-1-12 GDK-PIXBUF GLIB-2-30 GLIB-2-32 GLIB-2-34 GLIB-2-36
           GLIB-2-38 GLIB-2-40 GLIB-2-42 GLIB-2-44 GLIB-2-46 GLIB-2-48
           GLIB-2-50 GLIB-2-52 GLIB-2-54 GLIB-2-56 GLIB-2-58 GLIB
           FSET-EXT-STRINGS SWANK PLUMP-UTF-32 GLOBAL-VARS DECLARE-TYPES
           PARENSCRIPT SBCL+SAFE-STANDARD-READTABLE NAMED-READTABLES
           OSICAT-FD-STREAMS LPARALLEL 21BIT-CHARS CL-FAD CHUNGA FLEXI-STREAMS
           CLOSER-MOP CL-PPCRE-UNICODE CL-UNICODE CL-PPCRE
           CL-JSON-DOUBLE-FLOAT-IS-SUBSUMED CL-JSON-SINGLE-FLOAT-IS-SUBSUMED
           BORDEAUX-THREADS LPARALLEL.WITH-CLTL2 LPARALLEL.WITH-CAS
           LPARALLEL.WITH-STEALING-SCHEDULER SPLIT-SEQUENCE GRAY-STREAMS
           FLAT-NAMESPACE X86-64 UNIX CFFI FLAT-NAMESPACE SEQUENCE-EMPTYP
           FAST-IO-SV FAST-IO SBCL-USES-SB-ROTATE-BYTE CL-JSON-CLOS CL-JSON
           THREAD-SUPPORT ASDF3.3 ASDF3.2 ASDF3.1 ASDF3 ASDF2 ASDF OS-UNIX
           NON-BASE-CHARS-EXIST-P ASDF-UNICODE X86-64 GENCGC 64-BIT ANSI-CL
           COMMON-LISP ELF IEEE-FLOATING-POINT LINUX LITTLE-ENDIAN
           PACKAGE-LOCAL-NICKNAMES SB-CORE-COMPRESSION SB-LDB SB-PACKAGE-LOCKS
           SB-THREAD SB-UNICODE SBCL UNIX)

[Ambrevar]

I believe this is a duplicate of #324. Can you confirm?

[xelxebar]

As far as I can tell, this is a separate issue. All input (except dash) is ignored, including Latin characters. Note that the env -i invocation clears the *_IM_MODULE etc. variables, so if I understand correctly, input methods should be irrelevant in this case.

[Ambrevar]

Indeed, I made the wrong assumption reading your initial post. Could you run nyxt with the --verbose flag in a shell, open the minibuffer, then press a latin key and report the output? It'd be great if you could highlight the exact output that's generate upon the keypress.

[xelxebar]

Comparing against LANG=C nyxt, it looks like dispatch-input-event is not getting called, even though the keypresses are registering:

<DEBUG> [08:44:07] nyxt renderer-gtk.lisp (on-signal-key-press-event gtk-window) -
  NYXT::KEY-STRING: NIL NYXT::KEYCODE: 50 CHARACTER: #\Nul
  NYXT::KEYVAL-NAME: "Shift_L"
<DEBUG> [08:44:07] nyxt renderer-gtk.lisp (on-signal-key-press-event gtk-window) -
  NYXT::KEY-STRING: "H" NYXT::KEYCODE: 44 CHARACTER: #\H NYXT::KEYVAL-NAME: "H"
  NYXT::MODIFIERS: ("shift")
<DEBUG> [08:44:07] nyxt renderer-gtk.lisp (on-signal-key-press-event gtk-window) -
  NYXT::KEY-STRING: "e" NYXT::KEYCODE: 40 CHARACTER: #\e NYXT::KEYVAL-NAME: "e"
<DEBUG> [08:44:08] nyxt renderer-gtk.lisp (on-signal-key-press-event gtk-window) -
  NYXT::KEY-STRING: "l" NYXT::KEYCODE: 33 CHARACTER: #\l NYXT::KEYVAL-NAME: "l"
<DEBUG> [08:44:08] nyxt renderer-gtk.lisp (on-signal-key-press-event gtk-window) -
  NYXT::KEY-STRING: "l" NYXT::KEYCODE: 33 CHARACTER: #\l NYXT::KEYVAL-NAME: "l"
<DEBUG> [08:44:08] nyxt renderer-gtk.lisp (on-signal-key-press-event gtk-window) -
  NYXT::KEY-STRING: "o" NYXT::KEYCODE: 39 CHARACTER: #\o NYXT::KEYVAL-NAME: "o"
<DEBUG> [08:44:09] nyxt renderer-gtk.lisp (on-signal-key-press-event gtk-window) -
  NYXT::KEY-STRING: NIL NYXT::KEYCODE: 133 CHARACTER: #\Nul

Here is the entire log: nyxt.log.

[Ambrevar]

I think the culprit is `printable-p`: for some reason it returns nil with CJK. Here the code: ``` (defmethod printable-p ((window gtk-window) event) "Return the printable value of EVENT." ;; Generate the result of the current keypress into the dummy ;; key-string-buffer (a GtkEntry that's never shown on screen) so that we ;; can collect the printed representation of composed keypress, such as dead ;; keys. (gtk:gtk-entry-im-context-filter-keypress (key-string-buffer window) event) (when (<= 1 (gtk:gtk-entry-text-length (key-string-buffer window))) (prog1 (match (gtk:gtk-entry-text (key-string-buffer window)) ;; Special cases: these characters are not supported as is in KEY values. ;; See `self-insert' for the reverse translation. (" " "space") ("-" "hyphen") (character character)) (setf (gtk:gtk-entry-text (key-string-buffer window)) "")))) ``` So either gtk-entry-im-context-filter-keypress never inserts the event in key-string-buffer, or `gtk:gtk-entry-text' returns nil. Can anyone think of something else?

[jmercouris]

I cannot think of anything else. I think it is the gtk:gtk-entry-text which is returning nil. Do CJK locales have any problems in any other GTK programs?

[xelxebar]

This is the first time I have encountered anything like this, having run with LANG=ja_JP.utf8 for years.

[Ambrevar]

The above function is a rather unique hack, so it is not a big surprise :/

[Ambrevar]

Now that #1157 got merged, you should be able to use any input system from the minibuffer (now called prompt-buffer).

@xelxebar Can you confirm if works for you?

[jmercouris]

I have had other users confirm that it now works!

[xelxebar]

Thanks for working on this! The issue reported here is fixed: Nyxt works as expected with LANG set to CJK locales and English input. That's a huge improvement. Unfortunately, however, Japanese inpust still fails for me: see my comment on #866.