Pressing shift-comma inputs '>' instead of '<' in x0vncserver

[alanc]

When using x0vncserver to control a Xorg server which is using a US keyboard layout, pressing shift-comma should enter the '<' key but instead enters '>'.

This appears to be because it's sending the keycode for the pc105 layout key with < and >, for which the shifted form is '>' instead of the comma keycode.

This can be seen via 'xev -event keyboard' - on the console keyboard directly connected to the X server, shift-comma generates these X events:

KeyPress event, serial 28, synthetic NO, window 0x2800001,
    root 0x296, subw 0x0, time 1049420270, (91,99), root:(827,156),
    state 0x10, keycode 50 (keysym 0xffe1, Shift_L), same_screen YES,
    XLookupString gives 0 bytes: 
    XmbLookupString gives 0 bytes: 
    XFilterEvent returns: False

KeyPress event, serial 31, synthetic NO, window 0x2800001,
    root 0x296, subw 0x0, time 1049420678, (91,99), root:(827,156),
    state 0x11, keycode 59 (keysym 0x3c, less), same_screen YES,
    XKeysymToKeycode returns keycode: 94
    XLookupString gives 1 bytes: (3c) "<"
    XmbLookupString gives 1 bytes: (3c) "<"
    XFilterEvent returns: False

KeyRelease event, serial 31, synthetic NO, window 0x2800001,
    root 0x296, subw 0x0, time 1049420742, (91,99), root:(827,156),
    state 0x11, keycode 59 (keysym 0x3c, less), same_screen YES,
    XKeysymToKeycode returns keycode: 94
    XLookupString gives 1 bytes: (3c) "<"
    XFilterEvent returns: False

KeyRelease event, serial 31, synthetic NO, window 0x2800001,
    root 0x296, subw 0x0, time 1049420878, (91,99), root:(827,156),
    state 0x11, keycode 50 (keysym 0xffe1, Shift_L), same_screen YES,
    XLookupString gives 0 bytes: 
    XFilterEvent returns: False

but when hitting shift-comma on a TigerVNC viewer running on Windows, connected to the x0vncserver, it instead reports:

KeyPress event, serial 28, synthetic NO, window 0x2800001,
    root 0x296, subw 0x0, time 1054851162, (166,113), root:(1636,170),
    state 0x10, keycode 50 (keysym 0xffe1, Shift_L), same_screen YES,
    XLookupString gives 0 bytes: 
    XmbLookupString gives 0 bytes: 
    XFilterEvent returns: False

KeyPress event, serial 31, synthetic NO, window 0x2800001,
    root 0x296, subw 0x0, time 1054851483, (166,113), root:(1636,170),
    state 0x11, keycode 94 (keysym 0x3e, greater), same_screen YES,
    XKeysymToKeycode returns keycode: 60
    XLookupString gives 1 bytes: (3e) ">"
    XmbLookupString gives 1 bytes: (3e) ">"
    XFilterEvent returns: False

KeyRelease event, serial 31, synthetic NO, window 0x2800001,
    root 0x296, subw 0x0, time 1054851620, (166,113), root:(1636,170),
    state 0x11, keycode 94 (keysym 0x3e, greater), same_screen YES,
    XKeysymToKeycode returns keycode: 60
    XLookupString gives 1 bytes: (3e) ">"
    XFilterEvent returns: False

https://wiki.archlinux.org/index.php/TigerVNC#Unable_to_type_less_than_character_.28.3C.29
suggests the workaround of:

$ x0vncserver -RemapKeys="0x3c->0x2c"

I've found using xmodmap to make the '<'/'>' key look like the ','/'<' key works as well:

$ xmodmap -e 'keycode  94= comma less comma less'

[alanc]

This is not the right fix, as it hardcodes US/qwerty keymap assumptions, but it works:

diff --git a/common/rfb/VNCSConnectionST.cxx b/common/rfb/VNCSConnectionST.cxx
index d2206f9b..b3ab0d77 100644
--- a/common/rfb/VNCSConnectionST.cxx
+++ b/common/rfb/VNCSConnectionST.cxx
@@ -40,6 +40,7 @@
 #include <rfb/Encoder.h>
 #define XK_MISCELLANY
 #define XK_XKB_KEYS
+#define XK_LATIN1
 #include <rfb/keysymdef.h>
 
 using namespace rfb;
@@ -568,6 +569,14 @@ void VNCSConnectionST::keyEvent(rdr::U32 key, bool down) {
     key = XK_Tab;
   }
 
+  if (key == XK_less) {
+    if (pressedKeys.find(XK_Shift_L) != pressedKeys.end() ||
+        pressedKeys.find(XK_Shift_R) != pressedKeys.end()) {
+        key = XK_comma;
+        vlog.debug("Key remapped to 0x%x", key);
+    }
+  }
+
   if (down) {
     pressedKeys.insert(key);
   } else {

I suspect the real fix looks more like the vncKeysymToKeycode() function from unix/xserver/hw/vnc/InputXKB.c instead of the simple XKeysymToKeycode() currently used in unix/x0vncserver/x0vncserver.cxx.

[CendioOssman]

Indeed. x0vncserver is more of a simple example than something proper. So it would need all of that complexity from Xvnc.

[CendioOssman]

A small tweak would be to make it find a key with the right shift state, even if it cannot add fake key presses. I can have a quick look to see if that is possible.

[CendioOssman]

Can you test #496 and see if it solves your issue?

[alanc]

Yes, thanks, it solves this issue, and works well with most keys. However, if I press the "Super" key (the diamond on my Sun keyboard, which should be the same as the Windows key on PC or the Command key on Mac), before this change it brought up the GNOME 3 overview mode, but after it crashes x0vncserver with:

X Error of failed request: BadValue (integer parameter out of range for operation)
Major opcode of failed request: 132 (XTEST)
Minor opcode of failed request: 2 (X_XTestFakeInput)
Value in failed request: 0x0
Serial number of failed request: 224
Current serial number in output stream: 224

[CendioOssman]

I'm afraid I can't reproduce the issue, but I have a guess. I've updated the commit. Could you give it a new try?

[CendioOssman]

Ping @alanc

[CendioOssman]

Likely fix merged.