Some Cyrillic characters are non-printable

[z1trus]

Hello.

While typing in cyrrilic (for example in russian language) some characters appear like "ó" (must be russian "y") or don't appear at all. In stdout there are many messages like:

"19/04/2016 23:56:35 added missing keysym to X display: 248 0x6db "Cyrillic_sha"

If disconnect and connect again remote session, there may other characters will be unavailable. Using "-remap" key doesn't solve the problem.

In attachment there is file which I use for remapping.
remap.txt

[codedokode]

I have checked this bug on Debian 8.

• uname: Linux debian 3.16.0-4-586 Add support for the X Composite Extension. #1 Debian 3.16.7-ckt9-3~deb8u1 (2015-04-24) i686 GNU/Linux
• /etc/issue: Debian GNU/Linux 8
• x11vnc server: built from commit 0c2dc5e430, run with xvfb 2:1.19.2-1+deb9u2
• x11vnc version: 0.9.15 lastmod: 2018-02-04
• VNC client: TightVNC on Windows

I was connecting with TightVNC client to x11vnc + xvfb server on Debian. I started xterm and tried to type all 33 characters of cyrillic alphabet (абвгд... and so on). They were displayed correctly. After 27 typed letters, I was unable to type remaining 6 letters, nothing happened (first 27 letters still were working).

Excerpt from log:

added missing keysym to X display: 093 0x6c1 "Cyrillic_a"
added missing keysym to X display: 097 0x6c2 "Cyrillic_be"
added missing keysym to X display: 103 0x6d7 "Cyrillic_ve"
added missing keysym to X display: 120 0x6c7 "Cyrillic_ghe"
added missing keysym to X display: 132 0x6c4 "Cyrillic_de"
added missing keysym to X display: 149 0x6c5 "Cyrillic_ie"
added missing keysym to X display: 154 0x6a3 "Cyrillic_io"
added missing keysym to X display: 168 0x6d6 "Cyrillic_zhe"
added missing keysym to X display: 178 0x6da "Cyrillic_ze"
added missing keysym to X display: 183 0x6c9 "Cyrillic_i"
added missing keysym to X display: 184 0x6ca "Cyrillic_shorti"
added missing keysym to X display: 197 0x6cb "Cyrillic_ka"
added missing keysym to X display: 202 0x6cc "Cyrillic_el"
added missing keysym to X display: 217 0x6cd "Cyrillic_em"
added missing keysym to X display: 219 0x6ce "Cyrillic_en"
added missing keysym to X display: 221 0x6cf "Cyrillic_o"
added missing keysym to X display: 222 0x6d0 "Cyrillic_pe"
added missing keysym to X display: 230 0x6d2 "Cyrillic_er"
added missing keysym to X display: 247 0x6d3 "Cyrillic_es"
added missing keysym to X display: 248 0x6d4 "Cyrillic_te"
added missing keysym to X display: 249 0x6d5 "Cyrillic_u"
added missing keysym to X display: 250 0x6c6 "Cyrillic_ef"
added missing keysym to X display: 251 0x6c8 "Cyrillic_ha"
added missing keysym to X display: 252 0x6c3 "Cyrillic_tse"
added missing keysym to X display: 253 0x6de "Cyrillic_che"
added missing keysym to X display: 254 0x6db "Cyrillic_sha"
added missing keysym to X display: 255 0x6dd "Cyrillic_shcha"

I also tried to restart x11vnc and type characters in other order and got the same result - after 27 characters x11vnc runs out of 1-byte values.

So I cannot confirm that cyrillic characters are displayed incorrectly. Probably author had some problem with fonts or encoding. It can be cheched by creating a shell script with command like echo абвгд and running it. If the characters are displayed incorrectly, it is not x11vnc's problem (I have tested it, in my case characters were displayed correctly).

But I can confirm that all of cyrillic charactes cannot be typed, only 27 of them.

Also I tried to type Japanese characters (Hiragana and Kanji) into xterm and Opera, they are just ignored, not displayed and there is no messages in the log.

[xcme]

I have the same problem with cyrilic "у". After a couple of hours of meditation I found how to avoid it:
"-remap oacute-Cyrillic_u,Oacute-Cyrillic_U".

Note also, there MUST be the same keyboard layout on both sides - on the client and on the server too. I use Windows 8.1 as a client, Ubuntu 16.04 as a server and UltraVNC viewer as an VNC application. I have ENG and RU layouts on both sides and when I have ENG on VNC viewer and ENG in remote application or RU in VNC viewer and RU in remote application it works well with the fix above. But it case when there is ENG/RU pair or RU/ENG pair the keyboard behaviour is unpredictable - sometimes it works well, sometimes some character are unprintable or sometimes it doesn't work at all. This it's VERY annoying bug because keyboard layout is application-based, not system-based, so I need to check it every time I switch my windows in Ubuntu.

I use VNC from time to time since 2005 and may add there ALWAYS were some problems with cyrillic layout. But in the nowadays we have Github. It means we have a chance to fix it. =) If you have ideas how I can help you to reproduce and confirm the bug or what I may try fo avoid it just let me know.

[codedokode]

In the X server there is a layout, that maps a keycode to a keysym (a character code). Keycode is a number from 0 to 255 and it corresponds to a key number on a physical keyboard. When VNC server receives a keysym from remote end, it looks for a matching keycode in the current layout. If there is no matching keycode, then it finds an unused keycode and maps new keysym there. When there are no more unused keysyms left, VNC server cannot add new keysyms anymore so new characters cannot be entered.

This logic can be seen in several functions:

• keyboard(): https://github.com/LibVNC/x11vnc/blob/master/src/keyboard.c#L3413
• xkb_tweak_keyboard(): https://github.com/LibVNC/x11vnc/blob/master/src/keyboard.c#L1756
• modifier_tweak_keyboard(): https://github.com/LibVNC/x11vnc/blob/master/src/keyboard.c#L2730

There is also a code that cleans up those temporary keycodes 300 seconds later in check_add_keysyms(): https://github.com/LibVNC/x11vnc/blob/master/src/keyboard.c#L444

The solution (a hack to be precise) could be to reuse those temporary keycodes. For example, xdotool uses a single unused keycode as a "scratch keycode" and for every new character (that is not present in a layout) maps a character to the scratch keycode, notifies X server (and all the clients) that the layout has changed, sends the keypress event to the X server and then reuses the same scratch keycode for another character: https://github.com/jordansissel/xdotool/blob/master/xdo.c#L1027

[xcme]

@codedokode, I have created a log which may help us with this issue. I posted it here: https://pastebin.com/27GqXSD3

Let me explain what it means. There is a series of attempts where I pressed THE SAME key in different layouts and then just released it. I save log for each case. The first part in XX/XX pair means my local layout in VNCViewer and the second part means remote layout on remote x11vnc server. Let's go through it.

The first two cases when I pressed a key when there is the same layout on both sides. After I switched the layout from EN to RU on my side (and remote side too) the latin 'z' has been transformed to 'Cyrillic_ya'. All works perfectly for both this cases.

Next two cases it RU/EN pair. First time it works well, but then I changed a window (an application) on remote side and second time I got 'null' instead 'Cyrillic_ya'. Nothing appears in my text editor when I did it.

The next two cases for EN/RU pair. I see there is a some trick with 'ISO_Level3_Shift'. I don't know what is means but it works. Then I pressed Shift+z in order to obtain "Я" (capital 'Cyrillic_ya') on remote side. It clearly says it got the 'Left Shift' and capital 'Z'. But then it released 'Left Shift' and then emulated 'Cyrillic_ya' and I got "я" in lowercase. Thus, in this scenario the Shift key doesn't work at all and it's impossible to type something in uppercase.

I hope it helps.

[xcme]

I want to add I don't familiar with C language and with VNC internal logic. But I can test this app in various scenarios. If I can to do something which help you to understand how to fix all these problems just let me know.

In short, I see two main problems now:

1. There is a 'null' in some cases instead a normal keycode.
2. The Shift key doesn't work at all in some cases.

I was thinking about your previous comment and I think the app want to use a tricks in some cases but it couldn't to do it correclty. :)

[patrakov]

For the record, this problem is not specific to x11vnc. TigerVNC's x0vncserver also has it.