Python app crashes when pressing any key in a non-English layout

[maxpain]

Python app crashes when pressing any key in a non-English layout:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/selkies_gstreamer/gstwebrtc_app.py", line 1007, in <lambda>
    'on-message-string', lambda _, msg: self.on_data_message(msg))
  File "/usr/local/lib/python3.10/dist-packages/selkies_gstreamer/webrtc_input.py", line 533, in on_message
    self.send_x11_keypress(int(toks[1]), down=True)
  File "/usr/local/lib/python3.10/dist-packages/selkies_gstreamer/webrtc_input.py", line 321, in send_x11_keypress
    self.keyboard.press(keycode)
  File "/usr/local/lib/python3.10/dist-packages/pynput/keyboard/_base.py", line 392, in press
    self._handle(resolved, True)
  File "/usr/local/lib/python3.10/dist-packages/pynput/keyboard/_xorg.py", line 242, in _handle
    with display_manager(self._display) as dm:
  File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__
    next(self.gen)
  File "/usr/local/lib/python3.10/dist-packages/pynput/_util/xorg.py", line 78, in display_manager
    display.sync()
  File "/usr/local/lib/python3.10/dist-packages/Xlib/display.py", line 182, in sync
    self.get_pointer_control()
  File "/usr/local/lib/python3.10/dist-packages/Xlib/display.py", line 833, in get_pointer_control
    return request.GetPointerControl(display = self.display)
  File "/usr/local/lib/python3.10/dist-packages/Xlib/protocol/rq.py", line 1368, in __init__
    self.reply()
  File "/usr/local/lib/python3.10/dist-packages/Xlib/protocol/rq.py", line 1380, in reply
    self._display.send_and_recv(request = self._serial)
  File "/usr/local/lib/python3.10/dist-packages/Xlib/protocol/display.py", line 612, in send_and_recv
    gotreq = self.parse_response(request)
  File "/usr/local/lib/python3.10/dist-packages/Xlib/protocol/display.py", line 719, in parse_response
    gotreq = self.parse_error_response(request) or gotreq
  File "/usr/local/lib/python3.10/dist-packages/Xlib/protocol/display.py", line 745, in parse_error_response
    req = self.get_waiting_request(e.sequence_number)
AttributeError: 'BadRRModeError' object has no attribute 'sequence_number'

After that, I was unable to access the webrtc app:

0:03:15.931713640   106 0x7fd9bc0b0520 WARN         sctpassociation sctpassociation.c:899:handle_association_changed:<GstSctpAssociation@0x7fd980002590> SCTP event SCTP_COMM_LOST received
0:03:15.931733758   106 0x7fd9bc0b0520 WARN                 sctpenc gstsctpenc.c:872:on_sctp_association_state_changed:<sctpenc1> error: SCTP association went into error state
0:03:16.518575545   106 0x55dac020ff00 ERROR        sctpassociation sctpassociation.c:483:gst_sctp_association_send_data:<GstSctpAssociation@0x7fd980002590> Association not connected yet
0:03:16.518588352   106 0x55dac020ff00 WARN                 sctpenc gstsctpenc.c:646:gst_sctp_enc_sink_chain:<sctpenc1> error: Failed to send data
0:03:16.518606074   106 0x55dac020ff00 WARN                 basesrc gstbasesrc.c:3127:gst_base_src_loop:<appsrc1> error: Internal data stream error.
0:03:16.518608505   106 0x55dac020ff00 WARN                 basesrc gstbasesrc.c:3127:gst_base_src_loop:<appsrc1> error: streaming stopped, reason error (-5)
INFO:signaling:Connection to peer ('10.0.0.39', 59252) closed, exiting handler
INFO:signaling:Cleaned up 1 session
INFO:signaling:Also cleaned up 0 session
INFO:signaling:Closing connection to 0
INFO:gstwebrtc_app:stopping pipeline
INFO:gstwebrtc_app:data channel closed
INFO:gstwebrtc_app:setting pipeline state to NULL
0:03:17.327469669   106 0x55dac0210d20 WARN                audiosrc gstaudiosrc.c:227:audioringbuffer_thread_func:<pulsesrc> error reading data -1 (reason: Success), skipping segment

[ehfd]

This is going to take some time. Do restart Selkies to reconnect. In the container, you can terminate selkies-gstreamer using htop and it will automatically reload.

[ehfd]

I believe this is a pynput problem.

[ehfd]

Should be solved with #9.

[ehfd]

Revamping of the Javascript libraries are necessary, especially vuetify and guacamole-keyboard.

[ehfd]

Priority Number 3 IMO (with #9 and #22) @danisla

[maxpain]

Any updates?

[ehfd]

Currently discussed in #85

[ehfd]

In short, need to implement xorg.c into ctypes (courtesy of @cruizba).

I think I found a possible way to fix all inconsistencies and errors related to the keyboard, potentially paving the way for the removal of pyinput and, possibly in the future, python-xlib. You can find my work in this branch of my fork: GitHub - cruizba/selkies-gstreamer.

Let me share the backstory of how I arrived to this.

Upon further investigation, I realized that certain keysyms which reach this section of webrtc_input.py, particularly those representing Spanish characters on my keyboard (like ñ), lacked corresponding keycodes. I've noticed that X server loads some generic keycodes, but it doesn’t cover all of them. This might relate to the default keyboard setting, which is typically a Generic 105-key PC, but I am not sure.

In a remote desktop environment, the server is unaware of the client's keyboard. Therefore, modifying the server's keyboard configuration seemed impractical as a solution to me...

To explore how other projects handle keyboard input remotely, I looked into Neko. Neko, along with Guacamole keyboard lib in the browser, manages keyboard events from the browser. When the keysym reaches the neko server, they process input data through a set of C functions in xorg.c, which they integrate with Go.

The key function of xorg.c is XKey(). It appears that XKey dynamically assigns KeySyms to Keycodes when there is no existing association. If a keysym is sent without a corresponding keycode, it registers this new mapping in the XOrg keyboard configuration (As far as I can understand from the code). This approach is also utilized in TigerVNC's source code for keyboard keys mapping!

Consequently, I've created xorg.py to integrate the build of xorg.c and changed the keyboard part of webrtc_input.py to utilize the new xorg class. This solution works exceptionally well! It detects all keyboard keys correctly without altering any remote keyboard settings and while using my own keyboard configuration.

The challenge was creating a new addon and compiling the xorg.so file to implement this functionality. The project now requires additional headers and libraries. However, the good news is that this addon could potentially replace python-xlib for mouse interactions and other X interactions.

If you're interested in testing it, my branch is fully functional. Simply run vscode with the devcontainer as usual and execute the task [run] re-build, re-install and run selkies-gstreamer. It's configured to compile and install the library.

No, files present at https://github.com/cruizba/selkies-gstreamer/tree/fix-inputs/addons/xorg-iface are just a simple module using x11 libs, it is not part of the X library.

[ehfd]

This error itself should be fixed in the next release. Using a fork of Python-Xlib which fixes this.

AZERTY Keyboard is handled in #9.

[cruizba]

I am so happy to see this :).

[ehfd]

@cruizba You deserve an acknowledgment. We're also trying to solve the keycode (Spanish) issue with a ctypes version of your .so file.

I hope everything comes together to the next release.