International Input and XKeyboard Development
Common Lisp applications that use the X11 protocol are missing important X11 extensions that would allow them to support internationalized input. Specifically, the xkeyboard protocol must be fully implemented to rectify these problems for any application utlizing CLX.
In 1987, the X11 standard was formalized. It is the de-facto standard for graphics primitives on the *nix desktop. X11 is a client-server protocol that abstracts away direct calls to hardware rendering. It allows graphical desktops to be rendered in mainframe environments where the server is physically separated from the client. If you’ve ever forwarded X11 over ssh, you’ve interacted with a remote X11 server. Most of the time the X11 server is resident on the local machine, and the client-server connection is over a Unix socket.
It is important to understand that almost all user facing applications are clients to the server (all the way down to your window manager). This means that if you write an X11 client with a library that doesn’t support the modern features of your server, that application will not take full advantage of the server. Other applications written with different X11 client libraries which support modern features will have no trouble responding to the new extensions in the appropriate manner.
It may sound like I’m harping on a subtle distinction that would never come up in the real world. However it has in the case of internationalization and localization support in the Common Lisp X interface (CLX). Further, there seems to be no resources out there for eager contributors to read and understand the problem from end to end. This article aims to rectify that.
For many years the standard library implementing X11 client protocols in C was Xlib. In fact it is still widely used by higher level tool-kits that provide further GUI abstractions. Xlib has its own set of problems and cruft associated with its long shelf-life. In 2001 XCB (X C Bindings) was developed to address these problems. XCB is unique in that it defines the X protocol (and its extensions) in XML format and then generates the C bindings dynamically. If you write an X11 client application in C, in 2018, you should be using XCB.
Most non-C languages interface with Xlib through some sort of foreign function interface. This has the advantage that features added to Xlib are carried over to languages which interface from it. The situation for Common Lisp is different. The Common Lisp X interface (CLX) does not interface with Xlib, instead it implements the protocol directly. I’m speculating, but I expect that this exists because of the Lisp Machines of the 70s and 80s. They were entirely different animals from Unix, and had Lisp baked into their hardware. In that environment, X11 would have been a protocol to allow a lisp machine to accept clients from Unix mainframes. It makes sense then that the library would be written in Common Lisp from the ground up.
What would happen next is still causing echos today in modern lisp applications. In the time period of 1987-1993 the Lisp Machine market collapsed in the wake of the AI winter. The CLX manual, dated 1988, came at the end of the Lisp Machine’s relevance in computing research. In 1991 Linus Torvalds released the first version of Linux, and ushered in an era of unix-like desktops. The ubiquity of a free version of unix-like operating system and the explosion of the internet created market pressures which lead to internationalization support in Xlib that didn’t exist in the Common Lisp ecosystem. Extensions such as xkeyboard (XKB) have never been fully implemented, and it is difficult as a sole developer to fully implement a protocol standard as complicated as XKB. This is further complicated by the fact that end users typically want to be able to type non-english letters and it is not clear how to communicate this in a way that points developers to the XKB extension.
There has been significant progress. SBCL supports unicode characters, and rendering toolkits can render the glyphs appropriately. The missing secret sauce is a implementation of XKB. A few people have ventured deep enough into this problem to identify the missing piece, but a full solution has not made it into sharplispers/clx, the community’s source for CLX.
McClim (a major graphics toolkit) has placed a significant bounty on
this very issue. The success condition for that bounty has been that
“input [is obeyed by a] keyboard layout set with setxkbmap
asynchronusly at runtime by the user from another process (i.e
terminal).” Another more promising effort is available as a standalone
extension in quicklisp. I have just discovered this code, and I have
no idea how close it is to solving the issue.
If you’ve made it this far, you’re probably eagerly consuming any information on this topic and want to contribute. My nominal plan is to host a fork of sharplispers/clx. I will contact Mikhail Filonenko about taking over his copy of xkeyboard and I will add stubs of function definitions for xkeyboard. Contributing will be a matter of opening a PR with an implementation of a missing function. Once we have the majority of the protocol implemented we can contribute it back to sharplispers/clx and the whole Lisp community will profit.
Even if you are not affected by internationalized input, having XKB available will be useful as it makes it possible to do things that are impossible with the core implementation of keyboards. Specifically, it becomes possible to have multiple keys trigger the same key event which makes it possible to have multiple prefix keys in StumpWM. It also makes it possible to support weird keyboard layouts that have different modifiers than what a generic keyboard can handle.
Hopefully this will serve as a roadmap of how and why Common Lisp has poor support for international input and will help dispel some of the confusion around the topic. If you are new to Common Lisp and affected by this issue, take a look at my fork of CLX. There may be something you can contribute to which will help us solve the problem once and for all. If you have any comments, you can tweet me @dec4fc0ffee.