Pluggable layout support

[mildred]

The qwerty-lafayette is a new qwerty-bqsed french layout that is not yet reqdy to be included in the xkb maps. Still, it would be desirqble for users to be able to install this layout (see issue fabi1cazenave/qwerty-lafayette#15 in French).

The diff required to install this layout system-wide is: https://gist.github.com/fabi1cazenave/2171d69d3d8d6e298225

The problem with this diff is that it is not possible to create a package from it because it requires patching the rules files. It would be desirable if libxkbcommon could read additional rule files in directories.

For example, /usr/share/X11/xkb/rules/evdev.d/lafayette.lst and /usr/share/X11/xkb/rules/evdev.d/lafayette.xml could be read as part of /usr/share/X11/xkb/rules/evdev.lst and /usr/share/X11/xkb/rules/evdev.xml respectively.

It would be even better if libxkbcommon could read files from ~/.xkb/rules/evdev.d/*.{lst,xml}

[bluetech]

I agree this is desirable.

There are a few parts here:

1. The xml files are not used by xkbcommon itself, and xkbcomon doesn't know about them. They are usually used directly by keymap-chooser GUIs. So making this pluggable might be problematic.
2. The keys/types/compat/symbols files are searched as part of xkbcommon's include path. xkbcommon does in fact already include ~/.xkb in its default include path. So you can have files like ~/.xkb/symbols/myfr.
3. The rules files (.lst) are also searched as part of xkbcommon's include path, and you can use your own file like described above. The problem with that is that rules file do not currently have include statements, so to use that, you would have to copy the file into ~/.xkb/rules, more or less. There is a patch here https://bugs.freedesktop.org/show_bug.cgi?id=74422 which adds an ! include statement. We can use that as a starting point.

The ~/.xkb/rules/evdev.d/ idea is also interesting and might be worthwhile to pursue.

Overall we need to come up with a complete coherent design for this; unfortunately the original design of the RMLVO system did not consider this use case.

[bluetech]

Oops, I confused .lst files with the rules files themselves. The situation with .lst is the same as .xml as far as xkbcommon goes.

[ariasuni]

The keys/types/compat/symbols files are searched as part of xkbcommon's include path. xkbcommon does in fact already include ~/.xkb in its default include path. So you can have files like ~/.xkb/symbols/myfr.

It doesn’t seem to work.

~ ❯ cp /usr/share/X11/xkb/symbols/fr ~/.xkb/symbols/frcopy
~ ❯ setxkbmap frcopy bepo
Error loading new keyboard description

[bluetech]

It's a bit confusing, however setxkbmap does not use libxkbcommon. setxkbmap operates against an X server which uses another program, xkbcomp, to compile the keymaps.

If you use a Wayland compositor, then it does use libxkbcommon.

[ariasuni]

Ok, so how are we supposed to use a custom layout? Is there a method that’s working both on Xorg and Wayland, besides editing system files directly?

[bluetech]

Perhaps setxkbmap -I ~/.xkb frcopy bepo would work and give you parity? (Can't try it myself at the moment).

[ariasuni]

I use setxkbmap at the moment but it doesn’t work on Wayland.

Also I read here about /etc/X11/xkb but I don’t understand if/how I can use that.

[bluetech]

I use setxkbmap at the moment but it doesn’t work on Wayland.

On Wayland, you'll need to use whatever is your compositor's method of configuring XKB. But ~/.xkb will be in the search path by default.

Also I read here about /etc/X11/xkb but I don’t understand if/how I can use that.

This was never implemented.

[ariasuni]

On Wayland, you'll need to use whatever is your compositor's method of configuring XKB. But ~/.xkb will be in the search path by default.

So could I drop one or a few files for a custom keyboard layout in ~/.xkb/ and select it in (e.g.) Plasma’s configuration? Otherwise, that would be interesting to have — editing system files is not a satisfying solution…

[ilmaisin]

Couldn't the evdev.xml files be generated dynamically? It looks like the rule files pretty much contain the same data. I might eventually try and build some kind of prototype of the generation algorithm, but I don't feel qualified to work on the xkb itself.

[whot]

the biggest problem with evdev.xml is that it's loaded directly by all clients that require it. IMO the best option around this is to replace it with a proper API and make clients use that. This way the storage format and where it sits becomes irrelevant.

But, fwiw, the rules files are slightly different to the xml files and do not contain the same data - though anything listed in one will/should/must have an entry in the other. You could merge both together and split them back out during build but I doubt there's a lot of value in that.

[whot]

#146 is that API now as mentioned in the comment above. With #108 it's now possible for libxkbcommon to parse keymaps and symbols in $XDG_CONFIG_HOME/xkb/, so at least from libxkbcommon's perspective this issue is now solved.

[ossilator]

i didn't read the PRs that precisely (and in particular didn't read the code), but i gained the impression that they're only about user-level ($HOME) overrides. while helpful, that doesn't cover the system-wide use case particularly well - that would require systemd-ish .d directories or more traditional "shadow" directories like (/usr/local/)site-xkb/ (compare perl) and/or /etc/xkb. please clarify.

[whot]

the $HOME overrides are the primary use-case but that's largely controlled by include paths. The infrastructure is now in place to add additional include paths though tbh a .d style directory layout wasn't really planned for.

Having said that - I wonder how much of that is needed. If you have a few enthusiastic users during a testing phase, they'd likely be happy to install in $HOME. Once you're past that stage merging into xkeyboard-config is the way to go anyway. So that intermediate stage of site-wide but not upstream yet - is this a common enough use-case to support?

[ossilator]

my use case is that i use the us(altgr-intl) layout with swapped y & z keys (guess why ...). that's needed system-wide (for the VT setup, then the login manager's keymap) and at user level (GUI switcher in KDE). i have my doubts that this would be ever accepted upstream ...

[whot]

I think it would be possible to extend libxkbcommon's meson options to make xkb-config-root a colon-separated path list and add all those paths to the default includes, e.g.

meson builddir --xkb-config-root=/etc/X11/xkb/:/usr/share/X11/xkb

Would need some further investigation for all the corner-cases but this seems to be the easiest way to handle this.

[whot]

I've had a look at the above and while it appears relatively trivial to add a single path, that's not true for the colon-separated list. Let's say we add an extra path so our lookups are, in that order:

• $XDG_CONFIG_HOME/xkb - let's call this the home path
• /etc/xkb - let's call this the etc path
• /usr/share/X11/xkb - let's call this the system path

For rules files we shadow files, ruleset evdev means the first rules/evdev file in the lookup paths. To work properly, that file has to be either complete (the system file) or ! include one that is more complete (home/etc files). It matters whether an entry is before or after an ! include statement so this can't be done automatically.

To include, we need a path and the two placeholders %H and %S resolve to $HOME and /usr/share/X11/xkb respectively.
This makes comma-separated path list difficult because there is no %S equivalent that can apply to the whole list. Which in turn makes writing rulesets awkward because you need to now hardcode the full path to include - hardly a flexible approach and effectively dependent on meson options. In comparison, a single path is trivial, adding %E for the etc path is easy.

Yes, you could have an %P placeholder that searches through all paths but I don't think it is a sensible choice here. It's too hard to guarantee the correct files are included. For example, ! include %P/filename in a file residing in the second lookup path may include from the first lookup path instead of the intended second lookup path. Fixable in the parser, but it gets more and more complicated that way.

Notable, libxkbregistry XML handling does not shadow, it merges all evdev.xml files. But libxkbregistry has no effect on libxkbcommon and the keymap itself.

For kccgst files we currently also shadow files but this is a problem. The file symbols/us from the home path hides the one from the system path. This makes it impossible to add a variant to a system layout, the directory and file names are part of the API and us(myvariant) variant must be in the symbols/us file. But variants usually include the main layout (e.g. include "us(basic)" i.e. file symbols/us section basic) and those are hidden. So right now we cannot add variants to an existing system layout because the file resolution won't work. The us(myvariant) will have to be a myus(myvariant).

As for a /etc/xkb.conf.d approach: usually the conf.d approach is used so you can accumulate several files instead of having one config file. This doesn't work when the filename is part of the API. I think with some parser changes it may be possible so that at least the directory name doesn't matter but us(foo) variant still needs to be in the us file. So you can only have one anyway and the conf.d approach is pointless.

Finally: we could change kccgst to have a merge behaviour so that the symbols/us file is loaded from home, etc, and system paths in order until the required section resolves. This makes pluggable layouts possible because you can then have your us(foo) variant in /etc/xkb/symbols/us and it will handle include statements correctly.

So in summary:

• adding /etc/xkb is doable, /etc/xkb.conf.d/ is not feasible
• adding variants to system layouts is not possible at this point either way

[ossilator]

the path stacking problem is rather easily addressed by something like gcc's #include_next statement.
and i see that you created a smart include statement that uses content addressing. 👍

[whot]

Good point, but I'm not sure #include_next is necessary for the use-cases we need to cover. It requires changes to the parser, any documentation that exists and everyone to add it. I'm struggling to see a case where you need to hide the system layout files, i.e. where the default behaviour of include_next (as currently implemented in #162) isn't the correct one anyway.

[ossilator]

hiding (in the sense of overriding) may make sense to (temporarily) mask an error in the system/distribution without modifying it and without confusing users with new configuration choices.
anyway, i mentioned include_next only as a working alternative to the weirdly complicated path placeholders you were pondering, and have no idea if/how it would fit.

[whot]

Yeah, ftr, I can come up with some niche use-cases where it makes sense (like the one you mentioned) but supporting those is not really our priority :)

[ilmaisin]

So, the next step will be to make GNOME and other desktop environments to use libxkbregistry, like this?

By the way, is there a risk of systemd somehow misbehaving on systems that have only one user, and that user has sudo permissions? In such cases, the per-user locale and keyboard settings are applied system-wide, IIRC.

[whot]

not sure how exactly that would happen, tbh. all we're using here is secure_getenv() to look up the various directories. so if anything, the sudo case is less likely to work than likely to pick up keymaps from the wrong directory.

But otherwise, the same rules apply - if you set a configuration that doesn't exist, parsing the keymap will fail. and if you set a global configuration to a keymap that's private to a user's home directory, that too will fail because from XKB's point of view, the configuration doesn't exist.

[bjohas]

@ilmaisin - I'm very keen to have pluggable layout support in gnome 3 on Ubuntu (20.04 onwards). I've subscribed to https://gitlab.gnome.org/GNOME/gnome-desktop/-/merge_requests/79 - is there anything else I should subscribe to for updates?