Not working with Gnome on Wayland in Linux

[mayanksuman]

Summary

Plover (latest github version) is not working on Gnome on Wayland.

Reproducing

1. Use Gnome in wayland mode (default mode)
2. Navigate to plover git directory and launch it by .\launch.sh
3. Enable the output. However, it is not enabled.

The output of ./launch.sh might be useful.

running launch
running build_py
running build_ui
running egg_info
writing plover.egg-info/PKG-INFO
writing dependency_links to plover.egg-info/dependency_links.txt
writing entry points to plover.egg-info/entry_points.txt
writing requirements to plover.egg-info/requires.txt
writing top-level names to plover.egg-info/top_level.txt
reading manifest template 'MANIFEST.in'
warning: no files found matching 'tox.ini'
writing manifest file 'plover.egg-info/SOURCES.txt'
generating plover/gui_qt/resources_rc.py
running compile_catalog
compiling catalog plover/gui_qt/messages/fr/LC_MESSAGES/plover.po to plover/gui_qt/messages/fr/LC_MESSAGES/plover.mo
copying plover/gui_qt/resources_rc.py -> build/lib/plover/gui_qt
running build_ext
Warning: Ignoring XDG_SESSION_TYPE=wayland on Gnome. Use QT_QPA_PLATFORM=wayland to run on Wayland anyway.
xkbcommon: ERROR: Key "<CAPS>" added to modifier map for multiple modifiers; Using Control, ignoring Lock

Plover Version

Latest from github (Plover 4.0.0.dev8; commit: 2943a97)

System

OS: Debian Linux (kernel version: 4.19.0-2-amd64)
WM: Gnome (3.30.2)

[benoit-pierre]

Wayland is not supported. Not sure if it provides everything needed for Plover to even support it.

[0chroma]

Has anyone ever looked into adapting plover to act as an input method engine? Idk if it's feasible at all but that's how most of the asian language input systems are implemented.

https://blogs.gnome.org/happyaron/2011/01/15/linux-input-method-brief-summary/

[antoniusf]

Hey everyone,

I've recently been persuaded to start using sway, which is also a wayland-based window manager, so I've started to look into this issue a bit more. I thought it might be a good idea to share what I've learnt so far, so that other people can pitch in without having to start all the way from the beginning.

Quick recap: what is Wayland?

Wayland is a protocol that allows programs to talk to a compositor, which is the thing that makes all of the windows appear on your screen. Each application tells the compositor what should be drawn inside of its window (basically it just sends an image, several times per second), and the compositor then takes all of these images, places them in the right positions (i.e. where you put the corresponding window), and gives you a screen to look at. It also takes your input (keyboard, mouse, etc.), and forwards it to the currently active window.

This is also kind of where the problems for plover start, since the base protocol is very restrictive about these kinds of things. It neither allows programs to intercept keyboard input (only the active window receives these events), nor does it allow them to emulate it.

Luckily, wayland also has an extension mechanism that can allow programs to do these kinds of things, provided that the compositor supports the relevant extensions. I am saying this in advance, since both of the solutions below require such extensions. Neither is stabilized yet, and at least one of them is still undergoing active development. Unfortunately, I don't know how well gnome supports these yet, but I will be able to talk about their support in sway.

Input methods

Thanks to @0chroma for bringing this one up, this is actually a very interesting approach. Again, a short introduction:

Input method protocols are what allow a special program (often called an input method engine, or IME) to control text entry, usually by intercepting keyboard events and generating the appropriate output. To do this, it gets special capabilities that allows it to directly manipulate text, without having to emulate a keyboard. This can avoid certain problems, for example, with autocomplete in a browser's address bar. It also means that the active application, into which the text is to be entered, needs to actively support its end of the input method protocol – if it only listens to keyboard input, it won't know about any of this. However, most apps do have support for this as its kind of a requirement for Chinese, or Japanese, or Korean, etc. users.

As far as I can tell, there are currently two ways for an IME to work under wayland. The first is a system called "ibus", which is largely independent of wayland and (I think) integrated directly into gnome. I don't quite understand how it works, but it somehow allows a user to add IMEs and switch between them, and then allows the active IME to do the things described above. (I think, I really don't know a lot about ibus, please correct me if I'm wrong.)

The second approach is an extension built directly into wayland (called input_method_unstable_v2). It allows a program to register itself as an IME directly with the compositor, and then also do the things described above. The nice thing about it is that it is well documented and very easy to get started with. The not-so-nice-thing is that parts of it are still under development, sway hasn't yet released a version that supports it (since the changes were merged only a few months ago), and I don't know whether gnome supports it at all but I would honestly be a bit surprised. I also have some UX questions, like how people will be able to switch between different IMEs over this interface. (Currently, the IME has to try to register when it starts, which will simply fail if another IME is already active.)

In principle, I like the input method approach a lot. It maps very cleanly onto what plover does (take some form of user input, then translate it into normal text and output that somehow), and it avoids the slightly-annoying-but-I-guess-noone-really-cares problem with things like autocomplete. In principle, it would also allow plover to detect when the cursor moves to a different input field, so that it could reset the internal translation buffer. I guess you could even do fancy stuff like detecting if the cursor has been moved to a different location and adapting to that, but I don't expect anyone to need this.

Currently, the input method protocols are also the only way for a background program to intercept keyboard input. (Though this is still being worked on for the wayland extension.) This would mean that the keyboard machine type would definitely need one of these protocols to work.

The obvious disadvantage is that input methods can't do keyboard emulation. So things like shortcuts, cursor keys, etc. are not things that an input method can activate. The only things it can do are insert and delete text.

Keyboard emulation

Yes, there's a protocol extension for this! It's called virtual_keyboard_unstable_v1, but as far as I can tell, it's a bit more stable than the input method extension. I would guess maybe gnome has support for it too, since it's meant to be used for on-screen keyboards? I haven't looked into this too much, since the keymap hackery to get it to output full unicode felt a bit too arcane to me, but there's a tool that can output arbitrary unicode text over this interface, so this might be a good starting point for porting to plover.

I think that the advantage of this is that it maps very well to how plover currently operates, i.e. by keyboard emulation. Of course, it loses the advantages of having a proper IME, but plover currently doesn't have those anyways. And since things like arrow keys or shortcuts do need keyboard emulation, using this interface might be unavoidable. On the other hand, only the input method extension actually allows grabbing keyboard input, which is necessary for the keyboard machine.

Conclusion

Overall, it seems like it might be optimal to combine the two approaches: An input method for (optionally) capturing keyboard input and outputting normal text, and a virtual keyboard for actual keyboard emulation. In fact, this approach is also taken by one of the experimental input methods written for wayland right now (where it's necessary to forward keyboard events that the IME doesn't handle).

For now, I think I will continue to experiment with input_method_unstable_v2, since that is the simplest protocol to work with, and I just want to be able to use plover again. However, this will only help people who use compositors that support this new protocol, which I think excludes gnome. (Hopefully they'll adopt the new protocol at some point?)

Anyway, I hope that I could give a brief overview of all the things left to do to give plover full wayland support (a lot), and the compatibility problems it may face. As I said, a lot of the wayland stuff is still being worked on, and it may still be some time until it is stabilized and widely adopted. If you have any questions about any of this, please feel free to ask. (I'm sorry that I didn't take as much time to write this as I probably should have, and these things are kind of complicated and might deserve more of an explanation.)

[buffet]

It should also be possible to provide a evdev backend, that could be usable from X, Wayland and even the TTY, I think.

[antoniusf]

Hi, sorry for the late response, and thanks for the tip! On my system, it seems that I can't access the evdev devices as a normal user, is this expected? I've read a bit and apparently I can add myself to a special group that would allow me to access these devices without being root, which would circumvent the issue. I'm not sure if this is desirable though, wouldn't it undermine Waylands security model?

[buffet]

Yes, that is expected (for security reasons).

It doesn't undermine Wayland's security model, as there are many other ways to completely log everything happening when you run stuff as root (or give out permissions for everything to everyone); just Google for "Wayland keylogger" to find many great disussions about this.

[antoniusf]

Oof, I didn't know that there were keyloggers for wayland. Thanks for the info! So the plan with evdev would be to ask people "to use the evdev backend, please add yourself to the input group"? Maybe I'll start looking into evdev too, it seems like it might be simpler (and more compatible) than a normal wayland backend.

[antoniusf]

Oh, I forgot to ask: Are you interested on working on this, too? Or did you just want to make a suggestion?

[buffet]

That would be the idea if we decide to go with udev.

Generally I'd be interested, mostly because I want to have Plover on Wayland though, not because I'm highly motivated.

[antoniusf]

Generally I'd be interested, mostly because I want to have Plover on Wayland though, not because I'm highly motivated.

Okay, yeah, same for me :D

That would be the idea if we decide to go with udev.

Wait, did you mean evdev? Or is udev also important somehow?

[buffet]

Eh, I meant evdev. Was very tired yesterday.

[antoniusf]

Okay, no problem! I don't know a lot about this, that's why I wanted to be sure. Thanks :)

[dragon788]

Can you hook into libinput or does going lower level with evdev make more sense? Working with libinput would cover Wayland and Xorg with a consistent interface as well maybe.

https://wayland.freedesktop.org/libinput/doc/latest/what-is-libinput.html#libinput-and-wayland

https://wayland.freedesktop.org/libinput/doc/latest/what-is-libinput.html#libinput-and-x-org

[antoniusf]

Hmm, so I skimmed the documentation very briefly and I didn't see anything about hooking into libinput. (evdev just provides files you can read/write from, if I remember this correctly.) Would there be any advantages to looking into libinput more deeply?

[dragon788]

After reading through a little more, I think you are write that if you are attempting to make a "virtual" keyboard interface you'll likely need to talk to evdev or a /dev/eventN device directly as libinput is more about enumerating the capabilities of devices or correcting for "quirks" of touchpads etc.

[antoniusf]

Okay, thanks for looking into this!

[verbalshadow]

Do you have a repo we can look yet? @antoniusf

[antoniusf]

Oh, thanks for asking, I probably would have forgotten about this on my own. So, there's some code I wrote a few weeks ago, which will basically make this work if you're using sway and an external stenotype (i.e. not the keyboard machine because implementing keyboard grabbing is kind of involved apparently).

The code is still unpublished because there are a couple of problems with it.

First, it needs a library to interact with wayland. The library that seems most mature here is pywayland, but that only provides v1 of the input_method protocol, but we want v2 (I don't even know if sway supports v1). It also requires libwayland to work, which is probably fine but is also the kind of thing that feels like trouble. There are also two other libraries (see sde1000/python-wayland and sapoturge/python-wayland) which can both work directly off of the wayland protocol files, which means you can just hand them the protocol file for input_method_v2 and everything should work. I hadn't found the first one yet when I started this, and I wasn't super happy with the second one, so... I decided to spend a day and write my own. This works for me, right now, but is obviously not shippable. The sde1000/python-wayland library might improve the situation a bit, but it isn't published anywhere so you would still have to ship it along with plover.

The second problem is that, in general, the entire implementation is currently more in the style of "I would like to be able to use plover again, right now", and it does some slightly confusing things that are okay for me but that I probably wouldn't be comfortable with other people using.

I haven't done any work on the evdev side of things, apart from looking into it briefly. For wayland, even if the problem with the interface library is fixed, someone would still have to sit down and implement keyboard grabbing, which last time I checked was still being worked on in sway. (See here for a library that actually uses it.) Doing this seems kind of annoying, and I don't think I will have the energy for it in the next few weeks. And even then, I don't think it would work for anyone without an appropriately patched sway. (i.e. basically no one, I guess.)

In the next few days, I can look over my current code and maybe add a bit of text on the pitfalls that it currently has and then push it to my fork, just in case someone is in the same position as me and just wants something that works, under sway, if you have an external stenotype. But I want to be clear that I don't think the current code is going to be particularly helpful in actually solving this issue in general, there's just too many problems right now. I'd guess that evdev really is going to be the way forward here, and maybe I'll start looking into that a bit more when I find the time, or someone else does it.

[frazerk]

Just wanted to mention that UHID seems like it's probably the correct solution for creating a proper virtual keyboard. I don't see any python wrappers for it, however.

[antoniusf]

Thanks for the suggestion! I just took a brief look at the documentation, and it seems to be fairly low-level. (You would have to emulate the HID events sent by a keyboard, right?) Do you think that there's an advantage to using this instead of evdev?

[kamalmarhubi]

Ah, I was planning on getting started with steno over the holidays, but I run on wayland (gnome)... :-(

I guess this means I may have some time to help on this issue? We'll see!

@antoniusf

Unfortunately, I don't know how well gnome supports these yet, but I will be able to talk about their support in sway.

I think neither of these will work in gnome because they're both wlroots extensions (for now). I'll do a bit of reading up on ibus and the v2 input method protocol, and see if there's a reasonable chance of abstracting over them both.

[antoniusf]

Sorry to hear that this has put a roadblock to your steno learning? :/

I look forward to hearing what you find out about ibus! (Don't stress though, it's totally okay if nothing comes back.)

[kamalmarhubi]

@antoniusf any chance you can share your rough code? would be useful to understand what kinds of changes had to happen. I'm unfamiliar with basically every aspect of this issue, so any connect would help :-)

[antoniusf]

Hmm, I'm not sure I want to publish the full code (due to the hacky wayland interface library I mentioned above, I should probably switch to something better), but I can show you the code that does the important stuff? If having the full thing (something you could run yourself under sway) is important to you, I can try to make it publishable in the next few days.

Context (as far as I understand it):

The file that's important for this is plover/oslayer/keyboardcontrol.py. It checks which OS you're using and then imports an OS specific file that implements two classes, one for capturing keyboard input and one for emulating keyboard output. It then subclasses them and makes them available to the rest of plover.

In principle, I would implement the wayland interface in the same way, by making a separate file for the wayland interface, just like there is one for the X interface right now, and then use some method to dynamically decide which to use.

The reason why I'm explaining all this is that the current version doesn't do it. All I did was modify keyboardcontrol.py to only support wayland, because I mainly wanted to get wayland support working and not think about how and under which conditions to dispatch to the wayland version instead of the one that uses X. But in a proper, mergeable implementation, that (I think) is what you'd want.

I don't think I can really explain wayland in this space, but if you're actually trying to understand the code, you may find the specification for the relevant part of the protocol useful, along with this section on basic principles of the wayland protocol.

I'm not sure how useful it is for you to read all this, since there already is a working implementation for this interface, and it's also the thing that won't help you if you're on gnome. ibus (ie what gnome uses) is totally separate from wayland, and I'm not sure that knowing how all of this works in wayland will help you with ibus. The wayland spec I linked above might help you understand how input methods work in principle (it did for me), but I'm not sure how accessible it is if you're not already familiar with how wayland works.

All this being said, here is what I currently have in my keyboardcontrol.py, to make keyboard emulation work (in parts) in wayland: https://gist.github.com/antoniusf/e78b2ffc6ae0e4bee7b8c63148655162

I hope that what I've written made sense to you, and that it wasn't too much. Otherwise, feel free to ask more questions.

[kamalmarhubi]

@antoniusf meant to say thanks for the pointers. I don't have time to poke around at the moment, but I really appreciate the in-depth explanation!

[antoniusf]

No worries, I totally understand! Thanks for getting back to me though.

[MarkoSchuetz]

This does not seem to have been mentioned before https://github.com/philipl/evdevremapkeys. Looks like it would be useful.

[antoniusf]

@BryanJacobs Would you consider reposting your bug report in the pull request itself? That feels like a more sensible place for it to be.

[killbox998]

One work around for using unicode with uinput is the compose key. Most desktop environments have a way to input a unicode character. for example in gnome CTRL+SHIFT+U, followed by a unicode code point in hex, followed by ENTER with input that codepoint. I have an implementation of plover using python-uniput and I use this method to write unicode characters. I know KDE also has this capability.

Also about uinput and root: all you need is to be part of the input group. this gives you user access to all input devices.

[ellbur]

Here is a demonstration of how plover can be modified to use evdev for basic keyboard emulation on Linux. It doesn't support unicode, and it probably lacks other features, but it is sufficient to be usable. It is also quite a simple modification.

If this would be a useful addition, I am happy to finish it. I think the next steps would be:

1. A configuration option, maybe under 'Output', for 'Use evdev on Linux', so that users only have to turn on evdev when necessary and use X otherwise.
2. Some form of error reporting so that the user knows if /dev/uinput is not writable and how to fix it (which will be the default on most distros).

[antoniusf]

Thanks for your efforts. I must admit that I'm unsure how this will continue to develop – Benoit Pierre had been working on the output plugin system (#1276), which was intended to be basis for work like this, and would have enabled both the evdev and wayland-based approaches to exist as plugins. However, with his moving on from the project recently, that PR has been closed.

I continue to believe that implementing output plugins is the right strategy to avoid having to maintain these new, specialized operating system interfaces in the plover core, but I am not sure how that work will continue now.

[johngebbie]

Also about uinput and root: all you need is to be part of the input group. this gives you user access to all input devices.

I don't think this is true unless you have udev rule installed which grants it so like:

KERNEL=="uinput", GROUP="input", MODE="0660", OPTIONS+="static_node=uinput"

Which is what dotool uses.

[Mateiadrielrafael]

Why not simply run plover as it's own user with an udev rule giving evdev/uinput access to it's primary group? I've used a similar approach before for another tool of mine and it works pretty well (although there needs to be a tutorial on how to do this for less experienced people)

[antoniusf]

Yeah, I think that that is definitely also an option. And a number of people already have made evdev backends, so feel free to just try and see if you can't make it work. Maybe, with good enough documentation, you could even make it work for less tech-savvy users, though it would still certainly be a hurdle.

[triallax]

I started using https://git.d2evs.net/~evan/plover-wtype-output on Sway a few days ago, and it's been working perfectly so far (kudos to Evan!).

[svenkeidel]

Hi, I tried plover-wtype-output with the plover main branch on Sway on nixos, but to no success. I checked that the plugin loads, but none of the functions KeyboardEmulation.send_* is ever called. @mhmdanas, could you elaborate how you got this working?

[LogicalOverflow]

On nixos with sway, I am using plover built using this and that "just works" - i.e. I just start plover, turn on output and it can output to all windows (xwayland or not)

[svenkeidel]

@LogicalOverflow, thank you for your suggestions. I tried your nix package. Plover now outputs steno in QT windows, but not anywhere else e.g., in the terminal (alactricitty) and browser (firefox). But this works good enough for me for now. Thanks.

[triallax]

Hi, I tried plover-wtype-output with the plover main branch on Sway on nixos, but to no success. I checked that the plugin loads, but none of the functions KeyboardEmulation.send_* is ever called. @mhmdanas, could you elaborate how you got this working?

I wish I could remember the details, but it's been a while. What version of Plover are you using?

[svenkeidel]

I tried multiple versions.

4.0.0dev10, 4.0.0rc2, main/HEAD all work when enabling plugin plover-wtype-output, but for me only in QT windows.

Starting from 4.0.0rc2 I get an error [MainThread] WARNING: Qt: Wayland does not support QWindow::requestActivate(), which causes plover to fail to output steno at all. I fixed it by setting environment variable XDG_SESSION_TYPE=x11.

I also tried Benoit's wayland branch https://github.com/benoit-pierre/plover/tree/wayland-sway. This works in all window types, but is based on an old version of plover and will probably not be merged into main as far as I understood.

[triallax]

4.0.0dev10, 4.0.0rc2, main/HEAD all work when enabling plugin plover-wtype-output, but for me only in QT windows.

It works for me in all windows.

Starting from 4.0.0rc2 I get an error [MainThread] WARNING: Qt: Wayland does not support QWindow::requestActivate(), which causes plover to fail to output steno at all. I fixed it by setting environment variable XDG_SESSION_TYPE=x11.

I get that error but it doesn't affect anything, I don't think it has anything to do with outputting steno in my case at least.

Just to be certain, do you have https://github.com/atx/wtype installed? It's a prerequisite for the plugin to function, and should be available in the Nix repos.

[stargazer0334]

Has anymore progress been made? Having difficulties building an appimage from source.

[dhelgerson]

hi, I'm running arch and I installed plover from the aur. it installed no problem but when I enable it, I still get a regular qwerty output. any help?

Edit: apparently it works on XWayland apps, not native wayland ones. I've also tried plover-git which compiles the latest github version

[antoniusf]

@stargazer0334 i'm afraid not, ever since benoit left and with him the idea of output plugins, which was how this was supposed to be integrated. people on the discord might know more though, i haven't been keeping up-to-date there.

@dhelgerson i assume plover-git uses the main branch though right? the changes here aren't even merged, so it makes sense that that wouldn't help. building this yourself is currently the only way to get a working thing, i think :/ (someone else please chime in here if i'm wrong!)

[david-boles]

https://git.d2evs.net/~evan/plover-wtype-output seems to no longer be accessible. Does anyone have an updated link or a copy?

[eclairevoyant]

https://git.d2evs.net/~evan/plover-wtype-output seems to no longer be accessible. Does anyone have an updated link or a copy?

https://github.com/svenkeidel/plover-wtype-output

[dhelgerson]

@dhelgerson i assume plover-git uses the main branch though right? the changes here aren't even merged, so it makes sense that that wouldn't help. building this yourself is currently the only way to get a working thing, i think :/ (someone else please chime in here if i'm wrong!)

@antoniusf, thank you for the reply. building plover is a little confusing for me at the moment. could you point me to the branch I need to build and relevant build instructions?

[lividhen]

This works on hyprland sort of. If you start typing on a window you cant switch to another window unless the plugin is disabled and re-enabled.

[amyipdev]

I have Plover working - no patches, just paru -S plover-git. Unlike what was mentioned by @lividhen, I can move across windows (including between Wayland-native and XWayland) and keep using it no issue. Currently on hyprland 0.36.0, wayland 1.22.0, wlroots 0.17.1; plover is the most recent as of this comment, reporting as 4.0.0rc2

[lividhen]

Hmm ill have to try updating things when I get home from work. Thank you for listing version numbers!

[halbGefressen]

Since wtype was not working for me, I made plover-output-dotool. I think it works as intended on Plasma 6 / Wayland, but I just did this last night in like 2 hours, so no guarantee that it is bug-free. Didn't test on GNOME, but should also work.

[antoniusf]

hi @dhelgerson, I'm really sorry for the long delay, life kind of got in the way. If you're still looking, the branch you need is here: https://github.com/benoit-pierre/plover/tree/wayland-sway. You'll want to clone that repository, and then check out the branch. There is information on how to build in doc/developer_guide.md. tox -e launch is the command you're looking for. If you need further assistance (depending on how familiar you are with python and the command line), you could consider asking for help in the discord and see if someone can help you there, so you're not stuck on me alone. Sorry again, I get that this is frustrating but that's how it is currently I guess.

[dhelgerson]

hi @dhelgerson, I'm really sorry for the long delay, life kind of got in the way. If you're still looking, the branch you need is here: https://github.com/benoit-pierre/plover/tree/wayland-sway. You'll want to clone that repository, and then check out the branch. There is information on how to build in doc/developer_guide.md. tox -e launch is the command you're looking for. If you need further assistance (depending on how familiar you are with python and the command line), you could consider asking for help in the discord and see if someone can help you there, so you're not stuck on me alone. Sorry again, I get that this is frustrating but that's how it is currently I guess.

Thank you. I'll try this out when I get the chance.

[FirelightFlagboy]

Hello, I'm reporting that I've made a nix flake to test both dotool and wtype plugins.

On my system dotool seems to be the only plugin that work across multiple windows (firefox, vscode, kitty)