Unable to bind mouse buttons/clicks

[MostHated]

Classification:

UI/Usability

Reproducibility:

Always

Version

AutoKey version:
Gtk 0.95.10

System
Pop_OS! 20.10

Hello,
I am unable to bind any mouse clicks or buttons to any of the scripts. AutoKey is extremely fast, flexible, and would better than anything else out there if it could just listen for mouse clicks and bind them to scripts.

The main thing I am after is being able to change what the middle mouse click does in different applications. I already have all of the code for listening for window focus changes, detecting where in the windows the mouse is, etc, so that in VS Code I can have the middle mouse button do one action while in the code editing area but send a normal middle click when the mouse is over the tabs, etc.

The only thing I am missing is to be able to simply have AutoKey intercept the mouse click so that it can trigger my script. I have tried probably a dozen ways of going about it from remapping the middle click to a button press (which I don't like at all as holding it causes repeated presses when trying to use it as a normal web scroll on a page), while trying to use another script to watch what window I have focus on, then trying to use that to change my xbindkeys profile and restart it all in one go so that it takes effect (which has been nothing but a pain in the sack).

My most current attempt, which was using the EasyGesture software to intercept the mouse click, then lets you resend it as something else, or call a script. It actually works the closest to what I want, but it requires you to draw a tiny mouse movement, displays transparent boxes on the screen and text when you do it, and the location that it actually performs the middle click is not always where your mouse actually is at the time that you originally tried to click, etc.

I had always just used AutoHotKey on Windows and it was perfect for these types of things, but I think AutoKey would be even better due to allowing the use of python and all of the packages, the only thing it is missing is being able to capture a mouse click.

That said, is there any sort of way, or trick that I just can't think of to get AutoKey to fire a script when a mouse button is pressed? If not, might this please be considered for a future enhancement to the software?

Thanks,
-MH

[josephj11]

No specific suggestions for a workaround/solution. But here are some general thoughts. Python can do almost anything (and that's what our developers would have to use to add this feature.) So, if you can detect the middle scroll/click from Python, then an AutoKey script can use it.

I see a couple of avenues to explore. I don't really like the monitor script idea because it involves having an AutoKey script running long term instead of just on an event. That works, but doesn't get used a lot, so there may be bugs.

Figure out how to detect a mouse scroll event in Python. Have a running (monitor) script use that and when it detects it, do what you want or launch another script.

Same idea, but use some external tool to detect the mouse event. This tool could do one of several things.
It could run an AutoKey script externally using our scripting module - probably the best idea - doesn't need the monitor script.
It could set a semaphore such as touching a file that the monitor script would keep looking for and would reset once it finds it.

This is largely unexplored territory, so come over to Gitter and post about your progress. That's where the devs and advanced users hang out. They may be able to help and anything you discover may help them eventually add it into AutoKey.

[sebastiansam55]

My two cents:

There are some really cool python libraries out there that make it super easy to listen to input events. The one that I am personally most familiar with is the python-evdev library.

What I would do is make a small python script; along the lines of (from the evdev examples)

from evdev import InputDevice, categorize, ecodes
dev = InputDevice('/dev/input/event1')

print(dev)

for event in dev.read_loop():
    if event.type == ecodes.EV_KEY: #mouse clicks are considered to be "keys"
        if event.code == ecodes.BTN_LEFT:
             #do logic to handle left clicks
        elif event.code == ecodes.BTN_MIDDLE:
             #do logic to handle middle clicks
        elif event.code == ecodes.BTN_RIGHT:
             #do logic to handle right clicks

It sounds like what you want to do is reliant on the program knowing what window is being clicked on and perhaps in what window. To do this you can probably use the autokey-run command. I haven't tested it myself but I don't see any major issues assuming autokey works the way that I assuwe it does.

In case you're not a python whiz the autokey-run implementation in python will probably look something like this;

import subprocess
subprocess.call(["autokey-run","-s","<scriptname>"])

If you really wanted to dig into it you could probably rip the code straight from the autokey-run file in order to talk directly to the autokey dbus service.

[josephj11]

When you start listening to and responding to mouse events, does that "consume" them or do other processes that may already be watching/responding to them still see them? Where on the input event food chain does this fit and what possible interactions/conflicts do we have to avoid/be prepared for?

We already have some similar issues on the keyboard side where the desktop gets some events before AutoKey sees them and when a keypress that the user expects to go to the desktop is sent directly to the application window. These are not bad things, but we need to keep track of them to avoid confusion (eventually needs a wiki page).

[MostHated]

In my limited experience playing around with pynput, it seemed that the mouse click was not consumed, and was passed on. Though, I am merely a beginner when it comes to Linux system input, so would not really know what to try in order to stop it.

I was messing around with making an addon for the Linux version of the software suite Houdini using QT/Python before and was able to fully consume the input, but that may just be because QT had some methods specifically for controling propogation.

[sebastiansam55]

It can do either. What I typically do is "grab" the device exclusively for input signals, and create a new uinput device for the output signals. The good part about using a uinput solution is that it would be pretty agnostic about X, gtk/qt, and even runs in the TTY.

If you want to see a real world example of the evdev library you can check out this single file keyboard remapper (qwerty to dvorak)

https://github.com/sebastiansam55/uinput-keyboard-mapper/blob/main/uinputremapper.py

I think that my code should be pretty straight forward if you have same basic python under your belt, but feel free to DM me on gitter if you have any questions.

[NSC9]

Relevant comment

[NSC9]

I am considering opening a fiverr request to solve this ticket and use my own fork and share it freely with you all.

I really want to see AutoKey simply allow in the GUI the option to ‘bind to left mouse button’

I desperately want to convert my AutoHotKey script to be working smoothly on AutoKey. I’m afraid the workarounds proposed will lead to poor performance and introduce many bugs. The solution has to exist and be simple.

I rephrased this problem at #243. I want to believe that Linux based OSs like Mint can do anything Windows OS can do. I do not want to ‘run AutoHotKey on Linux through some janky API.’

Does anyone have suggestions for keywords for a fiverr request for the spirit of these tickets? My first guess was “coding Python Linux.” I have no idea where to start programming and contributing to the AutoKey repo so I am hoping someone might be able to suggest better keywords for the job’s bounty I want to request.

Of course I would link hired coders to these tickets for reference.

[josephj11]

@NSC9 Thanks for this initiative! AFAIK, this is the first time someone has considered investing significant funds (not directly to us) to benefit our project.

Not sure about the keywords to use. AutoKey is probably sufficiently well known so that you could include something like AutoKey enhancement or more generally, desktop automation.

Something to keep in mind is that @sebastiansam55 has developed a new keyboard and mouse interface for AutoKey which uses uinput and does most of what AutoKey currently needs, but works in Wayland. This works, but we are just starting to integrate it into our develop branch and it really hasn't been tested much by anyone else yet. No idea when that will get integrated into an actual release. As currently coded, AutoKey detects whether X11 or Wayland is running and chooses the appropriate interface.

This doesn't directly address this issue, but if/once it gets fully integrated and released, it may partially or completely replace our current keyboard and mouse interface.

This is relevant to you because you may want to avoid spending significant time, effort, and funds on a solution that is dependent on X11. Wayland is gradually taking over the Linux GUI world, so any solution you come up with should probably be able to work in that environment if you and/or we switch to using it as our primary choice.

X11 isn't going away any time soon, but it looks like it will eventually be depreciated.

Although I don't like referring users away from AutoKey to other projects, you may want to take a look at Espanso to see if it better addresses this issue. I have not used it at all, but it claims to already work in Wayland. It doesn't do everything that AutoKey does, but it is being actively developed. Notably, it does claim to have some ability to convert AHK scripts into its native scripting language.

For more information, see #775, #866, and #844.

[sebastiansam55]

@NSC9 I just updated the wayland branch to support this functionality. It may be a bit complex to setup but I've been using it as my daily driver for a while. you may want to try it out but it should be considered like a pre-alpha, I did have to reboot to get out of a softlock earlier today but that was because I'm actively developing on it which involves starting and killing it a bunch.

It does not have 100% support for all the autokey APIs at this time. (mainly mouse movement and I think the keyboard listen/wait for is not yet implemented because it's complicated).

If you want to try setting it up the instructions in the PR should get you there, if you have trouble open a discussion I think would be the best place.

I'll try to look into how/if mouse event support can be implemented on the x11 side but I'm not sure it can be with the way the keyboard support is implemented under x11

wayland/uinput branch: #844