Phrases are sometimes incorrectly expanded, missing character(s)

[ronjouch]

Classification:

Bug

Reproducibility:

Sometimes (fails ~ 30% of the time)

Summary

Phrases are sometimes incorrectly expanded, missing character(s).

Steps to Reproduce

• Create phrase myname@gmail.com
• Bind it to abbreviation m@g, with options:
  • Remove typed abbreviation
  • Trigger immediately

Expected Results

m@g should expand to myname@gmail.com

Actual Results

m@g sometimes correctly expands to myname@gmail.com (~ 70% of the time), but sometimes fails to do so, missing one or several characters and expanding to for example mynme@gmail.com or myname@mail.com or myname@gmail.c.

Version

autokey-gtk 0.94.1

Installed via: PPA

Distro: Ubuntu 18.04 LTS, fully up-to-date, running Xorg.

Notes

No workaround found

[ggada]

I'm facing a similar issue when I try to expand a phrase at the start of the line (v0.94.1 installed via pip).

Steps to reproduce:

1. Setup a replacement phrase.
2. Open up a google doc
3. Start typing a replacement phrase at the beginning of the line.

For example foo --> foobar

Then you type foo, you will be left with ffoobar (first character of replacement phrase doesn't get replaced) and sometimes a newline is added to the end of the replacement text even though there is no newline in the actual replacement text configured in autokey. (I verified that)

This is happening with both Ctrl+V and keyboard replace methods.

[luziferius]

I tried to reproduce it and failed.

• Tried both autokey-gtk and autokey-qt.
• Reduced my system CPU clock speed to (the minimum of) 800MHz.
• Tried local text editor and a javascript based web one (google docs).

I suspected that it might be some performance issue: The editor chokes when autokey presses the backspace key and loses keystrokes (because the javascript used in the web editor is slow), which result in incomplete operation and leftovers at the beginning.

Please start autokey in verbose mode (--verbose or -v switch on the command line), trigger a phrase and look at the output. It outputs a lot of stuff, but there are entries like when triggering a phrase:

2018-05-25 12:42:55,959 DEBUG - interface - Send special key: [<Key.BACKSPACE: '<backspace>'>]
2018-05-25 12:42:55,960 DEBUG - interface - Send special key: [<Key.BACKSPACE: '<backspace>'>]
2018-05-25 12:42:55,964 DEBUG - interface - Send special key: [<Key.BACKSPACE: '<backspace>'>]
2018-05-25 12:42:55,965 DEBUG - interface - Sending string: 'abcdef'

In my example, triggered the phrase »abcdef« using the abbreviation »foo«.
Count the number of backspaces sent and compare with the trigger length, it should match.

Another thing you could try:

• Open a terminal window, start showkey -a (and keep the terminal window focussed). The showkey program shows you all pressed keys. (Don’t use xterm as your terminal for this purpose, because xterm ignores all keypresses done by AutoKey.)
• Trigger a phrase. In my example, I type »foo«, AutoKey sends three backspaces (those: ^? 127 0177 0x7f), then expands the phrase (seemingly faster than showkey handles it).
• Check if everything looks fine (number of backspaces matches trigger length, phrase complete,
  etc.)

Example output, replacing »foo« with »abcdef«:

f       102 0146 0x66
o       111 0157 0x6f
o       111 0157 0x6f
^?      127 0177 0x7f
^?      127 0177 0x7f
^?      127 0177 0x7f
abcdef   97 0141 0x61
         98 0142 0x62
         99 0143 0x63
        100 0144 0x64
        101 0145 0x65
        102 0146 0x66

[ronjouch]

@ggada I also sometimes sometimes get your foo --> foobar case replaced as ffoobar (first character of replacement phrase doesn't get replaced).

@luziferius thanks for the fast feedback 👍, digging in with --verbose and showkey -a. But regarding

I suspect that it might be some performance issue: The editor chokes when autokey presses the backspace key and loses keystrokes (because the javascript used in the web editor is slow)

... I'm not sure: the issue isn't only happening in heavy webapps full of js potentially reacting slowly to input; I get the problem also in a terminal or Sublime Text.

[ronjouch]

@luziferius well, my memories were wrong: I'm able to reproduce the problem in heavy-on-keypress contexts (e.g. the Firefox URL bar, or a webapp), but not in a terminal on Sublime Text. And I tested showkey -a and --verbose, everything looks 👍 (good number of backspaces, and phrase always complete). So that confirms your intuition that it's a performance issue, and sorry for my unchecked affirmation above.

What can we do from here? Could you guide me to a delay-between-keysends patch I could test for a while?

[luziferius]

@ronjouch Thank you for checking back on this.
So this looks like it is indeed a performance issue in the target application. I added some delays to the key send function in the changeset referenced above.

You can either checkout my testing branch (here: https://github.com/luziferius/autokey/tree/phrase_expansion_timing_issues_test, use git to fetch the branch or download as a ZIP archive), or edit the source files directly and apply the fix yourself.

Either way, you may want to tweak the delays used, because I just guessed some arbitrary value.

[ronjouch]

@luziferius 👍, testing this in the next few days on my machines.

@ggada ping, maybe you want to give it a try too.

[ronjouch]

@luziferius feedback:

• I confirm your patch notes that this "leads to bad performance, as it slows AutoKey down considerably"; phrase expansion feels suuuuper choppy.
• Waiting 0.01s (and 10ms is already a lot, right?) the problem still happens in the Firefox URL bar :-/ . Trying more, but it quickly becomes 🤮.
• Parenthesis and maybe-useless/maybe-fruitful comparison to another system. Under Windows and with AutoHotKey, text expansion (called Hotstrings in AHK) is instant and reliable. Does Window have a different way to insert text? Have we in Autokey looked at all the APIs that Xorg (or whatever available subsystem) offer and that may be more reliable?

[luziferius]

• That was expected, since this is using the brute force slow-down method.
• Actually, it is 20 ms per key ;-) This 'fix' holds the key for 10ms, releases, and then waits again for 10ms to press the next key.
• AK can use three systems for phrase expansion (Available using the Paste using options):
  • Keyboard types each character, as if you were typing really fast
  • Clipboard uses the keyboard clipboard
  • Mouse Selection uses the mouse clipboard (it is normally used to paste the mouse selected text using the middle mouse button).

But each one uses the typing approach to delete the trigger text using the backspace key. Maybe some other approach is more fruitful, like pressing [Shift]+[Left] length_of(trigger_phrase) times, effectively selecting the trigger phrase. Then delete it using one backspace keystroke, and then pasting the phrase.

Which pasting mode do you use? I suspect your phrases use the Keyboard setting.

[ronjouch]

@luziferius yes, all my phrases expand using Keyboard, I don't want them to interfere with (mouse) clipboard. But thanks for the reminder, I could trade expansion reliability with clipboard being overwritten by Autokey.

1. By the way, if using the clipboard, couldn't Autokey restore preexisting clipboard contents after doing its [filling clipboard with expanded phrase, triggering Ctrl+V] dance? Want me to file an issue?
2. Second by the way 😄: I gave Mouse Clipboard a try and using it 100% crashes Autokey; known issue or should I file a new one?

Regarding

"Maybe some other approach is more fruitful, like pressing [Shift]+[Left] length_of(trigger_phrase) times, effectively selecting the trigger phrase. Then delete it using one backspace keystroke, and then pasting the phrase."

3. Yes, that could improve @ggada's reported case of foo --> foobar being replaced as ffoobar (first character of replacement phrase doesn't get replaced)...
4. ... but that won't improve my case of kbc --> keyboardcat being replaced as sometimes keboardcat, sometimes kboardcat, sometimes eyoardcat.

[luziferius]

I think, opening two issues is sensible, so go ahead.

1. About the clipboard replacement: I think, AutoKey is actually supposed to restore the clipboard after pasting the phrase…
2. It seems to be a new issue. (And only happening in the Gtk GUI.) Edit: This one is fixed. (interface.py Bugfix: Fix the mouse selection paste mode in autokey-gtk. #152)

I’ll see if I can come up with a fix (at least fixing the clipboard issues), but I have no experience with low-level X11 stuff, so I can’t promise anything further.

[ronjouch]

@luziferius thanks for the express crash fix! And here's #153 regarding restoring clipboard contents.

[hfr-hfr]

The problem still exists. In some heavier applications (Thunderbird, LibreOffice) the keyboard entry method is not deleting the trigger text completely and mixes up the order of the input text.

My workaround so far: I replaced phrases by scripts. For each script I disabled "remove typed abbreviation". Instead I send my desired output text and the length of my trigger text (X) to a function. This function first hits backspace X times with a delay in between. Then it iterates through my output text, also with a delay between each keystroke. "time.sleep(0.01)" works just fine for me.

It works, but it's not that convenient since I have to enter the length of each trigger text manually. Maybe it would be useful to have an adjustable delay as a global feature? Maybe in global settings? This would help a lot.

[herrdeh]

Here is my experience - hopefully it helps:

LibO

"iban" creates 3 different solutions:
ibDE3976090900222366720
iDE3976090900222273606
ibaD397609090022227360
... instead of:
DE39760909002222736600

"sgdh" reliably expands to:
Sehr geehrte Damen und HeSehr geehrte Damen und Herren,

Evolution

"sgdh" produces:
sgdhSehr geehrte Damen und Herre
occasionally:
sgdSehr geehrte Damen und Herren

"mdfg" goes to:
mdfgMit Dank und freundlichen Grüßen,
Wolf Drech

Firefox

In FF these results:
sSehr geehrte Damen und Herren,
->
sgSehr geehrte Damen und Herren,
sSehr geehrte Damen und Herren,
instead of
Sehr geehrte Damen und Herren,
sgr ->
sgSehr geehrter Herr
Sehr geehrter Herr
Sehr geehrter Herr
sSehr geehrter Herr
sSehr geehrter Herr
sSehr geehrter Herr
sSehr geehrter Herr

sge ->
sSehr geehrte Frau
sSehr geehrte Frau
sgSehr geehrte Frau
Sehr geehrte Frau
sgSehr geehrte Frau
###Version
AutoKey version: 0.95.1
Used GUI (Gtk, Qt, or both): unsure - using GNOME desktop
Installed via: http://ppa.launchpad.net/sporkwitch/autokey/ubuntu
Distro: ubuntu 18.04

I closed #189, which contained this info.

[luziferius]

I’ll drop my plan for this issue here, to keep you informed.
So this is very likely a performance issue or maybe input buffer ordering issue in the receiving application. It might be worth reporting or discussing on the LibreOffice mailing list or such. Especially since @herrdeh experiences almost full duplicates in output.

In this comment #151 (comment), I proposed an experimental patch that slows down autokey. I’ve rebased/updated the branch to the latest git master, so you can try this patch by yourself. The result was that it doesn’t really help. According to @ronjouch, it feels super slow and choppy. It improves the accuracy, but not up to 100% with the sleep time values used in it. Having a globally slow autokey, just to comfort some applications is a bad thing.

So my plan on this is as following:
I try to use this delay based approach and make it as configurable as possible, by allowing you to specify keystroke duration and delay between keypresses on a per-application basis.
The configuration will be added to the settings dialog and contains a table with these colums:

• window filter: A regular expression matching the window class. This allows identifying applications based on an internal WM_CLASS property that does not change during runtime and is stable enough to identify applications.
• Key press duration: Lets you configure how long each key is kept down
• Inter-key delay: Lets you configure how long autokey waits between releasing a key and pressing the next key.
• Maybe: The key press duration and inter-key delay specifically for the backspace key when used to delete the trigger phrase.

This allows adding as many “workarounds” as needed for specific applications, but does not slow down autokey in general use.

[dorian-marchal]

Same issue in VScode, here.

[nikolayhg]

have same issue since years. Pretty much as @herrdeh described it and with different results/outputs. Very often in Firefox.

[image357]

Same issue here: foo -> ffoobar

[luziferius]

I don't know if I can solve this, while retaining acceptable expansion speed. I think it is caused by the receiving application choking on fast input.

Use short abreviations and the clipboard pasting where possible to work around this issue.

[nikolayhg]

I would be much happier if the expansion is slower, but correct. Because I spent more time manually checking every character of the expansion. And sometimes my manual checking is not correct.

• The length of the abbreviation does not seem to play role.
• In the phrase settings I tried Paste Using Clipboard, but still receive incorrect expansions.
• I tried using a script, but the paste is not working as described in fill_selection script not pasting 'hello world' #245

[luziferius]

You can try to run AutoKey from this https://github.com/luziferius/autokey/tree/phrase_expansion_timing_issues_test feature branch in which I added some sleeping between key presses. This is the full diff against the current develop branch, showing the whole change:
develop...luziferius:phrase_expansion_timing_issues_test
The first sleep determines how long the key is approximately pressed, the second one defines the time between key presses

(Well, technically, it also contains these bug fixes when compared against 0.95.4)

You can download and extract a ZIP from the first link or just clone my fork and checkout that branch.
Then:

• Exit a running AutoKey instance
• Start it from the source code, by using the Zero-installation method as described in the README
• Try to expand something.
• If it does not work:
  • Exit AutoKey
  • open lib/autokey/interface.py and edit the sleep values around line 990, (see the diff link above.)
  • restart and retry

Please report back, if you found some time values that work reliably on your pc.

Edit: This branch implements roughly the same thing as that type_slow function you quoted.

[nikolayhg]

I found a temporary workaround that works in >80% of my current test cases (which is a big improvement). If there is a significant change in these initial statistics, I will report it.

Use the type_slow() function from https://github.com/autokey/autokey/wiki/Scripting#function-to-type-text-slowly . Might be worth to add it as a standard autokey function, so that the user does not have to defined it in every script. My tests were with delay=0.01 in Firefox.

import time

## Type a string character by character
##   from within AutoKey
##   with a small delay between characters
def type_slow(string, delay=0.1):
    if delay <= 0:
        # No delay, directly relay the whole string to the API function.
        keyboard.send_keys(string)
    else:
        for c in string:
            keyboard.send_keys(c)
            time.sleep(delay)

type_slow("my expansion")

Update: I get also similar results with delay=0.005
Update Last: still having issues, see below.

[nikolayhg]

Hm, my initial tests with type_slow() were mostly good (as described above), but now I still receive enough incorrect expansions to not be satisfied, even if there is longer delay=0.1 (see the gif below expanding 'brff' to a 'Best regards, \nFirstName FamilyName'). The delay delays the typing of the text characters, but the first abbreviation character 'b' is not affected by this delay.

@luziferius does feature branch delays also the deletion of the abbreviation? That might be one of the issues.

[herrdeh]

I tried this one as well

I just tried this script in LibreOffice Writer:

for _ in range(1000):
    keyboard.send_keys("Sehr geehrte Damen und Herren,\n")

Looks like this on my Autokey:
screen

My machine goes crazy with it - the app on which i tried to launch it gets blocked.

[opennomad]

It seems that a single line has a higher likelihood of success than multiple lines. I've been experimenting with populating the clipboard and then pasting it, which seems much more reliable on my end.

date_string = system.exec_command("date +'%A, %B %d, %Y'")
output = """## %s

*foo*:
*bar*:
*baz*:

""" % date_string
clipboard.fill_clipboard(output)
keyboard.send_keys("<ctrl>+v")

[lkorba]

It is weird. On my old Dell E7470 with latest Kali Linux I do not have this problem in any of the applications.
In my much newer Dell Latitude 7400 with Ubuntu 19.10 I see this problem most of the time... I use abbreviations mostly in Firefox...

Any progress finding out how to fix this?

[herrdeh]

Will we see any progress on that some time?

[DoctorSubtilis]

So far I solved this problem disabling in KDE Neon libreoffice-kde (and this worked).
Today I tried an upgrade of LO via PPA and it was a disaster: now I have to choose between a very bugged layout of LO (and autkey working) or a accettable layout of LO (but autokey not working correctly).

[bjohas]

Hello all, I've got the same problem (Ubuntu 20.04, GNOME Shell 3.36.1). For me it's usually the first character of the abbreviation that is left over (as described in some cases above). Would love a fix for this!

[josephj11]

I'm not a developer, but my understanding of this issue is that solving it, if possible, would require a major rewrite of that portion of the code. Just use Paste using clipboard with your phrases and in your scripts. That almost always works.

[DoctorSubtilis]

Yes, I confirm what said @josephj11

[miskovics]

I face the same problem. Now changed all phrases to use Paste using Clipboard (Shift-Insert), let's see if it will fix the issue for me.

If the only reliable option is Paste using Clipboard, may I ask why aren't the other options (Keyboard, ...) removed from the program? This would eliminate/dramatically reduce the number of people running into the problem described in this ticket.
Do we need 3 kind of Paste using Clipboard options or one would be sufficient?

@DoctorSubtilis are you confirming that Paste using Cipboard almost always works? Or that solving the problem would be a major rewrite?

@josephj11 Why would the change be a major rewrite? I seem to remember from the past that sending the characters was implemented at several places in the code. Is this now unified to one place? If yes, I assume only this part needs to be changed, plus the UI. I also assume that implementing the UI change is a bigger effort, so what about just skipping it for now and read the delay parameters from the configuration file which we have to edit with an editor?

[lkorba]

It is a problem with ibus

Simply try:
sudo killall ibus-daemon

and try again in Firefox

You're welcome... Now can someone please fix it? :

Thanks,
Lukasz)

[DoctorSubtilis]

@miskovics: sure!
Even though other minor problems rise, this way (sometimes instead of desidered phrase you get the clipboard content, as I already said elsewhere). But in these casesistenza it enough repeat the action and the result is what you expect.

[josephj11]

@miskovics I don't know the source code. I have very limited knowledge of Python. I am just recalling comments from developers.

We can't just remove an API call or even make it a dummy call to another one without potentially breaking a lot of user scripts. I did post on Gitter, suggesting that Paste using Ctrl+v be made the default option for new phrases, but didn't get much traction.

I've almost never used shift+insert. Never needed it. We do need more than one method though. E.g.: Most terminals use ctrl+v for something else and insert with ctrl+shift+v. Given the diversity of Linux, there are almost certainly other edge cases out there.

Yeah. Characters aren't being sent from just one place in the code. There's a weird difference between the behavior of keyboard.send_key() and keyboard.send_keys() that was just noted on Gitter. keyboard.send_key() doesn't seem to honor shifted characters like {, but keyboard.send_keys() does.

[miskovics]

@josephj11

API: as much as I can see, the AutoKey API (https://autokey.github.io/lib.scripting.Keyboard-class.html#send_keys) does not let a programmer specify which method is used to send the characters. BUT even if it was possible to specify the method via the API, we could still hide this configuration option from average users in the GUI and keep the APIs powerful, as they are, for the experts --> no confusion for beginners, plus scripts are still working as before.

Shift+Insert, ... : as much as I know, CTRL+SHIRT+v works everywhere, also in terminals. You are, of course, right about Linux's diversity, but then this would also mean that we just can't prepare for whatever might come in our direction. Maybe even the current options would not cover those special use cases popping up in the future, unexpected. --> we could just stick to CTRL+SHIFT+v, hide the configuration option in the GUI, and still have it in the configuration file for the experts. --> leaner UI, still same power.

@lkorba

Is this then an ibus-daemon, a Firefox, or an AutoKey bug?

[josephj11]

@miskovics What I meant was that your use of the API determines the method used. If you use the keyboard module, you're emulating typing. If you use the clipboard module, you're emulating pasting and your subsequent explicit call of keyboard.send_keys() with shift+insert, ctrl+v, or ctrl+shift+v determines the pasting method used.

Using the API is essential for almost all scripts. It's the only "simple" way for a script to talk to the active window. You could write a script that interacts with your system in another way, but that would need to use something like the subprocess() module which is arguably a more advanced technique.

The only place ctrl+shift+v works for me is in a terminal window. It does not work in most applications.

This option is only relevant when configuring AutoKey phrases. In a script, you have to code the API calls explicitly and can choose whatever works for the problem at hand.

The future:
Currently, there are three different clipboards in Linux. This has been a constant for a long time. I have no idea how long. These clipboards are widely used, so it is extremely unlikely that they would be removed or changed. AutoKey directly supports the use of two of these clipboards with the API - the regular clipboard and the X selection. (I have never figured out what the third clipboard is or how to use it, but I have seen references to it.)

If something new does come along, a large part of the desktops and other system utilities are written in Python, so Python will support whatever it is for their sake and we will come along for the ride because our scripts are written in Python and can access whatever they dream up.

Where it gets dicey, is when they want to replace X11, which we rely upon, with something new like Wayland. Currently, we do not support Wayland and have a help wanted issue open for anyone who would like to try to fix that.

[lkorba]

@miskovics I can only guess that it is:

• not a Firefox bug (it works in Firefox when you kill ibus)
• probably not an ibus bug
• probably autokey interoperability issue with ibus / way ibus treats virtual keyboard input

But I have no clue how ibus works, I only found out how to make autokey working again

[Izaya-San]

Same issue for Ubuntu 20.04 with Gnome writing into VsCode.

[tomboe311]

Same issue for Ubuntu 20.10 with KDE

[josephj11]

If you run into this, use Paste using clipboard (Ctrl+V) with phrases. (But it can't do things like phrase macros.)

If that doesn't work for you, you'll be stuck writing scripts that emit events slowly, one at a time, with delays in between them. See my type_slow() function for an example.

This is a longstanding bug and I don't think anyone knows how to fix it, so we may have to live with it indefinitely.

[opennomad]

i have found that terminating the ibus-daemon makes things very much more consistent for me on Ubuntu 20.10 (same on 20.04). Sadly, I have to run my local startup scripts outside the regular startup process, to make sure the ibus-daemon is dead before auto-key starts.

# kill ibus
killall ibus-daemon
# now start apps that struggle with ibus
NOHUP_OUT=${HOME}/.nohup.out
rm ${NOHUP_OUT} || true
nohup autokey-gtk >> ${NOHUP_OUT} &

I have a couple of other keys that go whacky when the ibus-daemon goes away (copyq, flameshot), and I start those up the same way.

I've been running this for months now and things have been pretty solid. Just switching to a new laptop I started having issues again, so I'll be sticking with this approach.

[josephj11]

@opennomad Thanks. I'm not familiar with ibus. I did a little poking around and it looks like it's not doing anything on my system (kubuntu 18.04):

bigbird@sananda:~/pq$ ibus engine

(process:26883): IBUS-CRITICAL **: 02:37:26.460: ibus_bus_get_global_engine: assertion 'IBUS_IS_BUS (bus)' failed
No engine is set.
bigbird@sananda:~/pq$ ibus list-engine
Can't connect to IBus.
bigbird@sananda:~/pq$ 
bigbird@sananda:~/pq$ ps -ef | grep [i]bus
bigbird@sananda:~/pq$ 

[allanlaal]

cross-posting my observation from #596 :

I set "type_delay": 0.001

I can't remember if this has already been discovered, but it seems with superfast keyboard emulation typing there is not enough time for entering keycombo characters like : / ? = & (shift + numrow keys on my estonian layout):
httpsxgis.maaamet.eemapsisappid82useridat11620958zlevel10,549052.10876467,6546996.6812135setlegend1821

Expected:
https://xgis.maaamet.ee/maps/XGis?app_id=UU82A&user_id=at&LANG=1&WIDTH=1620&HEIGHT=958&zlevel=10,549052.10876467,6546996.6812135&setlegend=UUKAT1_82=1

the tool I use to diff the results: convertonline.io/url/diff

[allanlaal]

"type_delay": 0.3 is the quickest that works for me (2.50GHz × 8 i7 at 80..150% (load: 7..14) + 32GB RAM, SSD)

this workaround is by definition so slow that with longer autocompletes I might aswell go make coffee while I wait, which kinda negates the point of using text files to copypaste stuff from vs Autokey.

has there been any other alternatives to slow typing? I tend to use my computer at its limit and other apps need 10x more load to start acting weird :|

[josephj11]

@allanlaal Some notes:

Paste using clipboard (Ctrl+V) doesn't work for you?

Since you're not using an EN-US locale/keyboard, you are probably using a Unicode character set that our keyboard methods won't process correctly anyway.

Are you using my type_slow() function or some other code?

Does this problem occur for you in all applications or just in some?

Usually, we see problems like this with LibreOffice and the Firefox address bar, but most other applications, especially simpler applications like terminals and plain text editors tend to work fine.

If this problem is persistent for you, you might want to try running xdotool from within an AutoKey script. xdotool (and the associated Python xdo module) uses a different API/library than we do, so sometimes it will have better results.

[josephj11]

I'm thinking of closing this and #380 as won't fix because the workarounds almost always work and no one has been able to figure out a solution in over four years. Also, if we are able to change to new input/output methods to accomodate Wayland, this would have to start over anyway.

Any thoughts?

[Elliria]

Yes to both. Although Ubuntu 18.04 isn't at EOL yet, it shipped with AutoKey 0.90.4 and each of these issues are about AutoKey 0.94.1 and AutoKey 0.95.10.