Mangled text over the wire

[nirvdrum]

I need to track the problem down, but I wanted to file the issue here so it doesn't get forgotten. But the basic issue is sometimes certain text forms come across mangled.

E.g., I have this mapped rule:

"cap deploy to ": Text("SKIP_ASSETS=true RUBBER_ENV=") + Function(lib.format.lowercase_text) + Text(" cap deploy")

And when I say "cap deploy to kevin" with > 0.5 probability, I'll get something like the following typed:

SKIP_ASSETS=true RUBBER_ENV=ke cap deployvin

[poppe1219]

I can confirm this. But I mostly get when dictating free text, i.e. through the Aenea client window.
It doesn't happen very often, but often enough to be frustrating.

[calmofthestorm]

I've never seen this.

[nirvdrum]

I only ever use aenea with the client window, so it's possible that that's the source of the problem.

[calmofthestorm]

I use it too, and have never had any issues with it. Very peculiar. Any guess as to whether it's happening on the client or server?

[poppe1219]

Too me, it seems like parts of phrases arrive in a different order than they were sent.

[calmofthestorm]

Oh interesting...maybe try reverting before I added the multithreaded cache flush and see if you can reproduce it? It used to send each keystroke as it arrived, but had poor latency/bandwidth adaptability, so I added a threaded buffer that flushes as often as it can. The connections on the server may be being serviced in the incorrect order. We may have to add sequencing of some sort to fix this:/

[calmofthestorm]

If I'm correct, we'd need to ensure that aenea_client checks with the flush thread that the previous RPC completed before sending the next one. Hopefully we can do this all client side without messing with the RPC API.

[calmofthestorm]

Ok hmmm...so I think it's because execute_batch uses multiple actions, which is a notification not a request, meaning that the JSON-RPC API can return immediately. I believe changing it to return something (anything really, perhaps the number of commands executed) would cause aenea_client's worker_thread to block during each flush, ensuring that it occurs in order.

[poppe1219]

I believe JSON-RPC numbers each package. Could that be used to maintain the order?

[calmofthestorm]

Here's an easy test: in config.py, set USE_MULTIPLE_ACTIONS to False and see if the problem goes away.

[poppe1219]

I can try that.
Though, It might take some time to be sure.

[calmofthestorm]

I'm not sure this is the problem, but it's a good a place to start as any.

[poppe1219]

As you suggested, I tried changing the config value to 'USE_MULTIPLE_ACTIONS = False' and restarted. But that doesn't seem to be the cause.
I was trying to say:
"cap you're right"
But Dragon interpreted my slurry speech as:
"cap you write not"
Then after getting transferred to my host machine, I got this:
"Ywriou te not"

It looks like they were mixed like a deck of cards. Could be a threading issue?

[calmofthestorm]

Hmmm...one other possibility is that the xdotool delay is set too low. I just was playing around last night and had similar issues but only when dictating into the Windows VM via xdotool on the host. To rule this out, you can change:
write_command(text, arguments='type --file - --delay 0')
to
write_command(text, arguments='type --file - --delay 12')

in server_x11.py (I plan to make this configurable via settings today)

To be clear, my issues were NOT using the client.

[calmofthestorm]

I'm certain the mangling I've had issues with is due to --delay 0 in xdotool. It works fine with higher delay, or delay 0 when not dictating into VirtualBox windows via the proxy. So I think your mangling is a different cause.

I updated aenea_client to work with the new library today, and I did notice a race condition between the capture and flush thread involving modifier keys, so it's possible it's fixed. That said, I'm not sure what I fixed could cause "Ywriou te not".

One more question for @poppe1219: I noticed this in aenea_client:

aenea/client/aenea_client.py

Line 137 in 0423954

self.client.server.key_press(key='Control_R')

a comment about VirtualBox capture and a comment -- can you tell me about why this was there?

[poppe1219]

The right ctrl button is the button that triggers keyboard capture on/off for VirtualBox. So when the "Start capture" button is pressed, the keyboard capture for the virtual machine is released at the same time. I don't know why the comment says that it doesn't seem to work, because it works just fine.
This is a specific solution for VirtualBox, and should probably be a feature that can be defined in the config.

[calmofthestorm]

Thanks. I added that back in a sequence-safe way just to be sure, though it's unlikely to have been the issue. If anything it was the modifier key race condition. Try the new version and let me know if you can still reproduce it.

[calmofthestorm]

Are you still having this problem with the current version?

[nirvdrum]

Let me try it a few more days. I haven't had any problems yet, but since I had a sore throat, I haven't been able to run it a ton.

[calmofthestorm]

Had a chance to try it? :-)

[mzizzi]

Have experienced something similar to this and I have a hunch that xdotool might be to blame. Server logs show that commands came in order.

2014-08-10 21:18:51,690 [DEBUG ] [server] echo 'cookie' | xdotool type --file - --delay 0
2014-08-10 21:18:51,702 [DEBUG ] [server] echo '[]' | xdotool type --file - --delay 0

The resulting text was "cooki[]e"

It is possible that, while the commands were invoked in order, the executions themselves overlapped.

I'll see if I can't add some more logging to capture start/end times of the xdotool processes.

Thoughts?

[mzizzi]

So I looked at this in even further detail today and my thoughts from the comment above are probably incorrect.

SimpleJSONRPCServer processes requests synchronously and os.system() is blocking. So it doesn't appear that two xdotool commands should ever overlap.

I'm currently stumped.

[calmofthestorm]

I took a look at xdotool to see if maybe I screwed up the popens. It seems to work for me. Might be worth testing a variant of this script to see if it causes problems.
https://gist.github.com/calmofthestorm/5031a495f06fc21a9574

I'm also pretty stumped. Since I can't reproduce the issue it's difficult for me to investigate.

[squigglybob]

Hi, I'm new to aenea and am excited to start using it and making grammars.

I have made it through all of the configuration instructions up to the test_hello_world_aenea part.
There is a connection between the client and server but what I get in my server terminal is

"Aelea remote setur oreratiolap" instead of "Aenea remote setup operational"

This doesn't seem to be the same problem as above of mixing up the output but more like I am using the wrong keyboard layout in my server. Is aenea dependent on the server using a particular keyboard layout? For the above I was using the uk keyboard layout. If I use dvorak which I usually use the output is even more garbage "Aṅȧ pṁry˙oẏgp rpṗaycrnal"

[mzizzi]

Welcome!

That will ultimately call into this function on the server:
https://github.com/dictation-toolbox/aenea/blob/master/server/linux_x11/server_x11.py#L334

Which should result in the following command being issued:
echo 'Aenea remote setup operational' | xdotool type --file -

When you paste the above command into the terminal xdotool should write the expected text into window. If you paste the above command into the terminal is text still mangled for you?

[squigglybob]

Hi Mzizzi,

Thanks for your quick reply :)

I pasted in the above code and got the same thing out. So it would seem to be a problem to do with my xdotool.

My xdotool version is:
xdotool version 3.20140217.1

The only other thing I noticed re xdotool before this point of the setup was when I ran test_client.py it had the following lines in the output of the non-daemonized server_x11.py terminal

xdotool: unrecognized option '--delay'
Usage: xdotool

I don't know if that gives any clues...

(Thanks for your help by the way :)

[mzizzi]

Odd, I'm running the same version without issue.

$ xdotool version
xdotool version 3.20140217.1

I did some digging and apparently this is a know problem with xdotool. Some have found workarounds, however. Reply 29 in the thread may be useful:
https://code.google.com/p/semicomplete/issues/detail?id=13

[squigglybob]

hehe, I was just doing some digging myself and found a similar post at http://askubuntu.com/questions/422480/xdotool-cant-type-y with the same answer.

Upshot is I typed in the code
setxkbmap dvorak

and now the message comes through fine. Another one of those too many cooks spoils the broth moments. There must be at least 2 or 3 things vying for control of my keyboard layout lol ;)

[squigglybob]

Thanks for your help, and for your digging, much appreciated :) Now it's off to the land of grammar writing.

[djr7C4]

I would still suggest using the latest development version of xdotool. I have had problems with aenea using features not present in older versions.

[mzizzi]

I would still suggest using the latest development version of xdotool. I have had problems with aenea using features not present in older versions.

This isn't a bad suggestion.

@calmofthestorm has anyone else reported actual mangled text recently? I haven't been able to reproduce this issue.

[djr7C4]

I personally have experienced mangled text with older versions, but it has not been an issue (at least for me) for a while.