MouseDrag - Pointer not drawn on screen

[pevogam]

Hi Marc,

Do you know if the cursor is supposed to be drawn during a mouse drag operation? Currently, using Fedora virtual machine and a Qemu hypervisor (for instance but also noticed on other OS vms) the mouse is not drawn during the drag calculations with any step size and extra timeouts until it reaches the last position when it is updated. Is there a way to force the cursor to be drawn on screen?

Thanks a lot,

[sibson]

Increasing the sleep here might help and if so it could be exposed as a command line arg rather than hardcoded.

[pevogam]

This is what I did - I increased this timeout to the daring 1.2 seconds but it didn't help. However, the cursor is drawn/restored in the end of the MouseDrag when we arrive at the destination position. I also tried reducing the step size to 1 pixel and increasing the timeout to 1.2 seconds but the mouse remains invisible until it reaches the end destination (which also lasts a lot for this step size and timeout among steps).

[NethajiDuvarakanathan]

Hi Pevogam,

I am glad to be here in this conversation.
I am trying to drag the mouse using vncdo -s serveripaddress drag xy
But nothing is happening.
Please could anyone help me in this.

[sibson]

I would expect the cursor to be drawn on a physical host display but perhaps not on any vnc or remote displays. Depending on how the vnc server and clients are implemented they might be busy processing the setCursor commands rather than sending screen or cursor refreshes.

Making the cursor visible was outside the design scope of the drag command. I'm curious what use case you have that requires it to be visible thoughout the operation?

[NethajiDuvarakanathan]

Thanks for the reply.
Here I would like to elaborate the info from your reply statements:
"I would expect the cursor to be drawn on a physical host display but perhaps not on any vnc or remote displays "
detailed info :

1. I hve installed VNC server in my mobile.
2. via VNCDOTOOL commands am trying to initiate commands & trying to control mobile icons / apps [ eg: vncdo -s serveripaddress move x y click 1
   vncdo -s serveripaddress key a

"Making the cursor visible was outside the design scope of the drag command. I'm curious what use case you have that requires it to be visible thoughout the operation? "

1. Here my intention is not to make cursor visible.
2. Rather, I have drag options in my mobile to increase / decrease dimming level , which i try to drag via VNCDOTOOL [eg: vncdo -s serveripaddress drag x y ]

mayb am not sure, the command which i mentioned might be wrong one.
If so, pls correct me.

Thanks.

[NethajiDuvarakanathan]

Hello Sibson,

Any suggestion plz

[hilbix]

On my mobile there is no such thing as a cursor unless I attach a mouse to it via USB. Perhaps some developer option might help to visualize the current drag position.

Fun fact: I'd vote that there is a cursor shown, but it is the invisible one.

[samiraguiar]

I ended up facing the same problem as pevogam described. I've investigated it a little and here's what I think that happens:

TL;DR: the sleep() calls in the drag() method don't work as expected. What this method is actually doing is: "sleep a little, add a mouse move to operation to a buffer, sleep a little, add another mouse move operation to a buffer and repeat this a lot. After all those operations are added, send them all at once to the VNC server". Since the server processes those operations in sequence one after the other without a break, we don't see the cursor stepping. It happens too fast and the server might be even optimizing this and skipping the intermediary operations.

When we call mouseDrag(), it will issue a bunch of calls to mouseMove, which will eventually result in calls to transport.write().

The transport that the protocol is using is the TLS Transport, which delegates the write operation to FileDescriptor. This FileDescriptor will append the bytes to be written to an internal buffer, and will only actually send them when the reactor tells it to do so. This happens because when we call write, it will call self.startWriting() and register itself in the reactor to be notified when it's allowed to write. Adding a call to reactor.doPoll(timeout=5) after each mouseMove() seems to solve the problem.

The reactor is an EPollReactor and its addWriter method will eventually add the FileDescriptor to a dict, tying it to the underlying socket used to communicated with the VNC server. It will only notify the FileDescriptor in the doPoll method, which is the doIteration method of this reactor.

The doPool method, which is actually the doIteration method, is only called in the main loop after runUntilCurrent(). But when using the API and importing vncdotool as a library, our method calls are proxied and delegated to the reactor. This means that the first call to the mouseDrag() that I mentioned above will be appended to the thread queue of the reactor, and will be executed by its runUntilCurrent method. Since this method runs before the doPool, all those calls to mouseMove will happen before the socket is pooled, which is the trigger for the FileDescriptor to actually send them. They will be added to a buffer in sequence, and only after they are all executed and the mouseDrag() method is finished, the reactor will poll the socket and have the transport send them all. The VNC server will get all the mouse move operations at once and will process them in sequence, without an actual "sleep()" in between them. The server will process those operations too fast and we won't even notice the mouse moving. It can also be optimizing this and skipping all the mouse move operations but the last.

The only solution I could find is to add a call to reactor.doPoll(timeout=5) after each mouseMove() call in the mouseDrag() method. However, I'm not really familiar with twisted so I don't know if this has any unwanted effects nor if this is considered best practice when using this library (the method is not private, though). However, I imagine that this method works in some cases (otherwise it wouldn't have been added), so I'm wondering if something in the way we're using it could be causing it to misbehave.

[sibson]

@samiraguiar I think you're on the right track here. time.sleep() is the problem because it doesn't yield to the reactor. I wonder if simply replacing time.sleep() with reactor.doPoll() gets us close enough. I suspect there is a slightly more elegant solution but am happy with doPoll() if that works.

[pevogam]

Closing as this was solved, thanks @samiraguiar and @sibson!