Move mouse to any point on screen with WinAppDriver

[khoachung89]

I have automation test that run on client and WinAppDriver with my tested app on server.
With the session created from capability 'app' = 'root'
var option = new AppiumOptions();
option.AddAdditionalCapability("app", "Root");
var session = new WinAppDriverSession<T>(new Uri(serviceUrl), option);

Now i want to move the mouse to a point on screen - even that point is out of my app's window. Up to now, i have to do 2 steps:

1. move mouse to top left of screen with -int.MaxValue - looks ugly:
   Actions a = new Actions(session);
a.MoveByOffset(-int.MaxValue, -int.MaxValue);
a.Perform();

2. move mouse to the desire point:
   a = new Actions(session);
a.MoveByOffset(myPoint.X, myPoint.Y);
a.Perform();

Because at the client, i have no idea how to get remote screen resolution or current mouse position. I cannot move the mouse to desire point with one action. Is there any better solution to get the same result? In my real scenario when i have thousands of automation test cases, the mouse move to top left of screen lot of times and looks silly.

Many thanks,

[DLightstone]

If it is a windows form calibrate the screen.
There is a window cloise button and a window minimize button in the upper right corner.

My guess is that you can get the scaling information that you need from their positions

[khoachung89]

@DLightstone my app is WPF one. Of course, it has the title bar with default buttons (minimize/maximize & close buttons).

However, i don't really catch your idea with getting scaling information from their positions.
I can get the the close button and its BoudingRectangle (the most related attribute i can see from Inspect) by something like:
closeButton.GetAttribute("BoudingRectangle")
With this information, i know the position of the Close button (maximize & minimize button also). How can i get the scaling information now? You mean we can get the scale between the size of window with screen size?

[DLightstone]

I do not know the reference for the position and size information provided by attribute retrieval. I have assumed them to be actual screen coordinates. I assumed move offset to be in actual screen coordinates rather than relative to the current application window.

I assumed that the positioning was to move was for purposes of either activating an application control located at that position or editing a pixel at that location. It became a question of - To position to a particular application pixel what absolute screen pixel is needed? That is a scaling problem.

I just wanted to get the locations of 2 application points (minimize button and maximize button were the first points I thought about. I completely forgot you could get the window size)

At start up of the program under test, maximize the screen (one time event), get the display screen size. It may be possible to get the display size via other means.

At any other time the then current application window size.

The 2 measurements give you the scaling info for the application window relative to the display screen.

[khoachung89]

I see what you mean. So you will use the screen at maximum size to measure the screen. It's a good idea because it least, i have to do it one time only.

The other point is the method MoveByOffset doesn't mean move the mouse to a point. It means move the mouse to the specified offset of the last known mouse coordinates. So if i don't know the last known mouse coordinates, i cannot move it my desire point. That's why in my question, i have to move the mouse to the top left of the screen :D It's a way to reset the mouse position.

[DLightstone]

There probably is a better means for measuring the screen. A user can change the resolution of the screen, so the selected resolution must be saved in the registry somewhere. That is probably what you will retrieve when you measure the screen by maximizing it.

The mouse is a device, it generates events. There probably is a mechanism by which you can subscribe to receive those events. How that mechanism might work I do not know, but I would be very much surprised if one does not exist.

see
https://docs.microsoft.com/en-us/windows/win32/inputdev/raw-input

[khoachung89]

I know we can get the mouse. The issue is how we get it from Appium. My scenario is:
My test case is running on client machine. It send request to a WinAppDriver server.
WinAppDriver server is running my tested app. WinAppDriver server receive request from my client machine and simulate the mouse move.
Now, from client, i want to send a request to get back the location of the mouse on WinAppDriver server, not the mouse on my client machine..

[DLightstone]

Generally I try to do as much testing as possible on a local machine.
If the applictaion under test can be installed locally things are much easier.

Don't get the mouse position from the application under test.
Have a second server application on the server host. One that just returns the mouse location when requested to do so.

Using Appium with the second application may just be a convenience (no need to make the second application a server). The application should have a form with text boxes for the mouse information that you want. Appium can retrieve from those text boxes

If you can't have such an application installed then you are testing on a production server, and should not be doing so.

[HenryColes]

Hi @khoachung89
Could you tell me how did you reset the mouse position before using MoveByOffset() method?

[Shakevg]

Hi @HenryColes

I used next c# code to click by coordinates with WinappDriver:

var element = session.FindElement(By...); 
new Actions(session).MoveToElement(element, 0, 0).MoveByOffset(dx, dy).Click().Build().Perform();

To reset the mouse coordinates use this:
new Actions(session).MoveToElement(element, 0, 0)