LiSendMousePositionEvent with multiple host monitors

[tomblind]

I started playing with this for touchscreen support on moonlight-embedded, but have run into some odd behavior when the host computer has multiple monitors. Instead of mapping the mouse position from the reference size to the monitor being streamed, the coordinate is being mapped to the entire area covered by all screens.

For example, if I have two screens (same resolution) side by side and I send a coordinate in the middle of the client's screen (referenceWidth / 2), the mouse cursor will be placed on the right edge of the host screen being streamed. However, moving past that point does not actually put the mouse cursor on the other monitor. This leads me to suspect it's not intentional behavior.

[cgutman]

Thanks for reporting. I was able to reproduce the issue locally.

However, it looks like it will need a fix in GeForce Experience. It also affects the official client on the Nvidia Shield, so I'm inclined to believe it's not just something I'm missing in the protocol.

It looks like GFE is incorrectly projecting the client's coordinates on the virtual desktop space. It doesn't look like they're taking into account the monitor's position on the virtual desktop. The behavior I'm seeing makes it look like they're passing MOUSEEVENTF_VIRTUALDESK to SendInput() and scaling to the virtual desktop size rather than to the correct monitor rect for the primary display.

I'm not sure if they're actually using SendInput() or sending these from one of their virtual USB devices, but if they are, all they have to do is stop passing MOUSEEVENTF_VIRTUALDESK and it appears SendInput() will just do the right thing.

If MOUSEEVENTF_ABSOLUTE value is specified, dx and dy contain normalized absolute coordinates between 0 and 65,535. The event procedure maps these coordinates onto the display surface. Coordinate (0,0) maps onto the upper-left corner of the display surface; coordinate (65535,65535) maps onto the lower-right corner. In a multimonitor system, the coordinates map to the primary monitor.
If MOUSEEVENTF_VIRTUALDESK is specified, the coordinates map to the entire virtual desktop.

from https://docs.microsoft.com/en-us/windows/win32/api/winuser/ns-winuser-mouseinput

[cgutman]

OK, I attached a debugger to nvstreamer.exe and didn't see any calls hitting my User32!NtUserSendInput breakpoint. It makes sense, because they stopped using SendInput many releases ago for relative mouse motion in favor of their virtual USB device approach for compatibility with AppContainer (UWP) apps.

There are 2 HID-compliant mouse grandchild devices under the NVVHCI Enumerator in Device Manager. The first is the relative pointing device used by LiSendMouseMoveEvent(). The second is the absolute pointing device used by LiSendMousePositionEvent(). Disabling that second device breaks absolute pointer input, so they are definitely using their VHCI mechanism for absolute input too.

None of this is really relevant to their bug, but it was fun and worth documenting nonetheless.

[tomblind]

Thanks for looking into it. I was worried it was an nvida-side bug, but I don't have an official shield to test with to be sure. Also, thanks for the thorough explanation. It's interesting to hear how gamestream works under the hood. 😃

[mariotaku]

GeForce Experience 3.22.0.32 may have fixed this issue. Mouse pointer behaves correctly on moonlight-tv.

[cgutman]

Yep, can confirm this locally. Fixed in GFE 3.22 and RTX Experience 1.3.