System keys for android

[famus484]

Is your feature request related to a problem? Please describe.
I can't fully control host using Moonlight on Android. So if game freezes and I am far from home, I can't troubleshoot until home. System keys, like on PC clients, would solve this problem (for example, Ctrl+Alt+Del -> Task manager).

Describe the solution you'd like
When we use 3 finger tap for keyboard, make Ctrl, Super (Windows), Alt and Tab appear just above.

Describe alternatives you've considered
Using shortcuts mapped from the controller, but it is not that practical, as about all controller inputs are used in games. Maybe a sequence of shorcuts?

Additional context
Parsec has the functionality, I think. Maybe we can use the same design?

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

[fgielow]

I would like to be able to use a bluetooth keyboard on moonlight-android capturing all modifier keys. Currently if I press windows key for instance it will open android's search menu. Alt+Tab also will not work, though some shortcuts such as CTRL+L do work.

It'd be great if all keyboard keys were captured so I can use moonlight-android to fully control my remote host.

Is this possible somehow already?

[cgutman]

I don't think system key capture is currently possible on Android. The system combos always override applications.

We would need a new API for that in a future version of Android to request capture of all keyboard modifiers.

[famus484]

Thank you for the answer. The impossibility is for physical keyboard system key capture, but not sending system commands to host, right? Bluetooth does seem impossible, but wouldn't it be possible to capture system keys for USB devices by offering to capture the whole USB keyboard+mouse package?

From what I understand, Moonlight already does this for xinput controllers when "Override native Xbox gamepad support" is enabled. If it's easy to implement (that I do not know, do all usb keyboards behave the same?), it would allow much better host control from a phone/AndroidTV. As a bonus, by capturing the mouse, wouldn't Moonlight solve any mouse acceleration problem that linger still?

The alternative is VirtualHere, but if a connection problem arises while in game it's much harder to reestablish than Moonlight, and doesn't work as well outside home network.

But that would be another (albeit very awesome if possible and not too much work) request from this thread, where the system keys would be on the touch screen above keyboard when 3 finger tap.

[cgutman]

Thank you for the answer. The impossibility is for physical keyboard system key capture, but not sending system commands to host, right? Bluetooth does seem impossible, but wouldn't it be possible to capture system keys for USB devices by offering to capture the whole USB keyboard+mouse package?

Unfortunately not. USB HID devices are explicitly banned from the standard Android USB driver support in order to prevent key loggers.

But that would be another (albeit very awesome if possible and not too much work) request from this thread, where the system keys would be on the touch screen above keyboard when 3 finger tap.

Yeah, it might be doable now. The problem is that IMEs on Android vary so much and there's not always a reliable way to determine whether one is displayed (and some IMEs don't have any UI at all).

[famus484]

Unfortunately not. USB HID devices are explicitly banned from the standard Android USB driver support in order to prevent key loggers.

Ohhh, that's too bad. I am wondering what magic does VirtualHere does to allow for something like that. It captures the USB keyboard as long as I answer yes when Android asks me if I want to (with system keys working as intended). Perhaps it doesn't have to use the same kind of driver Moonlight would have to?

Yeah, it might be doable now. The problem is that IMEs on Android vary so much and there's not always a reliable way to determine whether one is displayed (and some IMEs don't have any UI at all).

Oh, I see! That would mean you would have to implement as many different solutions as there are IMEs, which would almost be as many phone manufacturers? If that's the case, could a special IME explicitely for Moonlight stuff work?

[cgutman]

Ohhh, that's too bad. I am wondering what magic does VirtualHere does to allow for something like that. It captures the USB keyboard as long as I answer yes when Android asks me if I want to (with system keys working as intended). Perhaps it doesn't have to use the same kind of driver Moonlight would have to?

Ah, it may have been changed then. It's certainly an interesting idea, but it's definitely not a trivial thing to implement.

[famus484]

Yes, I suppose not. Was implementing the native xinput drivers hard? Like, did you have to support all controllers one by one, or do all xinput controllers behave almost identical to one another? Could you use already written drivers that did a better job than native android (ex, from linux)?

I am trying to assess how hard it might be to implement such a thing. Is there a large class of standard keyboards and mice that are all expected to behave the same from kernel's POV? I think the idea of capturing devices could be very beneficial, but I guess that goes way out of the scope of this request...

[sivan-koren]

If the idea is just to send the Win key or other modifiers, wouldn't it be possible to translate some other key press that is available in the API? For example, could you capture a double press Ctrl, and send the host a Win key press? Double press and hold Ctrl, could send a Win key on signal. Etc.

[sivan-koren]

Also, at least on Google TV OS, Button Remapper (app on Play Store) allows the capture of the Left_Meta key on a USB keyboard (win key) by using Android's Accessibility controls/permissions. In this case it does succeed in overriding the Android system behavior for this key (i.e. Google Search does not launch when the key is pressed, optionally even overriding the key press in other applications). It does not have any option to pass that key press to Moonlight obviously, but in theory, Moonlight could use the same accessibility permissions to intercept the Meta Left key and send it to the host.

[cgutman]

I don't think using accessibility APIs would be a good idea. Unlike Button Remapper, it would not constitute "core functionality" of the app, which makes it a shakier case for justifying the use of accessibility APIs. It also requires extensive justification, in-app notification, and manual review by Google. https://support.google.com/googleplay/android-developer/answer/10964491

Google has a long history of cracking down on these APIs, and I'd hate to end up with the app stuck in review, suspended, or removed over some API I used for capturing Meta keys. It's just not worth it.

[xor-128]

I don't think using accessibility APIs would be a good idea. Unlike Button Remapper, it would not constitute "core functionality" of the app, which makes it a shakier case for justifying the use of accessibility APIs. It also requires extensive justification, in-app notification, and manual review by Google. https://support.google.com/googleplay/android-developer/answer/10964491

Google has a long history of cracking down on these APIs, and I'd hate to end up with the app stuck in review, suspended, or removed over some API I used for capturing Meta keys. It's just not worth it.

It's doesn't need to published on Google Play, making a build setting would be enough, Implementing accessiblity API is easy (which i made it under 1 hour), the main issue with this is Accessibility setting is not available to sideloaded apps in Android 13. It's not a issue to me I using my phone is rooted but the normal users probably gonna have a issue with that.