Unable to use keyboard on Unreal Engine game

[aneutron]

Hey !

First of all thanks for the amazing work. I noticed the keyboard is usable in a normal manner, and it works for apps such as Messenger, etc., and it works with Hearthstone too. However, it does not work with PUBG Mobile.

I just noticed this, and thought that maybe it's worth reporting, because on the one hand it could be something with Unreal Engine, but it could also be a bug in scrcpy.

Anyway, you're doing amazing work, thanks a lot !

[rom1v]

However, it does not work with PUBG Mobile.

I just installed the game on some device, it works for me. I can type text for example in the "team chat".

[aneutron]

Yes but the controls don't work. I've seen people using OTG keyboard and mice on YouTube to play.

[pete1414]

Any progress on this? Is there a way for ScrCpy to say an external OTG keyboard is connected? Some Apps don't recognize the keyboard because it's looking for an OTG or bluetooth keyboard.

[rom1v]

Is there a way for ScrCpy to say an external OTG keyboard is connected?

One possibility would be using HID over AOA to generate HID events.

I already implemented HID events forwarding (i.e. forward raw events from an USB keyboard or mouse) in the past (here is a sample project), but generating the HID events from SDL events requires some work (including reading the USB HID spec).

And like audio support in scrcpy, this would only work on Linux (which is still great).

This could also be interesting as a separate project: this would make the computer keyboard act like an OTG keyboard.

[pete1414]

Please see screenshot below. Top half is key events from Scrcpy, bottom half is keyboard events from bluetooth keyboard. In both situations I held down the 'w' key'. It seems Scrcpy sends Key up before it is released.

[pete1414]

I have a work around for this, not fully sure of the implications but it at least works for me in PUBG.
I am not too familiar with github, so I'll just explain what I did.
in scrcpy.c
I removed SDL_TEXTINPUT from the case statement.
I copied the call in KEYUP to KEYDOWN.
Then in Convert.c I added more key mappings to include alpha keys and number keys and a few others..

It now works in PUBG, I can also still type normally and paste text.. things that were handled in text input will not work like volume up etc, they could easily be moved to the key handling event, but I haven't bothered.

I have not thoroughly tested this.. really just wanted it to work for games and now it does.. no longer have to connect a bluetooth keyboard.

This applies to #122 and #127 as well I believe.

[rom1v]

Cool 👍

I am not too familiar with github, so I'll just explain what I did.

You should click on "fork", then push your branch to your github fork and post the link here :)

Then in Convert.c I added more key mappings to include alpha keys and number keys and a few others..

What is your keyboard layout? What if you press Shift+, (for example, on a French keyboard)?

I initially implemented key events using keycodes only (before scrcpy was published), but after reading documentation and discussions on irc #sdl, I think there is no way to work independently of your keyboard layout. Maybe I am wrong though.

things that were handled in text input will not work like volume up etc, they could easily be moved to the key handling event, but I haven't bothered.

On dev branch, volume shortcuts have been changed to Ctrl+up and Ctrl+down without text input.

[pete1414]

I don't see a mapping for "<" in the android keycodes.h, which is what my Shift + "," would be. But I think as long as a mapping is created you could use any layout.

Also, thanks to you for creating this! The app is amazing and I can't say that enough. I am an original kickstarter backer for the 'superbook' and this might replace my need for it... all though I am still looking forward to it.

[rom1v]

But I think as long as a mapping is created you could use any layout.

I agree, but it may not work with all layout without remapping (while with text input events it should work).

You said:

Would it be easier to have the user switch between modes? ie, press Ctrl G to enter 'game' mode. Press again to use text mode.

This is a possibility. 👍

[rom1v]

I added a command-line option: (see next comment)

scrcpy --raw-key-events

or:

scrcpy -k

Please try the rawkeyevents branch. If it works as expected, I will merge it into dev.

press Ctrl G to enter 'game' mode

I decided to use a command-line option, because there is no clear feedback about the current mode, which may be confusing if someone enables it without noticing.

EDIT: not sure I will keep it that way. Maybe I will remove the option and bind it to a shortcut. What do you think?

[rom1v]

I changed my mind, I remove the command-line argument, and bound Ctrl+k instead (the mode may need to be toggled while scrcpy is running).

[rom1v]

@aneutron @pete1414 Could you test the branch rawkeyevents and tell me if it fixes/workarounds the problem, please?

[pete1414]

I downloaded the rawkeyevents branch and compiled it in 32 bit and it works with Octopus + PUBG with no issues. I am not near a 64 bit machine at the moment but would not anticipate any issues.

Thank you!

[pete1414]

I should also add my feed back re Ctrl K. For me it does not make a difference because raw key mode seems to work for me for everything so I would just leave it in that mode. But for someone who needs to go back and forth I imagine they would prefer the shortcut.

[rom1v]

Thank you for your feedbacks 👍

For me it does not make a difference because raw key mode seems to work for me for everything

Even for ?!;:'"*? Probably because you use an English-QWERTY keyboard.

If it makes no difference for such layouts, I could add a -k parameter in addition to the shortcut.

[King-Slide]

The rawkeyevents mode is basically what I need. Only problem I have with it, is that I'm using a german keyboard and every key that does not have a mapping in convert_text_keycode is not handled at all.

My personal solution up until now was to make an exception for wasd and handle it like the new rawkeyevents mode. You can see what I did here:
https://github.com/King-Slide/scrcpy/commits/l2r

Would a hybrid mode that handles non-mapped keys as textinputs be possible?

[rom1v]

@King-Slide Thank you for your feedbacks.

Only problem I have with it, is that I'm using a german keyboard and every key that does not have a mapping in convert_text_keycode is not handled at all.

That's expected, that's why this option is disabled by default.

My personal solution up until now was to make an exception for wasd and handle it like the new rawkeyevents mode.

In King-Slide@465b915, if input_manager_process_text_input() returns false, you don't break and fall-through, but then the SDL_Event, with the active union member text, is accessed via the inactive key field, which is undefined behavior.

Would a hybrid mode that handles non-mapped keys as textinputs be possible?

I think it is not possible to have 1 mode to cover all cases. That's why I think exposing 2 modes, that can be toggled, would be a good compromise. What do you think?

[King-Slide]

Sorry for the return values. They were some remnants of a failed experiment.

Switching modes constantly is just frustrating, cause I forget that from time to time.

I created a branch from the current raw_key_events branch, introduced a toggleable hybrid_key_events mode, which is just a raw_key_events mode for a-z 0-9 and the space key:
King-Slide@26e2b8d

[rom1v]

@King-Slide Interesting. I didn't tested, but the hybrid could also include A-Z, right? (in a game, pressing shift+a should generate raw events)

[King-Slide]

@rom1v Here are the changes: King-Slide@d09babc

• hybrid mode is enabled by default
• uppercase letters are handled as raw_key_events in hybrid mode
• shift + numbers are handled as textinputs

Enabling raw_key_events or hybrid_key_events disables the other. It seems to work with my german keyboard layout as far as I can tell.

[rom1v]

@King-Slide Great. I didn't test (yet), but if this work both for typing text and for games, I think this could be the default and only mode (to avoid adding unnecessary options and keep it as simple as possible).

Note: you could use isalnum() to detect whether it's a letter or a digit.

[rom1v]

@King-Slide num chars (0 to 9) are not "portable" (e.g. the shift state depends on the layout).

As a compromise, I suggest to send letters and space as key events (unless ALT or META is pressed), and keep the remaining as text input event. (commit rawalpha)

This would be the only mode (=> simplicity) if it does not cause problems.

What do you think?

@aneutron @pete1414 Does it work in your games?

[rom1v]

Any feedbacks?

[King-Slide]

As far as I understand SDL a keypress triggers a textinput and a keyevent afterwards.

By using isalnum on the textinput, I can differentiate between a plain number input and a shift + number
input. If isalnum returns false, which it should on shift + any number key, I let it get handled like textinput, else I break out of input_manager_process_text_input.

To prevent double outputs I break out of input_manager_process_key when I detect shift + a number.

This way I don't have any double inputs, my number keys are handled raw and my current localization works because it is handled by input_manager_process_text_input.

You can see what I did here:
King-Slide@26a2179

[shashi007]

@rom1v Ah, Mouse feature is not implemented yet, I hope it will be soon.. Thanks. I love Scrcpy. :-)

[rom1v]

Yes, mouse forwarding is implemented (you can click on the screen), but not in capture mode for gaming.

[IAmMadMax]

Was looking for a solution to this and ended up here. Is there a scope for this "capture mode" to be added in future builds? Would really open gates for some good gaming experience. Coincidentally I was looking for ways to play PUBG mobile too, just like the others above.

Whether or not you will, you are doing great work. Cheers!

[rom1v]

Is there a scope for this "capture mode" to be added in future builds?

I did not investigate how to implement it yet (#269). If someone have time and is interested, please go ahead 😉 Otherwise I might look at it some day.

[walkiitm]

@walkiitm I just built a win64 version of master+rawalpha.

Could you test it, please?

Just download scrcpy.exe and replace it in your scrcpy directory.
SHA-256: 7a027131e5669dfb01e34cc75f080728ef5434cd3f21f0b3e8cc89ac30fe5887

Hey, both my Windows defender and Anti-virus warned me of running this file. It didn't happen with the original build.

[rom1v]

Hey, both my Windows defender and Anti-virus warned me of running this file.

What do they say?
Could you check its SHA-256 to verify that your file is exactly the same?

I downloaded it, its SHA-256 is the expected one, and if I scan it online with virustotal, everything is fine:
https://www.virustotal.com/fr/file/7a027131e5669dfb01e34cc75f080728ef5434cd3f21f0b3e8cc89ac30fe5887/analysis/1539415214/

It didn't happen with the original build.

Maybe the reason is that they detected that the original file has been replaced (same filepath, different content)?

By the way, you don't need this temporary file anymore, the feature is included in v1.4.

[walkiitm]

Hey, both my Windows defender and Anti-virus warned me of running this file.

What do they say?
Could you check its SHA-256 to verify that your file is exactly the same?

I downloaded it, its SHA-256 is the expected one, and if I scan it online with virustotal, everything is fine:
https://www.virustotal.com/fr/file/7a027131e5669dfb01e34cc75f080728ef5434cd3f21f0b3e8cc89ac30fe5887/analysis/1539415214/

It didn't happen with the original build.

Maybe the reason is that they detected that the original file has been replaced (same filepath, different content)?

By the way, you don't need this temporary file anymore, the feature is included in v1.4.

Okay. Thank you. Yeah must be a false positive. The antivirus uploaded the file to its servers(apparently cause it's an exe that's rare) and later reported that it's safe. Sorry for the confusion.

[Mehardeep]

Hi everyone please help me to find where pubg controller are implemented in the code
as A-W-S_D buttons are working in PUBG (scrcpy) but when i sent through cmd (adb input keyevent ) its not working

[rom1v]

@Mehardeep See 411aa4f and injectKeyEvent() on the server.