Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Virtual/screen keyboard support #80

Open
Beuc opened this issue May 12, 2019 · 11 comments
Open

Virtual/screen keyboard support #80

Beuc opened this issue May 12, 2019 · 11 comments

Comments

@Beuc
Copy link
Contributor

@Beuc Beuc commented May 12, 2019

Hi,

AFAICS there is no on-screen keyboard support in the Emscripten SDL2 port, which is a problem on mobile platforms.

There doesn't seem to be a browser API to control the virtual keyboard (or maybe I missed it?), so we could:

  • re-implement a basic keyboard (directly in SDL, or in HTML overlay - if HTML it needs to be included in the full-screen mode DOM element) or
  • trick the browser into displaying the virtual keyboard by focusing on a <input type="text"> element somehow
  • other ideas?
@Beuc
Copy link
Contributor Author

@Beuc Beuc commented Jul 26, 2019

(clarified a mistake about fullscreen)

Loading

@Beuc
Copy link
Contributor Author

@Beuc Beuc commented Jun 8, 2020

Several "javascript virtual keyboards" exist, notably https://github.com/hodgef/simple-keyboard which has no dependencies (though none of them describe why they were developed in the first place).

I'm not sure what stack should provide the virtual keyboard. I'm wondering what's the SDL team's stance on shipping additional javascript+css code in SDL2 to handle an IME?

Loading

@Beuc
Copy link
Contributor Author

@Beuc Beuc commented Jun 20, 2020

Alternatively, I made a test with an off-screen HTML text input, to leverage the browser's built-in support. This properly show/hides the virtual keyboard on focus/unfocus, and JS events allowed tracking what was typed. Keycodes are caught and forwarded to the application as well, for consistency with SDL2/Android.

https://www.beuc.net/tmp/ti1.html

Limitations:

  • Browsers seem to only pass invalid 229 keycodes in keydown/keyup events, except for Enter, or they just don't generate the keyboard event. This notably prevents the SDL2 app from supporting text deletion (backspace key). This looks like a general issue with virtual keyboards in Android browsers https://bugs.chromium.org/p/chromium/issues/detail?id=118639

  • The handling of insertText (~SDL_TEXTINPUT) vs. insertCompositionText (~SDL_TEXTEDITING) events looks tricky. In particular, the browser doesn't generate a final insertText (unlike SDL2), may generate events for past words if the user uses Backspace. If I delete the text as the user types, Firefox issues duplicate insertCompositionText events. In short, browsers don't seem to map well with SDL2.

  • There's no HTML5 API to detect whether there's a physical keyboard. We'd probably need to have this always enabled, or follow an heuristic on browser agent. It works fine on desktop though.

With these limitations it sounds like this isn't the right approach, but I'd be happy to discuss this further.
ti1.zip

Loading

@Beuc
Copy link
Contributor Author

@Beuc Beuc commented Jun 20, 2020

Also here's a sample SDL2 app for comparing the behavior between desktop/web/android:
https://github.com/renpy/renpyweb/blob/master/tests/testtextinput.c

Loading

@Beuc
Copy link
Contributor Author

@Beuc Beuc commented Jun 27, 2020

Referencing emscripten-discuss thread for info:
https://groups.google.com/forum/#!topic/emscripten-discuss/FN-8Tdxi1D8

Also given there was 0 reaction here after 1 year, it is unlikely that any solution will be implemented at the SDL2 layer :/

Loading

@feliwir
Copy link
Contributor

@feliwir feliwir commented Sep 10, 2020

@Beuc I'd find a solution for that problem useful aswell

Loading

@Beuc
Copy link
Contributor Author

@Beuc Beuc commented Sep 10, 2020

@feliwir In the end we had to re-implement a keyboard in-game:
renpy/renpy#2336
Browsers' behavior was too inconsistent, or maybe I just missed a better solution.

Loading

@ghost
Copy link

@ghost ghost commented Nov 22, 2020

I too could use this functionality. The off-screen text field thing is the type of workaround that would be consistent with Emscripten's approach. Myself, I'll try superimposing a visible text field first. A special soft keyboard like you're apparently doing for Ren'Py would be a bit heavy and non-native, but for a VN it's probably not a problem.

Loading

@ghost
Copy link

@ghost ghost commented Dec 3, 2020

HTML form superimposed over the canvas activates the soft keyboard but does not accept input (input being received by SDL with preventDefault). This means the form submit event is not recognized if you press the virtual enter key. Instead, there seems to be some weird keyboard issue where Enter isn't received at all or differs from the US QWERTY key code.

Any ideas?

EDIT: The Return scancode is the same on Android's soft keyboard (the "Go" button) as on a physical keyboard (the "Enter" key). Even though Emscripten input events are blocking, user code can detect Return and its ilk via soft keyboard, as long as other issues are resolved like lag.

Loading

@uyjulian
Copy link

@uyjulian uyjulian commented Dec 19, 2020

Loading

@ghost
Copy link

@ghost ghost commented Dec 19, 2020

Loading

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
3 participants