Intelligent keyboard mapping

[XuYicong]

Depends on #83

Introduction

I removed the playcover setting mouse mapping, and implemented a better way to do all that setting meant to do. There is no more settings, just press option to hide/show cursor and other things are done automatically.

Background

If you're familiar with playcover you must know we have a setting named mouse mapping, which is on by default. It exists out of people's needs to use keyboard mapping and mouse cursor at the same time, although most people who do want this feature don't even know it's existence.

An important reason why it's not known is that it's hard to use. Once mouse mapping is disabled, people cannot even type text. People also want to switch between keyboard-only mapping and keyboard-mouse mapping on the fly, which mouse mapping cannot do since it requires relaunching to take effect.

Theory

When you press option, keymapping would switch between enabled and disabled states. Here it disables/enables both keyboard mapping and mouse mapping. When mouse mapping is disabled, option key does not work (intentional) and keymapping is in the state where keyboard is mapped and mouse not. There is theoretically another state, where keyboard is not mapped but mouse does. People want to switch between these four states in game, but they have only one option key, which only supports two states.

It is noticed that when people want their keyboard not mapped, it only happens when people want to type text. That makes sense, because apps on playcover are designed for mobile devices. On these devices, there is no physical keyboard, but when you tap on a text input area, a virtual keyboard appears. You don't see the virtual keyboard in other situations. So we can assert that if an app does not show virtual keyboard on a mobile device, in playcover we can safely map physical keyboard, and vice versa.

If this is possible, keymapping control would become simpler: in the two boolean variables of whether keyboard is mapped and whether mouse is mapped, keyboard can be controlled automatically based on whether a virtual keyboard should have shown. People only use option key to control whether mouse should be mapped.

Methodology

There are two pairs of notifications KeyboardWill/DidShowNotification and KeyboardWill/DidHideNotification, which are also posted even no real virtual keyboard is shown/hidden, but only a text input becomes active or inactive. By observing them, we can toggle keyboard mapping state accordingly.

Implementation

Previously, when keymapping is disabled, all mapped keys would be removed. When keymapping is enabled, they are read from the keymap file. This is unnecessary, since keymap would not change unless keymap editor is opened and closed. Moreover, it won't even work if keyboard and mouse mapping are controlled by two different mechanism, as in this PR. So I moved them to the editor open/close part.

This in turn affects how scroll wheel works, since their handlers are previously switched on keymapping enabled/disabled, but now they should base on whether mouse mapping is enabled/disabled. So instead of switching handlers, I warp them in a single closure that checks whether mouse mapping is enabled on the fly. This may add one more layer of invocation, but the performance overhead should be negligible.

Experiments

I tested on four games: 原神(Genshin Impact in China), 崩坏三(Honkai Impact 3rd in China), Ni No Kuni: Cross Worlds and 戰雙帕彌什(Punishing Gray Raven in HK, MO, TW) and one App: 小红书(literally translates to "little red book". A social media platform in China). It works on all.

However, it has been reported on some low performance devices that weird behaviours could sometimes happen, including mouse/keyboard buttons not correctly pressing/releasing. This should not happen if you're not using Macbook Air, but the root cause is still under investigation.

Conclusion

This should stop people asking questions like "how to use mouse and keyboard at the same time".

Reference

keyboardWillShowNotification

[XuYicong]

The target functionalities of this PR have been completed, but bugs are reported on M1 MacBook Air, which device I don't own. The last commit adds a logging feature, which would record all fake touches plus cursor toggles, and store them in /Users/USER/Library/Containers/BUNDLEID/Data/Documents/toucher.log(aka. Data/Document under game data folder).

If you're testing this branch, there is a put a mark in log item in the keymapping menu, which is bond to CMD+L. Whenever you encounter a bug, press CMD+L. To report bugs, open an issue in my repo, send the log file with your bug report, so that the issue can be analyzed.

[XuYicong]

In the last commit, I have added support for fixed and free joystick, but haven't add an option for it. For now, if the joystick size is big, it would switch to free mode. An option should be added later.

Also, the draggable button now is available when cursor is shown. It would hide the cursor when it functions. And now the cursor would not hide if there are no keymapping for mouse moving (except draggable button) or left mouse button, even when option is pressed. This is useful for people who use alternative control, where option is used to map touchpad to touchscreen.

And when cursor is shown, right mouse button and middle mouse button can now act as mapped. And the traffic light button can now light up as expected when cursor hovers them.

Created new classes PlayController and PlayKeyboard to better organize code. The initialization process is now more clear.

Created a seperate debug menu to place debugging related commands. That menu as well as the toucher log can be toggled by Toucher.logEnabled.

[SH1SH1]

Is there any way to completely unmap the keyboard? Because there is a mobile game called Nishuihan, which is adapted to keyboard and mouse, and there is mapping that makes it inoperable. Or is there a way to realize that the keyboard input is w and the game receives w.

[XuYicong]

@SH1SH1 you can disable keymapping in playcover's per-app settings.