API for custom input event injection, to support additional input devices (multiple mice, spinners, trackballs, VR controllers, pen digitizers...)

[davthedev]

Describe the project you are working on:

A multiplayer pong / brick-breaker. Those games are best played with so-called spinners, which are infinite rotary controllers.
Those spinners are often plugged to the computer through an HID board, and seen as a one-axis mouse by the system.

For multiplayer capability, I need multiple mice support on the same machine, with distinction between each individual mouse / pointing device.

The current challenge in the existing situation is the following:
You can actually plug multiple mice on the same computer, but the OS will only render one mouse pointer. Its movement will be the average of all mice combined together.

Describe how this feature / enhancement will help your project:

Each individual mouse connected to the computer would be individually readable : movement, button presses, scroll wheel ticks. It would be possible to list available mice, monitor newly plugged mice and unplugged ones, like it is done with the joypads.

This possibility makes the following multiplayer game controls possible:

• Pong / two-player Arkanoid-like in versus mode. Each mouse moves one paddle.
• Old-school rail shooters (think House of the Dead) where each player moves its own crosshair with its assigned mouse
• Old-school multiplayer car racing games, where the steering wheel on the arcade cabinet was nothing else than a spinner with a steering-wheel-shaped knob.
• Trackball games like Segasonic (two-player game, each player has a trackball, the player has to roll the trackball as fast as possible to make the character run in the game)
• Multiplayer marble games
• Collaborative construction games. One notable example is World of Goo, where the multiple mice library mentioned below was actually used

Show a mock up screenshots/video or a flow diagram explaining how your proposal will work:

Describe implementation detail for your proposal (in code), if possible:

There is an existing library for that, which is lightweight and abstracts the low-level mice reading on major operating systems : ManyMouse. It is very minimal in its design, which should make it suitable for not bloating Godot. We appreciate how light Godot is and should remain so.
See here: https://icculus.org/manymouse/

I successfully integrated the library on a custom Godot build (older version) as a C++ module. My implementation does the following:

• Part of the ManyMouse API is exposed as GDScript bindings (not all API as it is a proof-of-concept, only the most essential one for now)
• Some custom GDScript code in the project receives the events, bundles the mouse movement events into less frequent, added-up, and propagates them as signals
• Game objects / kinematic bodies that need to move with mice inputs receive the signals

This implementation works, but has some flaws.
It bloats the signals pipeline. I run it on a Predator Helios 300 (powerful gaming laptop). As long as the mains is connected, everything runs fine. When I switch to battery, the computer may enter a power saving mode and all the game starts stuttering, certainly due to the signals overloading.

In addition, it makes redundant mice management as the normal Godot mouse event API is still around. Thus, we receive the mouse clicks in double, one in the custom multi mouse layer then one in the Godot mouse event API.

If this enhancement will not be used often, can it be worked around with a few lines of script?:

No, since it is the integration of a C++ library that adds new input capabilities.

Is there a reason why this should be core and not an add-on in the asset library?:

I developed the concept as a C++ module. On my situation, it works but has the flaws listed above.
Especially, I have not found yet how to inject custom events into the input event processing pipeline instead of using signals. This would be much more efficient.

Implementing the feature as an add-on would make sense, maybe. But, is it technically feasible today?

[vnen]

While I can see this feature is very important for what you are doing, I believe it won't be used by many other people. It might be useful for installations in museums or events, but the vast majority of players won't be connecting many mice to their computer just for a game.

One thing that could be done (if needed) is providing some bindings in the GDNative API so you can extend the input with a custom module that doesn't need to be built in the engine itself (so you can reuse and distribute without maintaining a fork).

Especially, I have not found yet how to inject custom events into the input event processing pipeline instead of using signals.

For this can't you just create a custom InputEvent and send via the Input singleton: Input.parse_input_event(my_event)?

[Janders1800]

I don't think its possible, mouse control is handled by the OS, and currently there is no OS that supports more than one mouse at the time.

[davthedev]

I don't think its possible, mouse control is handled by the OS, and currently there is no OS that supports more than one mouse at the time.

It is! I have a working prototype. The difference is that mouse events must be read at a lower level from a different module than usual. This is exactly what ManyMouse library does. Windows since XP, and other operating systems can handle it; up to 32 different mice are supported at the same time in most cases but this is really extreme :)
Another thing to note when reading at this level is that you get the raw movement, before the OS applies acceleration settings. It can be useful in some cases.

One thing that could be done (if needed) is providing some bindings in the GDNative API so you can extend the input with a custom module that doesn't need to be built in the engine itself (so you can reuse and distribute without maintaining a fork).

Exactly! If the feature is too niche to be worth an addition to the core, some API hooks to be able to do the same from an add-on would be the solution.

[beniwtv]

I too think this is too niche to be in the core part of Godot. A GDNative module would probably be better.

In addition, extending GDNative to be able to send input events to Godot's input subsystem (if not possible already) might even be helpful for us in the Godot OpenVR module in case we want to handle Steam VR input events one day.

[nathanfranke]

Why not add an integer (or related) to the Input class? If there is some native method getting these controls, we should try to attach a "controller number". As a fallback, just use the value '-1' etc.

I think it would be a good feature to include distinctive input for all devices including multiple keyboards, mice, USB controllers, etc.

[davthedev]

Why not add an integer (or related) to the Input class? If there is some native method getting these controls, we should try to attach a "controller number". As a fallback, just use the value '-1' etc.

I think it would be a good feature to include distinctive input for all devices including multiple keyboards, mice, USB controllers, etc.

Actually, the way mouse input is handled now is very different than the way it would be if you want separate mice data. You do not get the information from the operating system at the same level.
Especially, there are global pointer coordinates on screen beyond the sole game window. This is how you can click on objects without having the game capturing the mouse movement.

In comparison, the multiple mice information will give you deltas rather than absolute coordinates. Handling is necessarily done using a captured mouse approach, even if it requires simulating an internal mouse cursor.

The current way of handling the single OS-provided mouse cursor must remain, as it is the basis for interaction that most people expect. The multiple mice system should be a different class of events.

Multiple keyboards? This is a different story. I would say why not; it would be a niche usage too, at least allowing us to make a multiplayer game similar to The Typing Of The Dead.

[davthedev]

One thing that could be done (if needed) is providing some bindings in the GDNative API so you can extend the input with a custom module that doesn't need to be built in the engine itself (so you can reuse and distribute without maintaining a fork).

In addition, extending GDNative to be able to send input events to Godot's input subsystem (if not possible already) might even be helpful for us in the Godot OpenVR module in case we want to handle Steam VR input events one day.

I think we got it now. The real need is that API possibility to send the custom events from a GDNative module.
Note to admins: should we close this and open a new ticket for the updated feature request?

[aaronfranke]

@davthedev Sorry for the long silence. I think the best approach here is to improve the API to allow you to overcome the limitations you ran into when implementing this via GDNative, since this seems like something that should be doable with GDNative. If you want, you can just update the OP with the API changes that would be needed.

[davthedev]

Here are some news!

So far, I got a successful implementation as a GDNative module. It works exactly as before using signals to dispatch mouse events from one specific pointing device.
The missing part is how to inject custom events to the current pipeline.
Also struggling on getting things right regarding event polling. Because, in the current state, I rely on the normal game loop "process" call to do the polling (actually dumping queued events from the library), which loses the realtime status of built-in input events. This also causes a 2-frame delay in arrival, which results in a lag.

@aaronfranke I cannot edit the original post, event if I am the author

[Calinou]

So far, I got a successful implementation as a GDNative module. It works exactly as before using signals to dispatch mouse events from one specific pointing device.
The missing part is how to inject custom events to the current pipeline.
Also struggling on getting things right regarding event polling. Because, in the current state, I rely on the normal game loop "process" call to do the polling (actually dumping queued events from the library), which loses the realtime status of built-in input events. This also causes a 2-frame delay in arrival, which results in a lag.

Could you publish the results of your work on GitHub so that others can possibly continue working on it? Thanks in advance 🙂

[Riteo]

You can actually plug multiple mice on the same computer, but the OS will only render one mouse pointer. Its movement will be the average of all mice combined together.

Without resorting to weird tricks, this is an OS limitation.

That said, X11 and Wayland actually support multiple pointers, the former through XInput2 (as reported in #3750) and the latter through the core protocol (it's super trivial).

Also, manymouse exposes this feature for all major desktop OSes so it's definitely possible even outside of those two protocols.

The hard part would be enumerating the inputs (#3749) so that it's possible to detect not only if an event is coming from a different device but also what is emitting it, especially important after the release of the Steam Deck, which requires all of its programs to switch from its joysticks and its trackpads seamlessly.

I'm pretty sure that the part that's also not available on most platform APIs is the enumeration part. While Xinput2 has apparently ways of getting further information on input devices, the Wayland core API does not and would need the aid of libinput, from a brief web search.

Note that joysticks have already implemented both of these aspects: all of their InputEvents are tagged with a different device id per joystick and there is a solid enumeration API that uses other libraries to get all the needed information.