Gampad and arrow-keys navigation support

[kissmikijr]

Is your feature request related to a problem? Please describe.

I'd like to be able to navigate on the gui elments using a gamepad

[kissmikijr]

I am happy to work on this, however I'll definitely need some pointers where to start.

[emilk]

I am happy to work on this, however I'll definitely need some pointers where to start.

That would be wonderful!

The current focus system (TAB/shift-tab) is based on which order widgets are added, and is found in struct Focus in egui/src/memory.rs. In particular, Focus::interested_in_focus is called for every new widget that is added on screen.

To handle moving focus in the four cardinal directions we would need to also supply interested_in_focus with a Rect of the widget (that's the easy part). I think then the logic we would need be something like this:

• Store the Rect of the currently focused widget in Focus
• When user presses gamepad (or arrow keys), mark this frame as a search for a widget in the given cardinal direction
• Store the best_candidate next widget (initial: None)
• For each widget that is added with interested_in_focus, see if it is a better candidate than the current best_candidate (in the correct direction and closer, with some heuristics)
• At the end of the frame, move focus to the best_candidate

Alternatively:

• Store all interactive widgets in a IdMap<(Rect, Sense)>). This will be useful for other things as well, like Cache bounding boxes for "senses" #724
• When the user presses a gamepad button, find the best candidate from the above map.

This will be slightly more expensive, but probably not by much, and the code will potentially be much cleaner.

[akhilman]

My idea is to store the bounding boxes of widgets during rendering along with the corresponding senses.

Further, we can easily implement the activation of the widget in an arbitrary direction, even using an analogue stick. To do this, it will be enough, using the saved data, to filter the widgets with the necessary senses, filter the widgets in the right direction from the active widget, and finally select the nearest widget.

This size list will also help implement mouse event filtering, as described in #724.

[emilk]

This is the same functionality required in order to be able to focus widgets with the arrow keys, so I'll add that to this issue.

[TimonPost]

We need gamepad support for egui and I'm looking to either try doing something similar like https://github.com/idanarye/bevy-egui-kbgp or to implement this into egui directly. I kinda want to do it in egui as it looks like a valuable addition.

Making egui full-gamepad compatible is probably a slightly bigger task (e.g dragging sliders, checkboxes, inputting text). For us its important we can just do the navigation for now.

I am going to try giving the alternative approach a try, storing rects in a list, and then detecting what rects are closest in a certain cardinal direction. And also directly try implementing the keyboard arrows.

• Where should this list IdMap<(Rect, Sense)> live? Inside the Memory struct caches, or in data, or new field?
• Does egui want 'gamepad events' in it Event type? Or should this solution be more low-level that users can potentially query to make it easier to directionally toggle between elements? There is not a good way now to tell if a game-pad event occurred.

Any tips that would help speeding up the implementation?