Controller Support?

[bythos14]

Hello. Is there support for controller centric input? I'm interested in perhaps porting the library to the PS Vita, which only particularly supports controller inputs.

[mikke89]

Hi! Good question, we recently discussed this on the gitter chat, which you can read starting from here.

To summarize. There is no integrated support for this right now, so you will have to make a solution yourself on top of RmlUi. The simplest approach is probably to use attributes with an element id to point to the next element on "down", "right" etc. Then activate focus on the next element. Use the :focus selector to style it.

Of course, it is possible to do more fancy things such as automatically locating the next element based on an input direction, and tagging elements that can be navigated. If anybody makes such a solution I might be interested in integrating it into the library.

[natinusala]

Subscribing to the issue, I'm also interested in this but for the Switch 😛 The graphics layer abstraction makes it perfect to use with deko3d

[mikke89]

I'm glad to hear from more people, thanks. I'm still not sure what exactly would be useful from the library side. Perhaps you could expand a bit on your use case, maybe you have some suggestions for an api?

[natinusala]

For me at least, the need would be to say that something is "focusable", and give a way to:

1. Say what should be focused on start (and have it be focused automatically, don't wait for a tab press that's not coming)
2. Say what is the next thing to focus if the user presses an arrow key (mapped to the controller), given the currently focused thing

Kinda like what Android has but that works this time (Android keyboard navigation is really bad outside of Android TV apps).

For 2, we can either have it automatic or manual, with a preference for automatic if possible (with manual overrides if needed). For my own library I have a very basic layout system that allows me to know the relative position of everything for focus purposes, but i don't know how hard it would be to do the same thing with RmlUi.

For Switch I would also need independent scrolling (for polish only) as well as touch controls, detached from the focus system (as a bonus feature).

[mikke89]

For 1. we already have the autofocus attribute which does exactly this, see the ElementDocument::Show parameters for details.

For 2, I can see many approaches for this and I'm not sure what how it would work out best. For example, we could require that users add a special attribute to all navigatable elements. Or, we could say that we will navigate around all tab-able elements, automatically selecting the element closest in the direction being requested. Or possibly a new RCSS property.

For a library solution, I think we should try to make something general that would be suitable for most cases. This however also raises some questions for design of such a solution:

1. How do we deal with sub-menus? Eg. a common use case would be some menu on the left with sub-items on the right, expanded as the user enters this menu. Then we need to restrict navigation to this sub-part of the element hierarchy.
2. For controller / console support, it is also common to use shoulder buttons to navigate top-level menus, and joysticks/d-pads. But with keyboard, the same menus are typically "enterable" and "exitable" as in the previous point.

In some sense, the most general solution is to make use of already existing parts of the library, event callbacks, element queries and so on. So the main incentive to add navigation on the library side would be to make it simpler, but it also needs to be flexible simultaneously to cover more complex use cases. Otherwise, users would quickly find that they hit a wall so to say, and then they have to go about implementing it manually on top of the library anyway. This is why I might be a bit reluctant, I don't want to fall into the trap of designing features that mainly work for simple demos. The simple cases are already quite simple to do on the user side. Hope all of this makes sense.

What I would like to see, is some solution that is very simple to define for simple use cases, and where one can progressively define more complex behavior, covering all use cases described above and probably many others I haven't thought about.

[natinusala]

I am not familiar with RmlUi, IMO there should be a "focusable" flag on both the tags themselves and on the tag "class" if there is such a thing in RXML (every <button> is focusable, but for <div> you need to manually set the attribute). This way it's simple because you can use the built-in buttons and it will just work, while still being flexible for custom things.

How do we deal with sub-menus? Eg. a common use case would be some menu on the left with sub-items on the right, expanded as the user enters this menu. Then we need to restrict navigation to this sub-part of the element hierarchy.

Do we? Pressing LEFT when in the right sub-menu should just move the focus back to the left menu. To me the only thing to take care of are modal dialogs and menus.

I agree with the flexibility part, which is why we should gather all the needs and use cases before settling on anything. For my own thing I am currently rewriting everything because the first version lacked flexibility and was almost unusable in reality, except for the examples demo.

[mikke89]

Great, I think we generally agree on the goals here, and I'm glad you're willing to discuss it in more detail.

I'm not exactly sure what you mean in the first paragraph. We could either make a new RCSS property, use existing ones (in particular I'm thinking of tab-index or focus), or tag elements with attributes.

Do we? Pressing LEFT when in the right sub-menu should just move the focus back to the left menu. To me the only thing to take care of are modal dialogs and menus.

Do you mean some author script that does this? Otherwise we would need some way to move the focus back to that particular menu item in the left menu. In this case, one will probably want to script some events anyway, eg. opening and closing menus. Maybe a convenient approach is for the library to submit an event when the user navigates "out" of a marked element. Eg. if the user press left, and there are no navigatable buttons to the left of the current focus element, we submit an event.

[natinusala]

This is hard to explain without some drawings or mockups, what I was trying to say is that if the menu is "flat" and doesn't have any modal dialogs "above" it, we only need to handle all 4 directions, regardless of what is on screen. The generic "what is the nearest thing to focus in that direction" algorithm is enough for every use case, unless I am missing something ?

But as soon as you add "depth", we need a way to capture the focus to the current "depth" level. Modal dialogs for example are another level of depth, and the user should not be able to escape it. We also need a way to restore focus to where it was when closing the modal (so a stack of some sort).

[mikke89]

Ah, I see. What I meant earlier was generally what you call depth level here. And my point there was that once we change depth level, we need a way to remember which element within that level previously had focus. That is, because we can only have a single element at a time that actually is in focus.

Just so we're on the same page here. For example in this menu, once we navigate from an option here and back to the top menu, we want to focus on the same item that was previously selected.

Here are some suggestions:

• Let's call it navigation group instead of depth level. Since they can in principle be located anywhere with respect to each other in the general case.
• We can eg. define a navigation group using an attribute nav-group. Then all descending elements belong to that group, the navigation group is given the same name as this element's id.
• Another attribute nav-enter="[id]" : When clicked / entered, moves focus to the group or element with the given id.
• Attribute nav-back="[id]" on an element with nav-group. When user presses a designated "back" button on any element within that group, we navigate to that id.
• Generally, if the id is a normal element, we focus on that element. If the id is an element with a nav-group attribute, we focus on the element within that group that last had focus.
• Attribute: nav-focusable (?). This element can be navigated to.
  • All nav-focusable elements within a group are automatically navigated to when the user inputs a direction using some sort of hit detection.
  • The hit detection can be overridden by using the attributes nav-top/right/bottom/left="[id]".
• We emit events every time navigation enters/exits any element. Should be able to cancel the event to prevent navigation.
• We can create a new pseudo class :navfocus for active navigation elements. Only a single element within a nav-group can have an active navfocus, but it is retained when navigating to another nav-group. Thus, as opposed to :focus, multiple elements can have an active navfocus.

Actually, I think perhaps most of these attributes, especially the nav-focusable but maybe the others as well, are better suited using RCSS properties since then we can more easily use a selector to tag many elements in one go.

What do you think?

[natinusala]

I really like this navigation group idea. I think it solves the issue pretty elegantly. I even wonder if it could even be automatically set based on the parent element? A flag in the parent to say that "all focusable elements inside this element are in the same group".

However there is still the issue of modal elements, this is what I meant by "depth". If there is a modal yes / no dialog, the focus needs to go on there and it should not be allowed to leave until the user presses either one of the dialogs buttons. When the dialog is closed, the focus should go back to where it was in the "lower" element (for example the button that triggered the modal dialog).

So there should be a stack of focused elements somewhere to keep track of that... But would it not be extra hard to implement? How do you define if an element is "on top of" another? How do you know when to push and pop that stack?

[mikke89]

Great, I think we're on to something then.

Pretty much all elements can receive focus though, but we could make all elements that have a tab-index in the same group automatically navigatable?

We do already have a modal flag for whole documents at a time, would this be sufficient?

The way I'm thinking about this, navigation should never occur between nav-groups automatically. So in that sense it would act like a modal dialog "light", ie. we would still be able to use the mouse input to click on other things. Maybe you had some different thoughts about this?

You do have a point about navigating back to the previous nav-group. I mean, a separate modal document would solve it, but that may not be desirable in all cases. So perhaps we need to keep a stack then? But what if the navigation is setup in a way that it just keeps going forward, then the stack would just grow? Not sure how best to deal with it.

[natinusala]

We do already have a modal flag for whole documents at a time, would this be sufficient?

You tell me, I don't know how RmlUi works 😄 That would work assuming you ensure people always use that flag for all modals.

The way I'm thinking about this, navigation should never occur between nav-groups automatically

What do you mean by "automatically"? I didn't intend navigation groups to be modals in any way.

For my use case we don't need the mouse at all but others (PC games) could use controller + mouse. However if we count touch as mouse input, then Switch will need controller + mouse.

As for the stack thing, I only intended to use a stack for actual modals, not all nav groups, they are separate things to me. The focus stack should be tied to the modal elements stack. That way, each modal element in the stack has its own "last focused sub-element" attribute and it will not leak. If you keep going forward without ever deleting elements you will leak elements anyway. We just need to find a proper way to manage that stack to prevent leaking (= properly dispose of elements in the stack when they are removed from the screen (?)).

[mikke89]

Hm, the way I thought about navigation groups is that you only navigate within them when pressing up/down/left/right. It's only when you press "enter" or "back" buttons you navigate to another designated nav-group. I guess we could make the direction buttons work as well where that makes sense, but only to a designated nav-group id.

Maybe you could draw a crude illustration so I can understand your use case better? :)

[natinusala]

I think we are talking about two entirely different things. Here's what I have in mind so far:

1. nav-groups are used to "remember" where the focus was, so that when you enter the group it focuses the last focused element when you left, I don't see why we would need to group elements otherwise?
2. the focus stack for modals, to remember where the focus was when the user closes a modal element (and a modal element only)

[mikke89]

So how do you propose we decide when navigate to another nav-group, and to which nav-group?

I was thinking nav-groups serve two purposes:

1. Remember where the focus was as you say.
2. Restrict directional navigation to within same nav-group.

Then, it is up to the client to decide when to exit or enter the given nav-group, but typically with designated enter/back buttons.

I think modals are generally better implemented as separate documents, that should already be well supported.

[natinusala]

Restrict directional navigation to within same nav-group.

So you want a nav-group to only allow navigation in one direction between its elements, and to go from one group to another the user has to manually specify the path?

I made a (very crude) schema:

Here we see 3 navigation groups (the tabs, the settings and the buttons). Navigation between elements of the same group are automatic: the tabs group is horizontal, the settings group is vertical and the buttons are horizontal as well. Then, the arrows represent the manual mapping, for instance to go from the tabs to the settings you press DOWN, etc...

That would work for most situations I think. However I don't think we want to remember the focus position for all groups, it should be toggled with a flag.

[eefan000]

https://github.com/ZhuRong-HomoStation/RmlUE4
I'm also looking forward to this function!!!
When I have it, I will try to use it in UE4 project

[mikke89]

@natinusala That's a good illustration and description of what I'm thinking. And you make a good point, not all nav-groups should remember the focus. Maybe most of them shouldn't, at least it should be configurable.

We could also accommodate automatic navigation between nav-groups. Eg. when there are no hits within the nav-group in the given direction, we next try to hit a new nav-group in that same direction.

On consoles I would probably implement the top-bar with shoulder buttons, and Ok/Cancel/Apply with designated buttons. Not sure how or if we should accommodate that, since we now have two nav-groups that can simultaneously be navigated. Maybe the top bar is better navigated with scripting. Or we could perhaps make an attribute for that nav-group, eg. nav-button-left="shoulder_left" nav-button-right="shoulder_right" .

We should also discuss how to do hit detection for automatic navigation. The way I envision it is that we take a rectangle of the same size and position of the currently focused element, extend it in the input direction, and pick the closest nav-focusable element that hits this rectangle.

@eefan000 Interesting, I haven't seen this plugin before. Very cool to see integration with UE4 :)

[eefan000]

https://docs.unity3d.com/2020.2/Documentation/Manual/script-SelectableNavigation.html
You can refer to the navigation of unity

https://docs.unity3d.com/2020.2/Documentation/Manual/UIElements.html
This is very similar to rmlui

[natinusala]

We could also accommodate automatic navigation between nav-groups. Eg. when there are no hits within the nav-group in the given direction, we next try to hit a new nav-group in that same direction.

Then why bother making nav groups at all? Why can't we just use automatic hit detection for everything?

We should also discuss how to do hit detection for automatic navigation. The way I envision it is that we take a rectangle of the same size and position of the currently focused element, extend it in the input direction, and pick the closest nav-focusable element that hits this rectangle.

That would not work for my illustration because the three bottom-right buttons are not horizontally aligned with the settings list, so pressing down here would not lead anywhere.

[mikke89]

@eefan000 That's a great reference. Maybe we could add something like that as an RCSS property.

@natinusala

Then why bother making nav groups at all? Why can't we just use automatic hit detection for everything?

Maybe I'm overthinking this, but the whole point was to allow for a hierarchical navigation, with multiple focus-like states. Perhaps something simpler with manual override is better.

That would not work for my illustration because the three bottom-right buttons are not horizontally aligned with the settings list, so pressing down here would not lead anywhere.

Do you have a better proposal?

[natinusala]

Do you have a better proposal?

An efficient one? Sadly no, however I am curious to see how Android does it.

In my library I do it by traversing the known layout but it's cheating since the layout dictates how everything is placed relatively to each other, which is what we want here. If you know the layout, you know where everything is. I don't know if RCSS is simple enough to guarantee that.

edit I mean I don't know if you can know the exact layout by just looking at the RXML and RCSS, it's too complicated, there's more than that is there

[mikke89]

Yeah, we cannot determine relative placement from the RML. It has to be done based on the output from the layout engine, mainly boxes.

[eefan000]

After comparing the navigation of UE4 and unity, I think the navigation of unity is more perfect

[eefan000]

https://docs.microsoft.com/en-us/windows/uwp/gaming/ui-navigation-controller

https://docs.microsoft.com/en-au/windows/uwp/design/input/focus-navigation-programmatic

This document is also suitable for reference

[LWSS]

@bythos14 how's the vita port coming?

[bythos14]

@bythos14 how's the vita port coming?

Didn't go through with it. Will leave this issue open for others who are interested in the topic.

[andreasschultes]

I just want to mention here that css basic user interface model level 4 defines some useful things:
https://www.w3.org/TR/css-ui-4/#keyboard

[mikke89]

Nice find! Generally, it is good if we can be compatible with CSS, and this seems like a decent, basic start. Maybe a bit cumbersome to specify individual IDs in CSS rules. We could possibly later on add more options like ray-casting and grouping on top of that.

[andreasschultes]

There is also a working draft for spatial navigation
https://github.com/WICG/spatial-navigation
The GitHub page has at the end a interesting list with links

[gleblebedev]

#519

[mikke89]

This feature has been merged in #524. Special thanks to @gleblebedev for the initial implementation, and everyone else in here for providing a lot of useful material.

The documentation has been updated with the new properties: https://mikke89.github.io/RmlUiDoc/pages/rcss/user_interface.html#nav

I'm sure we can further improve this feature in the future. Usage feedback would be very much appreciated. I'm closing this issue, any further discussion or feedback should be done by opening new issues or discussion posts.