UI: Interaction has limited usefulness, should have feature parity with Input

[inodentry]

What problem does this solve or what need does it fill?

Currently Interaction only detects 3 conditions:

• Clicked: while pressed using mouse/touch
• Hovered: while the mouse pointer is on top
• None: none of the above

This is very limiting. For comparison, consider the kinds of things you can detect using Input<MouseButton>:

• pressed: the button is held down (like Interaction::Clicked)
• released: the button is not pressed
• just_pressed: only true the first frame when the button got pressed
• just_released: only true the first frame when the button stopped being pressed

Notably, just_pressed/just_released allow detecting the moment when the mouse click starts or ends. Having such functionality available for UI too, would be very useful.

Consider a button in some UI. In many scenarios, you probably want to trigger its action only once, not repeatedly every frame while the button is held down (as currently happens if using Interaction::Clicked). Further, having the choice between doing it when the button is just pressed vs. just released, can be very useful depending on the intended user experience. Doing it on release can allow the user to cancel / back out of accidentally pressing the wrong UI button, by moving the cursor out of it before releasing. Doing it on on press means instant action/response with no delay, important for buttons in many game UIs.

What solution would you like?

Interaction, instead of being an enum, could be converted to an API that mirrors that of Input, offering the pressed/just_pressed/released/just_released methods, + additionally also hover detection using a hovered method.

Pressing should only register if the event happened within the UI element. Pressing the mouse button outside, and then dragging the cursor in, should not cause the Interaction to become pressed.

just_released should only trigger if the release event happens within the UI element. If the user presses the UI element, then moves the cursor out of it and releases it there, there should be no just_released. Instead, we could provide a canceled method for detecting this situation.

Touchscreens should behave consistent with this behavior.

This would allow users to write UI code using the same familiar API as for gameplay input. UI has its own additional needs (like hover detection), and this is why I am not proposing to try to reuse Input itself outright.

What alternative(s) have you considered?

Extending the enum with more variants:

enum Interaction {
    None,
    Hovered,
    JustPressed,
    Pressed,
    JustReleased,
}

This is a more straightforward extension of what already exists. Hovever, it suffers from poor usability. The equivalent of the current Clicked (detect the button being held down) would have to check for either JustPressed OR Pressed.

Additional context

I am currently working around the limitations of Interaction by also accessing Res<Input<MouseButton>> and trying to use both simultaneously, so that I can implement the desired behavior. I need Interaction, so that I know if the specific UI element is being interacted with, and then I need Input<MouseButton> for just_pressed/just_released to figure out the missing information. This is very clunky and annoying, and does not support touch screens.

[alice-i-cecile]

I like the direction of this design.

What if we did an enum, but added pressed / released methods?

[inodentry]

I don't like the enum, because it requires the user to learn two different styles of API for gameplay (using Input) and UI (using Interaction). These are conceputally very similar, and I think having symmetric APIs would be very nice.

Further, only supporting methods has its benefits: UI interaction is even more complex that Input, supporting hover and cancellation, and that is tricky to expose clearly via enum variants, but easy to make methods that intuitively do the right thing.

I don't see a benefit to a "middle ground".

[mockersf]

I use changed detection for that.
Changed + clicked => just pressed, and so on

[inodentry]

That only works in simple scenarios.

For example, I recently had a use case where I had to track other state/components too, and then that meant that change detection wouldn't let my query access the entities I needed.

I could make multiple queries, use param sets, etc., to make it possible to use change detection, but at that point using Res<Input<MouseButton>> was easier.

Both of these "solutions" are just workarounds. The ergonomic solution would be to have Interaction track the relevant state, hence this issue.

[inodentry]

Also, I think it is impossible to detect "just released" with change detection. What would you do? if is_changed && interaction != Clicked? That would also trigger on hover. You can't detect what the old value was / what you changed from, only what you changed to. So you can't know if the previous state was Clicked.

[Weibye]

Related: #5769, #7240

While working on #7240 I had similar thoughts and I think this issue describes my preferred solution pretty well. We do need more granular interaction support.

Thought: Should Hover be separated out as its own thing? There are use-cases for widgets needing to do things on hover, regardless of "clicked, pressed, released" state.

enum Interaction {
    JustPressed, 
    Released, // (same as JustReleased)
    Pressed,
    None
}

enum Hover {
    JustEntered
    JustExited
    Hover
    None
}

[alice-i-cecile]

Oh I like splitting Hover out quite a bit. This is particularly nice because on touch devices you're never going to have a hover state.

[inodentry]

I don't like splitting Hover, because systems that handle UI interaction should be aware of and have easy access to all the possible interaction states that a UI Node can be in.

If a user wants to support UI interaction properly regardless of input device, they probably want access to it all.

With my proposed solution, if users don't care about hover ... they can just not call hovered to check for it. Nobody is forcing you to detect it.

Splitting it into a separate component "just because not everybody wants it" just adds friction for everyone who does. People now need to care about two separate components, and perform logic to check the values of both. That's worse UX imo.

---

Also, I really prefer not using enums, because enums force a very specific state machine mental model (the value can only ever be exactly one of the variants in the set) and makes the actual practical use cases clunky to handle.

Even the proposal above showcases how this is insufficient. What is the difference between JustReleased and Released? If I am a user writing a UI interaction system, how do I know what I am supposed to check for? It is confusing.

Just like in my original post, consider being like Input. Having an API based on method calls frees us from the strict "state machine model" that enums impose. Input can simultaneously be pressed and just_pressed, and indeed that's what it should be on the first frame when the press happened. Users can call the respective method depending on which scenario they care about to detect. It is much more user friendly.

I strongly urge us to consider moving UI interaction to a similar method-based API style.

Further, this allows us to have a cancelled method, as I had described before, to serve as a just_released if the user moved the cursor/touch out of the button (presumably because they changed their mind about wanting to press it).

If the mouse/touch release event happens within the confines of the UI node, then released and just_released will be true. If the mouse/touch release event happens outside the UI node, then released and cancelled will be true, but just_released will be false.

We cannot provide such nice user-friendly APIs with enums. UI interaction state is more complex and does not map cleanly to enums.

[jkb0o]

From my point of view, the current Interaction tries to cover two separate functionalities:

• react on input (execute some block of logic when Interaction becomes Clicked)
• change appearance of the ui widget based on the state (Clicked/Hovered/None)

I'd like to see something like PointerInput event in bevy with associated entities (what entities are affected by this event) and data (cursor position or motion delta). There is an example of enum items that should be covered: Down, Up, Press (up after down), Motion (cursor movement), DragStart (first motion while mouse/touch pressed), DragStop (up after drag start), Drag (mouse/touch motion while pressed).

With such PointerInput events it is easy to react on exact input and/or implement complex input behaviours like drag-n-drop.

The appearance aspect works great in html/css: each element may have one or more states (pseudo classes) like :press, :active, :hover, etc. User specify css rules to precisely define exact look of the element.

Bevy could provide something like UiState component and change it in response to PointerInput (or other inputs), for example:

// it could be just String/'static &str or some sort of ImmutableString 
pub enum UiStateEnum {
 Pressed,
 Hovered,
 Active,
 Focused,
}

#[derive(Component)
pub struct UiState(HashSet<UiStateEnum>);

Later user can write separate system that covers ui appearance in response to UiState changes.

With PointerInput and UiState it will be much easier to implement complex widgets (like sliders/text inputs/etc.)

I've been experimenting with such a approach for some time and it feels very comfortable and powerful in belly. This is how pointer_input_system looks like:
https://github.com/jkb0o/belly/blob/dbd52b64ad8a29b15ad107f14f467b77f058ed0a/crates/belly_core/src/input.rs#L195-L423