Gamepad / Keyboard navigation and interactions!

[ocornut]

EDIT: The navigation branch has been merged to master. See #1599. Further improvements/fixes will be pushed to mainline.

Opening a new thread because the old one ( #323 ) became overwhelmingly long and unwelcoming. This is a trimmed post with info that are still relevant.

TL;DR; there is a new branch that add supports for gamepad/joystick navigation. The same system can be used for keyboard navigation to some degree, but the initial focus is on gamepad. Typically you can use that to access your tools on PS4/XBone/Wii-U/etc without a synergy/mouse/keyboard setup. Best if you can still have even a virtual mouse around (e.g. on DualShock4 touch pad).

EDIT March 2018
Link to PNG + PSD depicting the controls for DualShock 4 and Joy-Con
https://drive.google.com/open?id=1k4328OV-w20pWZfNcfpH0UoxHHQRkbQi

EDIT: Now in master!
Branch: https://github.com/ocornut/imgui/tree/navigation
(checkout branch navigation or download from web and overwrite your imgui_xxx files)

I'm calling it beta because:

• This feature required changing lots of code. The branch probably has bugs.
• It is rough and work in progress. The more I add and fix things the more I see new things to do. My initial schedule projection was a joke. Long tail feature.
• But it is pretty useful already!
• I would ideally like to merge this in master but I can only do so with more testing and feedback.
  Even if you don't need gamepad navigation, using this branch without wiring the inputs would be useful testing.

What I would like from users:

• See how it fits in your real-world app and what we need to fix/add.
• Any bug report, questions, features request, welcome. Please be critical!

The development of this feature has been partly sponsored by Insomniac Games (thank you!).

My current mapping for DualShock4

D-Pad up/down/left/right: navigate, tweak values
Cross button: press button, hold to tweak/activate widget, enter child, etc.
Circle button: close popup, exit child, clear selection, etc.
Square button(TAP): access menu, collapsing, window options, etc.
Square button(HOLD)+Dpad: resize window
Square button(HOLD)+Analog: move window
Square button(HOLD)+L/R trigger changes window focus, ALT-TAB style
Triangle button: text input (requires user back-end reading back io.WantTextInput, possibly display an OS keyboard display).
L/R Trigger: slow down/speed up tweaking values
Analog stick: manual scroll.

Quick instructions

// Fill ImGuiIO.NavInputs[] float array every frame to feed gamepad/keyboard navigation inputs.
// 0.0f= not held. 1.0f= fully held. Pass intermediate 0.0f..1.0f values for analog triggers/sticks.
// ImGui uses a simple >0.0f for activation testing, and won't attempt to test for a dead-zone.
// Your code passing analog gamepad values is likely to want to transform your raw inputs, using a dead-zone and maybe a power curve.
enum ImGuiNavInput_
{
    ImGuiNavInput_PadActivate,      // press button, tweak value                    // e.g. Circle button
    ImGuiNavInput_PadCancel,        // close menu/popup/child, lose selection       // e.g. Cross button
    ImGuiNavInput_PadInput,         // text input                                   // e.g. Triangle button
    ImGuiNavInput_PadMenu,          // access menu, focus, move, resize             // e.g. Square button
    ImGuiNavInput_PadUp,            // move up, resize window (with PadMenu held)   // e.g. D-pad up/down/left/right, analog
    ImGuiNavInput_PadDown,          // move down
    ImGuiNavInput_PadLeft,          // move left
    ImGuiNavInput_PadRight,         // move right
    ImGuiNavInput_PadScrollUp,      // scroll up, move window (with PadMenu held)   // e.g. right stick up/down/left/right, analog
    ImGuiNavInput_PadScrollDown,    // "
    ImGuiNavInput_PadScrollLeft,    //
    ImGuiNavInput_PadScrollRight,   //
    ImGuiNavInput_PadFocusPrev,     // next window (with PadMenu held)              // e.g. L-trigger
    ImGuiNavInput_PadFocusNext,     // prev window (with PadMenu held)              // e.g. R-trigger
    ImGuiNavInput_PadTweakSlow,     // slower tweaks                                // e.g. L-trigger, analog
    ImGuiNavInput_PadTweakFast,     // faster tweaks                                // e.g. R-trigger, analog
    ImGuiNavInput_COUNT,
};

Current blurb in imgui.cpp (I know it is cropped by github but please read it)

 USING GAMEPAD/KEYBOARD NAVIGATION [BETA]

 - Gamepad/keyboard navigation support is available, currently in Beta with some issues. Your feedback and bug reports are welcome.
 - See https://github.com/ocornut/imgui/issues/323 discussion thread and ask questions there.
 - The current primary focus is to support game controllers.
 - Consider emulating a mouse cursor with DualShock4 touch pad or a spare analog stick as a mouse-emulation fallback.
 - Consider using Synergy host (on your computer) + uSynergy.c (in your console/tablet/phone app) to use PC mouse/keyboard.
 - Your inputs are passed to imgui by filling the io.NavInputs[] array. See 'enum ImGuiNavInput_' in imgui.h for a description of available inputs.
 - For gamepad use, the easiest approach is to go all-or-nothing, with a buttons combo that toggle your inputs between imgui and your game/application.
   Sharing inputs in a more advanced or granular way between imgui and your game/application may be tricky and requires further work on imgui.
   For more advanced uses, you may want to use:
     - io.NavUsable: true when a window is focused and it doesn't have the ImGuiWindowFlags_NoNavInputs flag set.
     - io.NavActive: true when the navigation cursor is visible (and usually goes false when mouse is used).
     - query focus information with IsWindowFocused(), IsAnyWindowFocused(), IsAnyItemFocused() functions.
   The reality is more complex than what those flags can express. Please discuss your issues and usage scenario in the thread above. 
   As we head toward more keyboard-oriented development this aspect will need to be improved.
 - It is recommended that you enable the 'io.NavMovesMouse' option. Enabling it instructs ImGui that it can move your move cursor to track navigated items and ease readability.
   When enabled and using directional navigation (with d-pad or arrow keys), the NewFrame() functions may alter 'io.MousePos' and set 'io.WantMoveMouse' to notify you that it did so.
   When that happens your back-end NEEDS to move the OS or underlying mouse cursor on the next frame. The examples binding in examples/ do that.
   (Important: It you set 'io.NavMovesMouse' to true but don't honor 'io.WantMoveMouse' properly, imgui will misbehave as it will think your mouse is moving back and forth.)

     // Application init
     io.NavMovesMouse = true;

     // Application main loop
     if (io.WantMoveMouse)
        MyFuncToSetMousePosition(io.MousePos.x, io.MousePos.y);
     ImGui::NewFrame();

   In a setup when you may not have easy control over the mouse cursor (e.g. uSynergy.c doesn't expose moving remote mouse cursor),
   you might want to set a boolean to ignore your other external mouse positions until they move again

1. THE INITIAL FOCUS IS ON USING A GAME CONTROLLER! The inputs are explicitly named using the word "pad" because the current scheme is optimized for game controllers. You can however trivially map keyboard keys on those inputs and it'll work if you use mouse+keyboard combo. Later on I will focus on keyboard and add especially named enums for keyboard (user expectation for keyboard controls are much higher/harsher than with gamepad).

2. Depending on your application using gamepad/keyboard might requires more fine-tuned controls of how inputs are dispatched and shared between your app/game and different parts of your imgui interfaces. We may need better options/tooling for that.

3. It is very probably missing or failing at something that will appear obvious to you. I've nailed many issues and big technical problems but barely started scrapping the surface of all possible usage scenarios. Please report!

4. The actual scoring function for navigating the graph hasn't been given a lot of love yet and is failing to behave as expected in various cases. Failure usually mean that you press a direction and don't end up exactly where you intended. Will keep improving it. Please report issues with screenshots!

5. Addition to the public API are rather minimal. New functions IsItemFocused(), IsAnyItemFocused(), SetItemDefaultFocus(), GetKeyPressedAmount(). I agressively made IsItemHovered() be aware of current navigation focus to maximize existing code just naturally working with Nav (typically tooltip pattern). New window flags ImGuiWindowFlags_NoNavFocus ImGuiWindowFlags_NoNavInputs. New colors ImGuiCol_NavHighlight (make it same or close to ImGuiCol_HeaderActive) ImGuiCol_NavWindowingHighlight (white and very transparent), a bunch of keys (read instructions), 1 IO setting NavMovesMouse, 3 IO outputs WantMoveMouse NavUsable NavActive.

6. The option io.NavMovesMouse is currently off by default. I recommend enabling it. When enabled. the mouse cursor can be moved by ImGui::NewFrame() when directional navigation is used. It does so by overwriting io.MousePos and set io.WantMoveMouse=true. It is up to your backend when that flag is set to apply the new mouse position in your OS. If you enable the option but don't honor those requests, ImGui will be very confused. (this is why I can't have it on by default).

7. If you are running this on VR, some suggestions: if you want to display ImGui as a static overlay (not affected by head rotation) you may want to reduce DisplaySize and avoid rendering over your entire framebuffer. You can also increase the style.DisplaySafeAreaPadding value. Popups should stay within this rectangle while you can still partly move regular window outside. It might be just better to display it within the 3D world but I haven't tried.

Following in the next message will be my test code for GLFW binding to map a DualShock 4.

TODO list

• A. Sort-out/finalize all the input bindings correctly.
• B. Menus: Navigating menus is still awkward in multiple ways.
• B. Investigate crossing over the boundaries of child windows, in particular those without scroll. Introduce a window flag to flatten child in term of navigation.
• C. Menubars inside modals windows are acting weird (broken in master as well)
• A. Problem various problem with graph navigation/scoring functions, currently biased toward vertical layouts.
• C. Using scrolling should activate scrollbar in a way the user can tell programmatically (e.g. Log window).
• C. NavHighlight clipping issue within child window.
• C. Merge all the old FocusIdx tabbing stuff into the new system.
• B. Resizing window will currently fail with certain types of resizing constraints/callback applied
• C. Popup: introduce a default validation button e.g. SetItemDefaultValidation() activable from anywhere in the window with Enter. Currently can use imgui_internal.h declared ImGui::PushItemFlag(ImGuiItemFlags_SelectableDontClosePopup, true); / ImGui::PopItemFlag()
• C. TreeNode: NavLeft to close, NavRight on closed node to open. How would it behave with buttons/items after a closed treenode, and/or multiple columns?
• B. Can't reliably use Left/Right within menus with regular widgets. Need to figure out a way to only use the Left/Right nav requests for menu open/closure as fallback to a failed moving request.
• C. Drag/Slider: experiment with keeping item active when activated, using cancel to stop editing.
• B. Popup: add options to disable auto-closing popups when using a MenuItem/Selectable (Menus API work #126) (not part of Nav)
• C. Lost of currently focused widget when using buttons that changes labels on click (obvious, but only made apparent with directional navigation- can we automagically work around it?)
• B. Bug with keeping visibility of navigated them within horizontal scrollbar. Stuck nav (visible in Horizontal Scrolling demo corner. Still there?).

[ocornut]

BINDING EXAMPLE CODE

Here's my ugly test binding for GLFW+DualShock4+DS4Window as rough guidance.

We shall make that less ugly but as soon as we venture into controllers it's hard to get anything portable and reliable. Will probably add a dedicated input visualization/configuration window to make it easier. Currently under "Keyboard,Mouse,etc." in the demo window there's a panel that shows all pressed inputs.

EDIT 2017/10/21: introduced ImGuiNavInput_KeyMenu which is the first explicitly keyboard-related enum. It handle the menu toggling like ImGuiNavInput_PadMenu but when held doesn't trigger the window highlight that is used for resizing/moving Windows with the pad.

Gamepad (GLFW, basing mapped on DualShock4+DS4Window)

// Setup directional navigation events/key mapping [BETA]
memset(io.NavInputs, 0, sizeof(io.NavInputs));

// Enable to allow ImGui moving mouse cursor when using keyboard/gamepad navigation
io.NavMovesMouse = true;

// Update Gamepad Inputs
const bool nav_uses_gamepad = true;
if (nav_uses_gamepad)
{
    #define MAP_BUTTON(NAV_NO, BUTTON_NO)       { if (buttons_count > BUTTON_NO && buttons[BUTTON_NO] == GLFW_PRESS) io.NavInputs[NAV_NO] = 1.0f; }
    #define MAP_ANALOG(NAV_NO, AXIS_NO, V0, V1) { float v = (axes_count > AXIS_NO) ? axes[AXIS_NO] : V0; v = (v - V0) / (V1 - V0); if (v > 1.0f) v = 1.0f; if (io.NavInputs[NAV_NO] < v) io.NavInputs[NAV_NO] = v; }
    int axes_count = 0, buttons_count = 0;
    const float* axes = glfwGetJoystickAxes(GLFW_JOYSTICK_1, &axes_count);
    const unsigned char* buttons = glfwGetJoystickButtons(GLFW_JOYSTICK_1, &buttons_count);
    MAP_BUTTON(ImGuiNavInput_PadActivate,   0);
    MAP_BUTTON(ImGuiNavInput_PadCancel,     1);
    MAP_BUTTON(ImGuiNavInput_PadMenu,       2);
    MAP_BUTTON(ImGuiNavInput_PadInput,      3);
    MAP_BUTTON(ImGuiNavInput_PadUp,         10);
    MAP_BUTTON(ImGuiNavInput_PadDown,       12);
    MAP_BUTTON(ImGuiNavInput_PadLeft,       13);
    MAP_BUTTON(ImGuiNavInput_PadRight,      11);
    MAP_BUTTON(ImGuiNavInput_PadFocusPrev,  4);
    MAP_BUTTON(ImGuiNavInput_PadFocusNext,  5);
    MAP_BUTTON(ImGuiNavInput_PadTweakSlow,  4);
    MAP_BUTTON(ImGuiNavInput_PadTweakFast,  5);
    MAP_ANALOG(ImGuiNavInput_PadScrollUp,   1,  +0.3f,  +0.9f);
    MAP_ANALOG(ImGuiNavInput_PadScrollDown, 1,  -0.3f,  -0.9f);
    MAP_ANALOG(ImGuiNavInput_PadScrollLeft, 0,  -0.3f,  -0.9f);
    MAP_ANALOG(ImGuiNavInput_PadScrollRight,0,  +0.3f,  +0.9f);
    #undef MAP_BUTTON
    #undef MAP_ANALOG
}

For keyboard (GLFW)

// Update Keyboard Inputs
const bool nav_uses_keyboard = true;
if (nav_uses_keyboard)
{
    #define MAP_KEY(NAV_NO, KEY_NO) { if (io.KeysDown[KEY_NO]) io.NavInputs[NAV_NO] = 1.0f; }
    MAP_KEY(ImGuiNavInput_PadActivate,      GLFW_KEY_SPACE);
    MAP_KEY(ImGuiNavInput_PadCancel,        GLFW_KEY_ESCAPE);
    MAP_KEY(ImGuiNavInput_KeyMenu,          GLFW_KEY_LEFT_ALT);
    MAP_KEY(ImGuiNavInput_PadInput,         GLFW_KEY_ENTER);
    MAP_KEY(ImGuiNavInput_PadUp,            GLFW_KEY_UP);
    MAP_KEY(ImGuiNavInput_PadDown,          GLFW_KEY_DOWN);
    MAP_KEY(ImGuiNavInput_PadLeft,          GLFW_KEY_LEFT);
    MAP_KEY(ImGuiNavInput_PadRight,         GLFW_KEY_RIGHT);
    MAP_KEY(ImGuiNavInput_PadTweakSlow,     GLFW_KEY_LEFT_ALT);
    MAP_KEY(ImGuiNavInput_PadTweakSlow,     GLFW_KEY_RIGHT_ALT);
    MAP_KEY(ImGuiNavInput_PadTweakFast,     GLFW_KEY_LEFT_SHIFT);
    MAP_KEY(ImGuiNavInput_PadTweakFast,     GLFW_KEY_RIGHT_SHIFT);
    #undef MAP_KEY
}

For keyboard (DX11)

    // Update Keyboard Inputs
    const bool nav_uses_keyboard = true;
    if (nav_uses_keyboard)
    {
#define MAP_KEY(NAV_NO, KEY_NO) { if (io.KeysDown[KEY_NO]) io.NavInputs[NAV_NO] = 1.0f; }
        MAP_KEY(ImGuiNavInput_PadActivate, VK_SPACE);
        MAP_KEY(ImGuiNavInput_PadCancel, VK_ESCAPE);
        MAP_KEY(ImGuiNavInput_KeyMenu, VK_MENU);
        MAP_KEY(ImGuiNavInput_PadInput, VK_RETURN);
        MAP_KEY(ImGuiNavInput_PadUp, VK_UP);
        MAP_KEY(ImGuiNavInput_PadDown, VK_DOWN);
        MAP_KEY(ImGuiNavInput_PadLeft, VK_LEFT);
        MAP_KEY(ImGuiNavInput_PadRight, VK_RIGHT);
        MAP_KEY(ImGuiNavInput_PadTweakSlow, VK_LMENU);
        MAP_KEY(ImGuiNavInput_PadTweakSlow, VK_RMENU);
        MAP_KEY(ImGuiNavInput_PadTweakFast, VK_LSHIFT);
        MAP_KEY(ImGuiNavInput_PadTweakFast, VK_RSHIFT);
#undef MAP_KEY
}

Will improve this skeleton in user examples, but I don't really expect to be able to provide working examples for all backends and controllers just because mapping gamepad buttons is an unsolved computer science mystery. I'll probably devise a set of functions where mapping can easily be setup from the outside.

[unpacklo]

Not particularly helpful feedback, but thought I'd let you know that I'm currently seeing these warnings on PS4:

imgui.cpp(2965,9): warning : add explicit braces to avoid dangling else [-Wdangling-else]
imgui.cpp(3788,13): warning : unused function 'IsKeyDownMap' [-Wunused-function]

We tend to compile with -Wall !

-Dale Kim

[karacaenes]

Hmm. Depends on how you design the ui but this could actually make imgui useful on embedded devices where there is no mouse and only a keypad as the only input device. I will give it a try soon, possibly with a directfb backend. I have such device at my hand, which only has a numberpad, up and down buttons an enter button in the middle of them and a couple of other keys that could be used as left/right or tab keys. Right now i am drawing everything with DirectFB myself and getting input one screen at a time. (enter username on one screen press enter, enter password on another screen press enter, see a loading screen, get presented with a menu that covers whole screen etc.)

[uglycoyote]

Thanks so much for the progress on getting gamepad navigation working.

We have an existing custom (non-imgui) tree-based debug menu in our game. I created an alternate implementation in ImGui and am considering deprecating the old one eventually. It mostly depends on getting the game-pad support right as we don't always have the ability to use mouse on our game builds.

The menu is essentially just a big tree containing mostly booleans, ints and floats and "triggers", so in the imgui implementation they are mostly checkboxes, sliders, and buttons.

I'm trying out the latest 2016-07-Navigation branch. At the moment I'm finding the dpad left/right behaviour particularly frustrating.

In my case the thing I'm trying to replace something that users have been using for about 10 years and the controls are ingrained in everyone's muscle memory so it would be nice to be able to configure the controls in ImGui to work as similar as possible to how the tree navigation works under that system.

This is how our existing menu system works and how I hoped ImGui would be able to configured to work:

When focused on a folder in the tree:

• Press PadRight or "A" (activate) to expand a folder in the tree
• Press PadLeft or "B" (cancel) to collapse the folder

When focused on a leaf node in the tree

• Press PadRight or "A" to toggle a checkbox or activate a button
• Press PadRight or PadLeft to adjust a slider
• Press B to collapse the parent folder and set parent as the focused item

In ImGui Pressing PadRight takes the focus away from wherever you are in the tree and off to never-never land. From looking at the code I understand that it's using some kind of scoring system for determining where the focus goes, but at least in my use case it feels wrong and unpredictable. It either focuses another item higher in the tree (presumably because that item is indented more to the right) or it jumps the focus to a "Clear" button that I have on the filter box at the top of the tree. Either way it's taking the input focus away from something that you just pressed the d-pad a few dozen times to get to, so it's frustrating in that regard.

I know others on this thread and the previous one have said similar things, but the rebuttal was that the dpad navigation needs to be able to work with multiple columns and with tree nodes which have multiple things in them that you can focus on. But in my case I only have one column and each tree node only has one focusable element, (basically the simplest kind of tree) so it would be nice if it worked well in this situation.

The scoring system leads to unpredictable jumping of focus which takes the user a long time to recover from. IMHO the scoring system should only be used when there's not some simpler more direct context-specific interpretation of the arrow keys. Once you enter the "tree navigation" context it should not use the scoring system anymore but a more custom logic for interpreting directional inputs such as the rules I listed above.

[ocornut]

NB: The branch has been renamed from 2016-07-navigation to navigation removing the date in it. I think the date made people think the branch was not keep up to date, while my intent was merely to note approximate date a branch was being worked on. Of course, for those long standing branch it doesn't make much sense. I'll be using the date when closing branches.

[ocornut]

Update: while I don't have a date for merging this branch, I have already merged many of the non-strictly-navigation-related changes back to Master, so the diff between branches is smaller (and some of the changes will be tested a little more due to being in Master already).

[inolen]

Just gave this an initial spin - it worked right out of the box!

Out of curiosity, I skimmed #323, what was the reasoning for moving to the NavInputs array vs reusing the existing KeysDown / KeyMap setup? When initially setting everything up, I realized I was just assigning NavInputs[key] = KeysDown[KeyMap[key]]; effectively.

Edit: Just realized, it seems to make navigation from multiple inputs work, e.g. also letting the user scroll through touch events.

[inolen]

@ocornut first off, apologies for all of the issue spam, I'm on a death march to get an initial frontend UI going for https://github.com/inolen/redream.

I've been playing with this branch to try and get my UI to work as I'd like, and after hacking around a bit it works really well, but I'm not sure how to make my changes in a manageable fashion.

I've posted a few photos before, but to reiterate, the UI is just like Netflix, Steam Big Picture, etc.:

What I wanted to do here was flatten the parent / child navigation, while still enabling the child window to scroll when moving left / right starting from one of the items in the child window.

To do so, I hacked this up:
https://gist.github.com/inolen/b460ae06d05f55390c6dde4709888657

What I've done is hacked in the NavMoveResultWindow, and removed the g.NavWindow->ScrollbarX part of the conditional that enables the window to scroll.

I have no problem doing more work on this, I just need a bit of direction - I'm very motivated to get this work done.

[ocornut]

what was the reasoning for moving to the NavInputs array vs reusing the existing KeysDown / KeyMap setup?
Edit: Just realized, it seems to make navigation from multiple inputs work, e.g. also letting the user scroll through touch events.

Also, in my experiments I found that input logic has to be specialized for gamepad type vs keyboard type, there are specifying controls that makes sense for typical gamepad (holding Square + L/R doesn't work exactly the same as Ctrl+Tabbing would work). So for now note that those enums are named after the Gamepad. The branch is also marked Beta for this reason that not everything has been 100% designed.

What I wanted to do here was flatten the parent / child navigation, while still enabling the child window to scroll when moving left / right starting from one of the items in the child window.

I think you are hitting a few barriers that we'd need to lift and this is an interesting use case. Navigating between Games and Options I'd imagine the normal Nav system would be practical, but for your Dreamcast cover browsing you are probably better off making something custom.

Similarly to your issue #1328 I think you may be using a generic-over-engineered piece of feature (the Nav system) for something you could more easy recreate yourself, may be worth going that path short term. And you are not going to get the nice custom smooth scrolling that your apps wants using the immediate focusing?

At the least I think we need to support a way for user code to hook/disable certain Nav inputs so they can be processed by the code, so you could process Left/Right yourself. And we probably need an internal flag to signify "x scrolling" allowed that's independant of ScrollbarX.

PS: I'm just blurting those thoughs but haven't actually dug into your use case or code very much. Your gist/patch is useful for reference. I appreciate it is an interesting use case that I would like to have solved, I just don't have so much time to think about it right now.

[inolen]

@ocornut thanks for the comments!

I agree wrt using the nav system, using something hand-rolled probably makes the most sense giving its limited scope right now.

[inolen]

@ocornut noodling some this morning, how would a nav hook / callback work?

Would there be an IsItemNavigated function that could be called after adding each item. with some option to disable the automatic scrolling, etc.?

[ocornut]

I don't know yet. Part of the work is figuring out the design. It's in my list and there is a decent chance I'll be able to focus on the Navigation branch on the upcoming few months.