Input routing / Fine-grained input usage reporting

[avoroshilov]

Hello!
I am not sure yet if it is a feature request or my misunderstanding of the intended usage of the dear imgui input system.

I am on version 1.70 WIP, docking branch - sorry, I failed to find answers in the FAQ/demowindow code, and neither I've seen similar things in the issues (although it somewhat relates to shortcuts API #456, but probably more broad). The problem I am looking at is using dear imgui in the complex input scenarios, including allowing windows (and child windows) to have their own hotkeys and allowing keyboard navigation and whatnot. But the problem I've encountered is that imgui only provides WantCaptureKeyboard without specifying which inputs did it process.

To illustrate the usage scenario: there is a window with background global hotkeys (for example, WASD for camera, or some other state switches if viewport is in a subwindow), and there are children windows which are more like "subspaces" - i.e. each have associated logic which have the associated set of hotkeys (for example when Subwindow1 is active, hotkey N shows debug normals; and when Subwindow2 is active, M spawns mesh, and N does nothing, or something other from when Subwindow1 is active). Theoretically this also can have multiple nested levels. These subwindows naturally can contain elements like input boxes, checkboxes and other elements which may be modified or navigated between using keyboard too.

Couple of straightforward ways of doing this that I can see would be a) prioritized input events subscription queue, which can consume input events if required (and block further propagation), and build this queue based on the activity of the elements - this effectively means running imgui first to fill the state or one frame lag; and b) polling the event queue when drawing elements to see if the element was active this frame and consuming/rerouting the input to the next frame - necessarily frame lag, and possible problems with the io internal buffers flushing and guarding from events going to another element - unless adding function to determine whether the element to be drawn is active is not considered a bad practice.
The way b is more imgui-friendly, but both approaches seem to share common set of problems.

What's more important, there is no way to tell which input exactly was consumed by the imgui, which can matter in cases with input box (wants to consume any alphanumeric and some more) vs check box (wants to consume spacebar/enter to change state) - or if that was simple navigational event (some subwindows/containers may block navigational events to use hotkeys for their own purpose - but that can probably be achieved using io.NavActive to some extent).

Thus my question is what is recommended in cases like that when using dear imgui, and maybe if that's not what currently easily achievable, we need to devise a minimum set of features (requests) required to implement this.

Thank you!

Edit: added comment about active ids.

[ocornut]

Apologies for not answering this earlier. This is an important topic and I will want to address this seriously at some point (i'm just a little overwhelmed at the moment!), but various teams have raised similar questions and I'll come in the agenda. Any other feedback/suggestions welcome in the meanwhile!

[ocornut]

I've been looking at this lately along with other scenarios of input routing, and been working on a system of tracking key ownership (in fact the need to unify some input enums which led to 1.87 came from this). So hopefully I'll come back with some ideas.

I realize this is not the right/ideal answer but the issues presented are most often mitigated or worked around by gating input checks behind certain states e.g. IsWindowFocused(), IsWindowHovered() with varying flags (including child windows etc.). Just wanting to put that information out there just in case people are wondering.

[ocornut]

Trying to converge into a decent solution for this.

• It's not using an explicit priority value because they tend to require different code location to be aware of each others
• Rather it is based on the principle that last requests win, so it is expected that recursive window would test for shortcuts before submitting child window.
• The "one frame lag" only affect the moment where a given scope/source can accept a shortcut, e.g. a new child window will only catch shortcuts from its second frame. But actual shortcut presses have no lag.

// - This eventually get wrapped/simplified into a helper API.
// - Owner id may be implicit (based on e.g. current id stack position) or explicit (arbitrary).
// - Need to decide how to handle shortcut translations for Non-Mac <> sMac
auto IsShortcutPressed = [](ImGuiModFlags mods, ImGuiKey key, ImGuiID owner_id = 0)
{
    if (ImGui::GetIO().KeyMods != mods)
        return false;
    if (owner_id == 0)
        owner_id = ImGui::GetID("");
    if (!ImGui::TestInputOwner(key, owner_id))
        return false;
    ImGui::SetInputOwner(key, owner_id);
    return ImGui::IsKeyDown(key);// , owner_id);
};

{
    ImGui::Begin("Input Routing 4");
    {
        ImGui::Text("Window 1 (focused=%d)\n(want 1+2+4. catch input without focus)", ImGui::IsWindowFocused());
        bool owner_1_shortcuts[5] = {};
        if (true) // ImGui::IsWindowFocused(ImGuiHoveredFlags_ChildWindows)
        {
            owner_1_shortcuts[1] = IsShortcutPressed(ImGuiModFlags_Ctrl, ImGuiKey_1);
            owner_1_shortcuts[2] = IsShortcutPressed(ImGuiModFlags_Ctrl, ImGuiKey_2);
            owner_1_shortcuts[4] = IsShortcutPressed(ImGuiModFlags_Ctrl, ImGuiKey_4);
        }
        ImGui::Text("1 %s", owner_1_shortcuts[1] ? "Down!" : "...");
        ImGui::Text("2 %s", owner_1_shortcuts[2] ? "Down!" : "...");
        ImGui::TextDisabled("3 %s", owner_1_shortcuts[3] ? "Down!" : "...");
        ImGui::Text("4 %s", owner_1_shortcuts[4] ? "Down!" : "...");
    }

    ImGui::BeginChild("Window2", ImVec2(300, 330), true);
    {
        ImGui::Text("Window 2 (focused=%d)\n(want 1+3+4)", ImGui::IsWindowFocused());
        //ImGuiID owner_2 = ImGui::GetID("focusscope2"); // Could be WindowID or any ID
        bool owner_2_shortcuts[5] = {};
        if (ImGui::IsWindowFocused(ImGuiHoveredFlags_ChildWindows))
        {
            owner_2_shortcuts[1] = IsShortcutPressed(ImGuiModFlags_Ctrl, ImGuiKey_1);//, owner_2);
            owner_2_shortcuts[3] = IsShortcutPressed(ImGuiModFlags_Ctrl, ImGuiKey_3);//, owner_2);
            owner_2_shortcuts[4] = IsShortcutPressed(ImGuiModFlags_Ctrl, ImGuiKey_4);//, owner_2);
        }
        ImGui::Text("1 %s", owner_2_shortcuts[1] ? "Down!" : "...");
        ImGui::TextDisabled("2 %s", owner_2_shortcuts[2] ? "Down!" : "...");
        ImGui::Text("3 %s", owner_2_shortcuts[3] ? "Down!" : "...");
        ImGui::Text("4 %s", owner_2_shortcuts[4] ? "Down!" : "...");
    }

    ImGui::BeginChild("Window3", ImVec2(200, 200), true);
    {
        ImGui::Text("Window 3 (focused=%d)\n(want 2+3)", ImGui::IsWindowFocused());

        //ImGuiID owner_3 = ImGui::GetID("focusscope3"); // Could be WindowID or any ID
        bool owner_3_shortcuts[5] = {};
        if (ImGui::IsWindowFocused(ImGuiHoveredFlags_ChildWindows))
        {
            owner_3_shortcuts[2] = IsShortcutPressed(ImGuiModFlags_Ctrl, ImGuiKey_2);//, owner_3);
            owner_3_shortcuts[3] = IsShortcutPressed(ImGuiModFlags_Ctrl, ImGuiKey_3);//, owner_3);
            owner_3_shortcuts[4] = IsShortcutPressed(ImGuiModFlags_Ctrl, ImGuiKey_4);//, owner_3);
        }
        ImGui::TextDisabled("1 %s", owner_3_shortcuts[1] ? "Pressed!" : "...");
        ImGui::Text("2 %s", owner_3_shortcuts[2] ? "Down!" : "...");
        ImGui::Text("3 %s", owner_3_shortcuts[3] ? "Down!" : "...");
        ImGui::Text("4 %s", owner_3_shortcuts[4] ? "Down!" : "...");
    }

    ImGui::EndChild();
    ImGui::EndChild();
    ImGui::End();
}

[ocornut]

There is a rather tricky flaw/issue in the solution I have been working on, but I'm still very much exploring this.

[sigmareaver]

Just wondering here, but should IsKeyDown be IsKeyPressed instead? I'm just thinking in most cases, people want shortcuts to trigger once per press.

The menubar item has that built-in shortcut string, but I assume that's just cosmetic. I think an interesting possibility would be the registration of key bindings to IDs. Something like this in setup/initialization, and shutdown:

ImGui::AddShortcut("MySaveButtonID", ImGuiKeyMod_Ctrl, ImGuiKey_S);
ImGui::RemoveShortcut"MySaveButtonID");

And let ImGui internally check if a widget needs to be triggered (e.g. has focus, io.WantCaptureKeyboard is true, so return true if shortcut pressed).

Edit:
I realize now that the shortcut code doesn't trigger if the menubar menu isn't actually visibly open. I suppose that does pose a problem.
I need some simple shortcut processing for a text editor, and decided to modify your lambda. This works well enough for me at present. I'm just including it towards the end of my ImGui window.

bool IsShortcutPressed(ImGuiKeyModFlags mods, ImGuiKey key)
{
    if (ImGui::GetIO().KeyMods != mods)
        return false;
    if (!ImGui::IsWindowFocused(ImGuiFocusedFlags_RootAndChildWindows))
        return false;
    return ImGui::IsKeyPressed(key, false);
};

if (IsShortcutPressed(ImGuiKeyModFlags_Ctrl, ImGuiKey_S))
    Save();

[ocornut]

Just wondering here, but should IsKeyDown be IsKeyPressed instead?

Yes sorry, that was a leftover of me tweaking things for debugging.

The menubar item has that built-in shortcut string, but I assume that's just cosmetic.

It will become not just cosmetic via the addition of an opt-in flag to BeginMenuBar/BeginMenu to actually enable interpret those shortcuts. When using this flag menu code can eventually run while menu is hidden.

I think an interesting possibility would be the registration of key bindings to IDs

You are described a retained API which will be possible to add on end-user side but is something we would try to avoid in imgui core. The equivalent to your idea of registering to activate a "remote" item would be something like:

if (ImGui::IsShortcutPressed(ImGuiKeyMod_Ctrl, ImGuiKey_S))
    ImGui::ActivateItem("MySaveButtonID");

Which can also be wrapped as one-liner, and wouldn't require a registration.

[ocornut]

FYI today pushed several commits related to this.
Everything is currently in imgui_internal.h and expected to be moved to public API by 1.90.
TL;DR; there's a large amount of work on key ownership + shortcut/input routing.

The key features answering the things raised by this specific issue is the shortcut routing system.

// [EXPERIMENTAL] Shortcut Routing
// - ImGuiKeyChord = a ImGuiKey optionally OR-red with ImGuiMod_Alt/ImGuiMod_Ctrl/ImGuiMod_Shift/ImGuiMod_Super.
//     ImGuiKey_C                 (accepted by functions taking ImGuiKey or ImGuiKeyChord)
//     ImGuiKey_C | ImGuiMod_Ctrl (accepted by functions taking ImGuiKeyChord)
//   ONLY ImGuiMod_XXX values are legal to 'OR' with an ImGuiKey. You CANNOT 'OR' two ImGuiKey values.
// - When using one of the routing flags (e.g. ImGuiInputFlags_RouteFocused): routes requested ahead of time given a chord (key + modifiers) and a routing policy.
// - Routes are resolved during NewFrame(): if keyboard modifiers are matching current ones: SetKeyOwner() is called + route is granted for the frame.
// - Route is granted to a single owner. When multiple requests are made we have policies to select the winning route.
// - Multiple read sites may use a same owner and will all get the granted route.
// - For routing: when owner_id is 0 we use the current Focus Scope ID as a default owner in order to identify our location.
bool                 Shortcut(ImGuiKeyChord key_chord, ImGuiID owner_id = 0, ImGuiInputFlags flags = 0);
bool                 SetShortcutRouting(ImGuiKeyChord key_chord, ImGuiID owner_id = 0, ImGuiInputFlags flags = 0);
bool                 TestShortcutRouting(ImGuiKeyChord key_chord, ImGuiID owner_id);
ImGuiKeyRoutingData* GetShortcutRoutingData(ImGuiKeyChord key_chord);

// Routing policies for Shortcut() + low-level SetShortcutRouting()
// - The general idea is that several callers register interest in a shortcut, and only one owner gets it.
// - When a policy (other than _RouteAlways) is set, Shortcut() will register itself with SetShortcutRouting(),
//   allowing the system to decide where to route the input among other route-aware calls.
// - Shortcut() uses ImGuiInputFlags_RouteFocused by default: meaning that a simple Shortcut() poll
//   will register a route and only succeed when parent window is in the focus stack and if no-one
//   with a higher priority is claiming the shortcut.
// - Using ImGuiInputFlags_RouteAlways is roughly equivalent to doing e.g. IsKeyPressed(key) + testing mods.
// - Priorities: GlobalHigh > Focused (when owner is active item) > Global > Focused (when focused window) > GlobalLow.
// - Can select only 1 policy among all available.
ImGuiInputFlags_RouteFocused        = 1 << 8,   // (Default) Register focused route: Accept inputs if window is in focus stack. Deep-most focused window takes inputs. ActiveId takes inputs over deep-most focused window.
ImGuiInputFlags_RouteGlobalLow      = 1 << 9,   // Register route globally (lowest priority: unless a focused window or active item registered the route) -> recommended Global priority.
ImGuiInputFlags_RouteGlobal         = 1 << 10,  // Register route globally (medium priority: unless an active item registered the route, e.g. CTRL+A registered by InputText).
ImGuiInputFlags_RouteGlobalHigh     = 1 << 11,  // Register route globally (highest priority: unlikely you need to use that: will interfere with every active items)
ImGuiInputFlags_RouteAlways         = 1 << 12,  // Do not register route, poll keys directly.
ImGuiInputFlags_RouteUnlessBgFocused= 1 << 13,  // Global routes will not be applied if underlying background/void is focused (== no Dear ImGui windows are focused). Useful for overlay applications.

I re-read original message from @avoroshilov and afaik this + input owner systems should solves most cases.
I'm closing this for now but expect specific issues to appear and we'll look into them (please open new issues).

Demos
Because the current api are under imgui_internal.h we currently cannot yet ship the Demo code in imgui_demo.cpp, but the features/demo_input_owner_and_routing branch has said demos over 3 commits.

See other message #456 (comment)