Keyboard input processing

[maraf]

I'm building an app that should work in a similar way as Intellisense in Visual Studio, but Windows wide. If global hotkey is pressed, I'm using windows hook to intercept input to foreground window and forward it my WPF app.

What I'm looking for, is there a way to forward input into WPF textbox (that is not focused) and let it process it? I'm able to manually append characters to the textbox, but I would like a solution that will be processing other events, like "deleting before cursor", "deleting after cursor", "jumping cursor over words when Ctrl+arrow is pressed", etc.

Is this textbox logic implemented in WPF or is it handled by underlaying windows thing? Is there a way to directly use that logic?

[miloush]

If you are trying to create any sort of global input experience, you should write an IME, that's what they are designed for; setting global input hooks is generally frown upon. Some helpful links:

• https://learn.microsoft.com/en-us/windows/apps/design/input/input-method-editors
• https://github.com/microsoft/Windows-classic-samples/tree/main/Samples/IME

Looks like you can also reach out to people internally if you needed help with that.

But to answer your question, the window messages are processed by WPF and delivered to the TextBox control through routed events.

You can inject input into WPF by calling InputManager.ProcessInput. However, for keyboard input, you would need to pass System.Windows.Input.RawKeyboardInputReport which is internal, but since you are happy with global hooks I imagine you are OK with reflection. If you are interested how that is usually generated from Windows messages, have a look at HwndKeyboardInputProvider.FilterMessage.

This will of course route the events as appropriate. If you want to route the directly to a specific control, you can technically raise the WPF events yourself directly on the controls, for example:

textBox.RaiseEvent(new KeyEventArgs(Keyboard.PrimaryDevice, 
    PresentationSource.FromVisual(this), timestamp, Key.C) { RoutedEvent = KeyDownEvent });

Note that WPF does way much work than that, dealing with routing, preview events etc.

More importantly, this will not give you any actual input to the TextBox, because it doesn't use KeyDown/Up events to type. It uses TextInput events (corresponding to RawTextInputReport) which are generated, non-trivially, by TextCompositionManager. The days when input was based on key presses is long gone, input can come from voice, ink, IMEs and other stuff. Someone also needs to be handling Alt+0169 kind of input, composing dead keys, dealing with selected input language (what if the TextBox has different input language/keyboard layout than the window you are stealing events from, or user changes it during typing?). You just don't want to be doing all of that.

Apps are free to do whatever they want with the input events you are intercepting, and they might turn it into input that you will not be aware of. I even know one or two "typing software" which, like you are intending, is using global hooks to turn key presses into different input (e.g. Indic), and I don't think you can guarantee any order in which the global hooks will be installed.
And then there is IMEs. Most Asian users would be using IME for typing - you press a few keys and then the input is commited into the text box - using Text Services Framework, not Windows messages you can intercept. You cannot have IME serving two different textboxes separately in parallel.

To summarize, WPF does all the processing, but in a global input processing pipeline, not in individual TextBoxes. Some of the processing has focus checks. So you either do the processing yourself and deliver it to your TextBox, or, you let WPF do the processing, but then have to somehow steal it and have it delivered where you need. I have some ideas how you could do that (using events on the InputManager), but I just don't think that's the route you should start drowning in.

Why don't you, instead, use UI automation API to get access to the text in the current input elements and use that?

[miloush]

Note: if you do decide to give UI automation a go, I would suggest, if possible, avoid the UIAutomation assembly in the framework. #7630

[maraf]

Thank you for ideas and resources!

I'm far from expert in this area. In last year's hackathon I went with global hotkey which opens almost standard window next to cursor and when user selects completion, it closes the window and uses winforns SendKeys to paste the completion back to previously selected input. In works in most typical scenarios I tested. But because of the focus change, it doesn't work in File Explorer location box for example.

That lead me to the idea with global hooks, but if you know about better approach, I'm all ears 🙂

I use UIAutomation to find the cursor location.

Ideally I would like to as much as possible mimic the experience with IntelliSense.

[miloush]

I see, so you have other issues to begin with :-)

1. Create your window with or set it to WS_EX_NOACTIVATE. If your app has other windows, also return MA_NOACTIVATE to WM_MOUSEACTIVATE. This will prevent it from getting focus when user clicks it.
2. Set Window.ShowActivated to false. This will prevent it from getting focus when you change its visibility/show it.
3. Not sure how you get the cursor location, but UIAutomation sounds a bit like overkill, there is simple GetCursorPos API.
4. Try to use the UIAutomation to get (and possibly track) the text in the control to populate your intellisense thingy.
5. Depending on your target audience and the content you have, you might get away with SendInput (used by SendKeys) to send the text to the target control. UIAutomation might be better in some cases.

[maraf]

1. Create your window with or set it to WS_EX_NOACTIVATE. If your app has other windows, also return MA_NOACTIVATE to WM_MOUSEACTIVATE.

Awesome 👍

2. Set Window.ShowActivated to false.

Already have this one

3. there is simple GetCursorPos API.

Sorry, caret, not cursor. I use either GetCaretPos or UIAutomation https://github.com/neptuo/Productivity.SnippetManager/blob/windowsHook/src/Neptuo.Productivity.SnippetManager.UI/Views/Controls/CaretPosition.cs#L17, because various apps provide various results to these methods. For example in Windows Terminal I had to do some additional trick microsoft/terminal#14664.

4. UIAutomation to get (and possibly track) the text in the control

Do you think of IUIAutomationTextPattern(2) and IUIAutomationEditTextPattern or do you know of anything "better"? For example Windows Terminal doesn't provide any text by these.

[miloush]

Ah consoles... sorry I don't know enough about those and nothing about the new terminal. There is ReadConsoleOutput APIs but I don't know if they are still (or ever were) of any use. IMEs/TSF seem to work in console though. Your best bet is their repo or try reaching out to Dustin Howett.