WPF Key Forwarding from WebView2 to Host application?

[RickStrahl]

Is your feature request related to a problem? Please describe.

I'm looking to use the Web Browser control as part of an editor application that uses ACE Editor (Markdown Monster). Currently the application uses the Web Browser control and while it works I'm starting to run into issues with not being able to update libraries due to IE 11 support.

I've spent some time exploring WebView2 and for the most part it looks like I can duplicate the complex interop functionality with it. However, one sticking point is key forwarding:

The Web Browser control allowed me to pass through keyboard handlers for menu, toolbar and other keyboard maps from within the editor (ie. a focused element in the browser) to WPF. Meaning while in the editor I can press say (alt-w-l) to close all documents which is a WPF top level menu option. There's no extra code to make this works in the old control as the keystrokes are forwarded. This worked fine in the Web Browser, but has doesn't in WebView2.

AFAIK the old IE Web Browser control handles all locally mapped DOM key events and if those are not handled inside of the browser, the keystrokes are passed up into the WPF host which then passes on to the appropriate WPF keyhandlers.

Without this feature, keymapping is going to be really difficult as essentially every possible key combination has to be monitored and handled both in the browser and in WPF and is made more complicated yet by custom keymappings and the fact that this application allows for addins to add UI controls of their own that the core application may not know about (ie. adding a custom menu choice with it's own menu mnemonic).

Is there any chance that key forwarding can be added to the WebView2 to behave in a similar fashion to the IE control?

Describe the solution you'd like and alternatives you've considered

I would like this to work the same as it does in the IE Web Browser control:

• Any client side handled keyboard events are processed in the browser
• Any key events that go unhandled are passed up to WPF

Demonstrate how it doesn't work

• Create a WPF form.
• Add WebView2 control
• Add a navigation textbox
• Add a button and set <AccessText>_Navigate</AccessText>
• Add handler to set browser.Source = new Uri(txtUrl.Text)
• Navigate to a form that takes focus in the browser
• Type in some other URL in the form
• Focus the browser in the input control
• Press Alt-N

nothing happens.

AB#29558592

[champnic]

Thanks for checking #112. I'm tracking this on our backlog separately. Anything we build for our core Win32 control will also be projected in some way for our .NET controls, so it's possible we'll come up with a solution that solves both of these issues at once, but for now we'll keep both issues. Thanks!

[RickStrahl]

Any word on this?

I've been looking to see if there's some way to work around this, and I have a partial solution of keyforwarding from the DOM but it's pretty unreliable. Having unhandled key forwarding 'just work' as it did with the Web Browser control would be a much better solution.

I can't imagine I'm the only one that needs to be able to build interactive solutions that involve user input inside of the WebView and that need to still be able to interact with the host form UI.

[champnic]

If your main use-case are accelerator keys, then we've exposed the ICoreWebView2::add_AcceleratorKeyPressed in the regular WPF KeyDown event of the WPF WebView2 control. It doesn't capture every keypress though.

The ask for forwarding all key events is not near the top of our backlog, so will probably be sometime later in 2021 before we implement it.

[RickStrahl]

For me specifically it's the accellerators (Ctrl/Alt/Win/Func combos) that are important. It's possible that the old control only forwarded special key combos - never checked since the only thing I cared about was the menu/command forwarding ones.

I assume in WPF it's this:

<wv2:WebView2 AccessKeyManager.AccessKeyPressed="webView_AccessKeyPressed" />

 private void webView_AccessKeyPressed(object sender, AccessKeyPressedEventArgs e)
        {
            Debug.WriteLine(e.Key.ToString());
        }

But this doesn't seem to actually ever fire. Whatever key combos I fire don't show up in the debug window (while in an edit box - or in this case in ACE editor).

[ukandrewc]

@RickStrahl Until this is resolved you can hook the keyboard directly using 'SetWindowsHookEx', etc.

[RickStrahl]

The issue isn't so much capturing the keyboard commands. There are ways to look at all keys. The problem is getting the 'right' keys forwarded to WPF to process. Figuring out what keys were already handled, and putting together 3 key accelerator pairs is not easy to do at the app level. I started down this path with some experimental code and I'm getting lots of edge cases and false positives to pass on due to multi-event tracking and timing you have to do to even think about capturing all the keys needed.

This is why the old WebBrowser control was so nice both in WPF and WinForms: It handled all that for you - only passed through what wasn't handled, and passed all those keys onto WPF in a natural way. It just worked.

For Interop between the browser and a top level host application I think this is a really important scenario and one that I would prefer not to have to implement (probably incorrectly) in every end-user application.

Not saying this because I don't want to get my hands dirty, but because I think getting this right is really difficult. It's a system level task that requires deep knowledge about how keys are handled and injected into the key buffers. IOW, manual implementation is very likely to get this wrong almost every time, leading to badly behaving Windows applications. I think nobody wants that including Microsoft... 😄

[champnic]

Sorry if my comment above wasn't clear. We are raising the corresponding KeyDown events for accelerators, so you should be able to do the following:

        private void webView_KeyDown(object sender, KeyEventArgs e)
        {
            if (e.Key == Key.P && (Keyboard.Modifiers & ModifierKeys.Control) > 0 && (Keyboard.Modifiers & ModifierKeys.Alt) > 0)
            {
                // Do something for Ctrl + Alt + P

                // Mark handled to cancel WebView2 handling
                e.Handled = true;
            }
        }

Let me know if that doesn't work for you, or you still have concerns about this approach. I've noticed a place in our code where we are missing hooking into the InputManager, which is potentially why AccessKeyManager isn't working here. I've opened a bug to follow-up on our side. Thanks!

[RickStrahl]

@champnic The problem is that it's not that easy as capturing a single key event. First the modifier combinations don't always come as both Alt-P down (for example) - some of it is Alt then W(indow) cLose for example. This doesn't capture in just the KeyDown event. You now have to capture key down track the alt key, then check for a period of time for other keys etc.

I say this because I started down this path to try to work around this - in my version I'm doing it inside of the DOM though to check for the modifier keys there (to avoid the JS -> .NET transition until needed) and only when state changes passing the keys to .NET. It works - sort of but even so it's still unreliable with some key combos.

What I'm getting at is that this is really not easy to implement at the app level. It's doable but to do it right takes a lot of effort. I'm suggesting that if it's possible to do this natively inside of the control and pass this forward in the same way to old controls did, that would avoid a lot of pain for a lot of developers. Any application that works with input inside of the Web Browser is likely going to need this functionality to truly integrate into a desktop application.

The other issue I see here with what you suggest is that intercepting every keystroke inside of .NET is likely going to cause a bit of overhead. I'm using a sophisticated editor (ACE Editor) in JavaScript and typing speed is rather important - I suspect sending every key event into .NET and checking for special keys is going to introduce some typing latency. Handled at the control (Win32 level) this is likely to cause much less overhead than the marshalled calls into .NET I suspect.

I think I've made my point here (I'm starting to repeat myself) - I get it if you don't want to implement this but I think there's a lot of value for a lot of developers in doing so even if it's kind of a hidden feature. When you need it, self-implementing is going to waste days of dev and testing to get it right.

[champnic]

Thanks for the added context @RickStrahl. Sounds like the base case of keyboard shortcuts/accelerators are working, but we're missing support for the mnemonics for menus (alt and then w, for example). I'll update our backlog scenario to reflect that and make it more clear. Does that capture the core of your ask?

[RickStrahl]

@champnic To be clear - I'm asking for key forwarding behavior similar to the old Web Browser control which forwards everything that isn't handled inside of the browser DOM to the shell. Take a look at how the old WB control works - there it 'just works'. Control shortcuts, menus etc. The only tricky ones are the multi-key mnemonics and key chords, but ultimately I'd like to see those just forwarded to the host if not handled in the DOM.

[mediabuff]

Webview2 needs to implement IKeyboardInputSink

https://docs.microsoft.com/en-us/dotnet/api/system.windows.interop.ikeyboardinputsink to interoperate with wpf

https://docs.microsoft.com/en-us/dotnet/desktop/wpf/advanced/windows-forms-and-wpf-interoperability-input-architecture?view=netframeworkdesktop-4.8

[RickStrahl]

@mediabuff Thanks for pointing out the specific APIs that handle this.

I really hope support for this makes it into the WebView2 control. The keyforwarding for me is a show stopper and prevents me from moving forward in the short term.

[mediabuff]

Absolutely! For many really world apps.

Also deep navigation integration - jointly with the hosting app on journal history - aka WFP/frame navigation is a must as well

[albahari]

Until 1.0.781, there was a nice workaround: you could handle the AcceleratorKeyPressed event, which exposes the WPARAM/LPARAM message properties, and then manufacture a windows message and send it to the main menu. Now that AcceleratorKeyPressed is no longer exposed, you must handle KeyDown, which doesn't expose the LPARAM property. If you know how to reconstruct the LPARAM from a KeyEventArgs, let me know.

[RickStrahl]

Ok, so here's an update on the key forwarding situation after a bunch more experimentation I think I found a solution to Menu and Toolbar shortcut forwarding issues.

After a bunch more experimentation I found that most 'special' keys are in fact forwarded into the host WPF form. Control keys and function keys seem to actually fire.

However alt keys that activate menus and shortcuts do not work and I think the reason for that is that they are getting sent but are getting ignored by the host Window because the window doesn't have focus. Because the window doesn't have focus the Alt key operation doesn't trigger the default Windows menu and shortcut activation functionality.

So it's possible to intercept the alt key via KeyDown on the WebBrowser control and then explicitly fire it into the WPF window after setting focus to it:

private void WebBrowser_KeyDown(object sender, System.Windows.Input.KeyEventArgs e)
{
    // Handle Alt-Key forward to form so menus work
    if (e.Key == System.Windows.Input.Key.LeftAlt)
    {
		Model.Window.Focus();
		
		// must be out of band
                Model.Window.InvokeAsync( () => SendKeys.SendWait("%"));

                e.Handled = true;
    }
}

Wrote up a blog post here:

WebView2: Forwarding Alt Keys to host WPF Window

I think this resolves the issue, but it still would be nice if this could happen automatically as it did for the old IE WebBrowser control so the extra key monitoring (which adds overhead especially for edit operations) isn't necessary.

[noseratio]

This keyboard issue is hindered by the fact that the WebView2 host's child window (created by HwndHost) and the actual hosted Chromium HTML renderer child window are residing in separate processes.

So, when the focus is inside a WebView2, the hosting WPF UI thread itself is focus-inactive, and the keyboard messages are posted to Chromium's own UI thread (in a separate process). Thus, the WPF menu accelerators and even some system hotkeys (e.g., Alt+Space for system menu) just don't work. The corresponding WM_KEYDOWN/WM_KEYUP/WM_SYSKEYDOWN/WM_SYSKEYUP messages simply don't reach the WPF thread's message queue.

As a workaround, I've played a bit with WH_KEYBOARD_LL, albeit without much success so far.

I'm able to reliably intercept all the desired keystrokes from the Chromium's thread (e.g., Alt+F for Menu/File). However, I can't automatically route them through the WPF hierarchy, without first setting the keyboard focus to the main window, i.e., Keyboard.Focus(Application.Current.MainWindow). This is similar to what @RickStrahl is doing above.

Some things I've tried, to get the keyboard events routed, while keeping the focus within the WebView2:

• Calling ComponentDispatcher.RaiseThreadMessage();
• Rising routed keyboard events with Application.Current.MainWindow.RaiseEvent();
• Calling IKeyboardInputSink.TranslateAccelerator().

As for IKeyboardInputSink.TranslateAccelerator, I believe, it's for reverse scenarios. I.e., when the container gives a hosted control a chance to handle a keystroke. That's conceptually close to IOleInPlaceActiveObject::TranslateAccelerator from the golden age of COM.

While the thing I'm looking for would be conceptually close to COM's IDocHostUIHandler::TranslateAccelerator (or IOleInPlaceFrame::TranslateAccelerator, or IOleControlSite::TranslateAccelerator). Basically, a control asking its container to process a specific keystroke.

Logically, IKeyboardInputSite would be the right place for this, but it doesn't have anything like TranslateAccelerator.

IDK, maybe WPF has a proper secret solution for this scenario, but I couldn't find one. I wish WebView2 was open-source, it might be easier to tackle this problem from that end.

Anyhow, I'd be reluctant to convert a WPF project to use WebView2 at this stage, until we could reliable use accelerator keys on native WPF controls. We shouldn't have to go through some sort of mumbo-jumbo workarounds for such a basic thing like using keyboard for UI navigation.

[szanto90balazs]

@champnic I read about all the keyboard event related issues and wondering are they on the roadmap to tackle them for the near future and do you have a sneak peak into what changes likely / unlikely coming that you can share with us? Many thanks 👍 I'm particularly interested in the scenarios, when the WebView2 is the main input sink, but any unhandled DOM key events are surfaced to the WPF world. Good example would be to let the HTML/JS document add event handlers for Ctlr+SomeChar override WPF events (Ctrl+Somechar input binding).

[champnic]

@szanto90balazs We are currently in the design phase of this work, but have a good direction forwards here. We are going to change the main input sink from the WebView2 out-of-proc HWND to be the in-proc app HWND, which should enable full support for keyboarding, mouse, etc. in WPF and Winforms :) It will require some additional changes to our controls (to make sure KeyDown isn't getting fired multiple times), but should fix a ton of issues and enable new capabilities.

[szanto90balazs]

@champnic Sounds wonderful, thanks for sharing, much appreciated.

[hakzhyena]

Ok, I am late to the party but I have the same requirement for WebView2 embedded in my MFC application. I have the handle to the parent window and I want parent window to take care of keyboard shortcuts. Please consider this @champnic .
Also, from the example in https://docs.microsoft.com/en-us/microsoft-edge/webview2/reference/win32/icorewebview2controller?view=webview2-1.0.1150.38 for add_AcceleratorKeyPressed, what is m_appWindow->GetAcceleratorKeyFunction() referring to?
Please give a similar workaround for C++ temporarily to handle keyboard shortcuts such as Ctrl + N, Ctrl + etc.

[RickStrahl]

@zhuhaichao518 I don't have any code to look at any longer with the WebBrowser, but I do remember distinctly that the WebBrowser would not forward keys that are handled internally.

This means:

• Internal commands that the browser consumes (ctrl-f/ctrl-q)
• Keys that are handled in JS code and not bubbled up once handled in JS code (cancelBubble)

You couldn't handle everything in .NET and that seems to be correct behavior to me. Handled keys should not pass to .NET.

This is limiting but if you really need that you can then use JavaScript to intercept those keys and forward them to .NET via private handling.

The scope of my original request is specifically to more advanced key combination - specifically the Alt-Menu sequences that trigger application menus, but essentially any unhandled keys should forward to .NET.

I've been able to work around everything except for the menu mnemonics. (Alt-W-C to close in my app). I have a hack that sort of works, but it's hideously complex (JS code + .NET code + managing key timings) and not reliable as it relies on checking keys in 6 different event handlers across JS and .NET.

[zhuhaichao518]

@RickStrahl Thanks for the information! This is quite useful for us on how to implement the new API to handle keys unhandled by browser. Hopefully this API will make the workaround easier.
I would like to mention that this API will be async as a result of chromium browser process architecture -- If you press A, B and when B is triggerred in KeyDown event, it might happen that A is still not triggered in the new API. This should also happen in your JS code + .NET code workaround (because JS code is also async with the main process).

[novac42]

@RickStrahl Regarding the new API to handle keys unhandled by browser we are working on, it will be a deferral interface to replace GetKeyState, but in our current design the keys supported are limited to SHIFT, CTRL, ALT, WIN and the toggle state of Numlock, Capslock, StrollLock. Would these keys be enough for your issue? Or you'd need to know the state of keys other than those?

[RickStrahl]

I'm not really sure what that means.

I don't need yet another event that tells me of some obscure key state. There are already plenty of those both in JavaScript inside of the browser, and in .NET. Keys in isolation aren't very useful.

What we are looking for here is key forwarding not capturing yet another event that you have to handle to check for specific keys. The point of this whole thread is that coordinating all those keystates into a coherent whole or at minimum letting keys pass through to the host that aren't handled inside of the browser (either by JavaScript or by the Browser shell).

Again, I point you at the old WebBrowser control and how it handles keys that aren't handled inside of the browser: they are passed through to the host, so complex mnemonics fire in WPF and the operating system without the control or the browser having to do anything.

Go look at how that works to get an idea what we're talking about.

[ow--]

Also please keep in mind things like capturing whether focus is currently in an editable DOM element, à la CEF's keyboard handling:

https://magpcss.org/ceforum/apidocs3/projects/(default)/_cef_key_event_t.html#focus_on_editable_field
https://magpcss.org/ceforum/apidocs3/projects/(default)/CefKeyboardHandler.html

[novac42]

@RickStrahl Sorry for the confusion - let me rephrase. A common usage of a sync key event API is use it with (GetKeyState function (winuser.h) - Win32 apps | Microsoft Learn) API to determine if another key(such as control, shift, a, b, c...) is also down when the key event is triggered. However, due to the browser process architecture, we will provide an async API to get keys unhandled by browser, and if you try to call GetKeyState inside this API in order to get the key state of another(target) key when the key is down, it is too late because it is possible that user press/release the target key during the async time. In order to solve this problem, we will provide a new interface to substitute to this interface. What we wonder is that do we need to provide all keys (a,b,c,...) in this substitution API, or just accelerator keys are sufficient(shift,control,win,alt)?  If there is not such use case, we prefer not to provide all keys because for us, we need to scan the whole keyboard and restore all key states if we have this use case.

[Eudritch]

@novac42 Handle all key and mouse events from Windows API PostMessage - What we need is a method for creating a browser app that can handle all mouse and keyboard input from Windows PostMessage even when the browser is not active. We want to be able to simulate these inputs as if they were being performed by a user. To accomplish this, we will forward all key and mouse events to the browser app, which will respond appropriately. This would allow to build various types of browser apps with unique features.

For example, you could use a stylus to trace over an image, and the browser app below or above with a passthrough state would capture the trace, which you could then save/share. This is just one example of what you could do with it, but the possibilities are limitless!

In simpler term: We need a way to control a browser app with mouse and keyboard, even when not in use. We'll send the input to the app and it will respond like a user.

[Eudritch]

I'm not sure how the browser would handle an out-of-bounds click, or a click emitted outside of the browser rectangle, but we essentially need a way to manage all keyboard and mouse events.

[ClosetBugSlayer]

@Eudritch the experimental feature will be released in Edge 109, which can be enabled with AdditionalBrowserArguments = "--enable-features=msWebView2BrowserHitTransparent". All keys will trigger OnKeyDown in this mode. The new API which provides key handlers after DOM is under code review.

The Edge 109 experimental feature gives me the pass-through access to open my main menu, but commands like Alt+Space and Alt+F4 do not make it through. They do get trapped by ICoreWebView2AcceleratorKeyPressedEventHandler.Invoke().

IKeyboardInputSink.TranslateAccelerator() is getting called on my HwndHost, but there is no way to forward the MSG into something like IOleInPlaceActiveObject.TranslateAccelerator() the way we could with IWebBrowser2. Therefore the WPF parent tab control is swallowing keys like the arrows and Home/End which is making it difficult to edit inside HTML forms.

[Optimierungswerfer]

The Edge 109 experimental feature gives me the pass-through access to open my main menu, but commands like Alt+Space and Alt+F4 do not make it through.

I can confirm this behavior in our C++/WinRT WebView2 control within an MFC host application.

[Optimierungswerfer]

Are there any updates on this? It is already not optimal that we have to rely on an undocumented browser flag in production ("msWebView2BrowserHitTransparent"). It would be nice if at least those Alt+Space, Alt+F4 Issues could be fixed.

[champnic]

We have an experimental feature for enabling it:
https://learn.microsoft.com/en-us/dotnet/api/microsoft.web.webview2.core.corewebview2controlleroptions.allowhostinputprocessing?view=webview2-dotnet-1.0.2730-prerelease

Alt+Space and Alt+F4 should be fixed in runtimes 128.0.2663.0+.

[drawbyperpetual]

I haven't been able to get msWebView2BrowserHitTransparent to work on WinUI 3. Has propagating the key events been implemented there? If not, will that be addressed with AllowHostInputProcessing?

[champnic]

@drawbyperpetual Sorry for the late reply - this only affects our WPF and Winforms controls.

[drawbyperpetual]

@champnic Thanks for the reply. So how can I receive key events on WinUI 3?

[Optimierungswerfer]

I noticed some odd behavior that happens when using either the msWebView2BrowserHitTransparent feature flag or the AllowHostInputProcessing API in the 1.0.2839-prerelease:
Open the Find dialog via Ctrl+F and then close it either by pressing Escape or clicking on the X.
After the dialog closed, the focus won't return to the host application as one would expect. Without the feature flag or the AllowHostInputProcessing setting, the focus returns to the host application as expected.

I am able to reproduce this in the WebView2APISample project from the WebView2Samples repo by adding the msWebView2BrowserHitTransparent feature flag in the //! [CreateCoreWebView2EnvironmentWithOptions] section of AppWindow.cpp.

[RickStrahl]

I haven't tried this, but isn't that what you'd want? If I CTRL-F in the browser, then exit, I'd want focus back in the browser where I started the search from?

[Optimierungswerfer]

As you say, we'd want the focus back in the WebView2 ("browser") in our application, but that is not what is happening. Only god knows where the focus goes in that case. Certainly not back to our application.

The search box is its own window, which is completely separate from our application and the web view (e.g. also does not get moved along unless explicitly notified using NotifyParentPositionChanged). When opening it, the host application window loses focus in favor of the search box window. When closing the search box window, the focus does not return to the host application/browser in the host application. It appears that the focus is nowhere. But that is only when using msWebView2BrowserHitTransparent or AllowHostInputProcessing. If we don't do that, the focus does get returned to the host application when closing the search box window.

You can see for yourself in the sample application as I described. It helps tremendously to turn on the Windows setting "Show accent color on title bars and window borders" in debugging this focus issue as you can clearly see, which window is focused.