WPF Key Forwarding from WebView2 to Host application?

[RickStrahl]

Is your feature request related to a problem? Please describe.

I'm looking to use the Web Browser control as part of an editor application that uses ACE Editor (Markdown Monster). Currently the application uses the Web Browser control and while it works I'm starting to run into issues with not being able to update libraries due to IE 11 support.

I've spent some time exploring WebView2 and for the most part it looks like I can duplicate the complex interop functionality with it. However, one sticking point is key forwarding:

The Web Browser control allowed me to pass through keyboard handlers for menu, toolbar and other keyboard maps from within the editor (ie. a focused element in the browser) to WPF. Meaning while in the editor I can press say (alt-w-l) to close all documents which is a WPF top level menu option. There's no extra code to make this works in the old control as the keystrokes are forwarded. This worked fine in the Web Browser, but has doesn't in WebView2.

AFAIK the old IE Web Browser control handles all locally mapped DOM key events and if those are not handled inside of the browser, the keystrokes are passed up into the WPF host which then passes on to the appropriate WPF keyhandlers.

Without this feature, keymapping is going to be really difficult as essentially every possible key combination has to be monitored and handled both in the browser and in WPF and is made more complicated yet by custom keymappings and the fact that this application allows for addins to add UI controls of their own that the core application may not know about (ie. adding a custom menu choice with it's own menu mnemonic).

Is there any chance that key forwarding can be added to the WebView2 to behave in a similar fashion to the IE control?

Describe the solution you'd like and alternatives you've considered

I would like this to work the same as it does in the IE Web Browser control:

• Any client side handled keyboard events are processed in the browser
• Any key events that go unhandled are passed up to WPF

Demonstrate how it doesn't work

• Create a WPF form.
• Add WebView2 control
• Add a navigation textbox
• Add a button and set <AccessText>_Navigate</AccessText>
• Add handler to set browser.Source = new Uri(txtUrl.Text)
• Navigate to a form that takes focus in the browser
• Type in some other URL in the form
• Focus the browser in the input control
• Press Alt-N

nothing happens.

AB#29558592

[champnic]

Thanks for checking #112. I'm tracking this on our backlog separately. Anything we build for our core Win32 control will also be projected in some way for our .NET controls, so it's possible we'll come up with a solution that solves both of these issues at once, but for now we'll keep both issues. Thanks!

[RickStrahl]

Any word on this?

I've been looking to see if there's some way to work around this, and I have a partial solution of keyforwarding from the DOM but it's pretty unreliable. Having unhandled key forwarding 'just work' as it did with the Web Browser control would be a much better solution.

I can't imagine I'm the only one that needs to be able to build interactive solutions that involve user input inside of the WebView and that need to still be able to interact with the host form UI.

[champnic]

If your main use-case are accelerator keys, then we've exposed the ICoreWebView2::add_AcceleratorKeyPressed in the regular WPF KeyDown event of the WPF WebView2 control. It doesn't capture every keypress though.

The ask for forwarding all key events is not near the top of our backlog, so will probably be sometime later in 2021 before we implement it.

[RickStrahl]

For me specifically it's the accellerators (Ctrl/Alt/Win/Func combos) that are important. It's possible that the old control only forwarded special key combos - never checked since the only thing I cared about was the menu/command forwarding ones.

I assume in WPF it's this:

<wv2:WebView2 AccessKeyManager.AccessKeyPressed="webView_AccessKeyPressed" />

 private void webView_AccessKeyPressed(object sender, AccessKeyPressedEventArgs e)
        {
            Debug.WriteLine(e.Key.ToString());
        }

But this doesn't seem to actually ever fire. Whatever key combos I fire don't show up in the debug window (while in an edit box - or in this case in ACE editor).

[ukandrewc]

@RickStrahl Until this is resolved you can hook the keyboard directly using 'SetWindowsHookEx', etc.

[RickStrahl]

The issue isn't so much capturing the keyboard commands. There are ways to look at all keys. The problem is getting the 'right' keys forwarded to WPF to process. Figuring out what keys were already handled, and putting together 3 key accelerator pairs is not easy to do at the app level. I started down this path with some experimental code and I'm getting lots of edge cases and false positives to pass on due to multi-event tracking and timing you have to do to even think about capturing all the keys needed.

This is why the old WebBrowser control was so nice both in WPF and WinForms: It handled all that for you - only passed through what wasn't handled, and passed all those keys onto WPF in a natural way. It just worked.

For Interop between the browser and a top level host application I think this is a really important scenario and one that I would prefer not to have to implement (probably incorrectly) in every end-user application.

Not saying this because I don't want to get my hands dirty, but because I think getting this right is really difficult. It's a system level task that requires deep knowledge about how keys are handled and injected into the key buffers. IOW, manual implementation is very likely to get this wrong almost every time, leading to badly behaving Windows applications. I think nobody wants that including Microsoft... 😄

[champnic]

Sorry if my comment above wasn't clear. We are raising the corresponding KeyDown events for accelerators, so you should be able to do the following:

        private void webView_KeyDown(object sender, KeyEventArgs e)
        {
            if (e.Key == Key.P && (Keyboard.Modifiers & ModifierKeys.Control) > 0 && (Keyboard.Modifiers & ModifierKeys.Alt) > 0)
            {
                // Do something for Ctrl + Alt + P

                // Mark handled to cancel WebView2 handling
                e.Handled = true;
            }
        }

Let me know if that doesn't work for you, or you still have concerns about this approach. I've noticed a place in our code where we are missing hooking into the InputManager, which is potentially why AccessKeyManager isn't working here. I've opened a bug to follow-up on our side. Thanks!

[RickStrahl]

@champnic The problem is that it's not that easy as capturing a single key event. First the modifier combinations don't always come as both Alt-P down (for example) - some of it is Alt then W(indow) cLose for example. This doesn't capture in just the KeyDown event. You now have to capture key down track the alt key, then check for a period of time for other keys etc.

I say this because I started down this path to try to work around this - in my version I'm doing it inside of the DOM though to check for the modifier keys there (to avoid the JS -> .NET transition until needed) and only when state changes passing the keys to .NET. It works - sort of but even so it's still unreliable with some key combos.

What I'm getting at is that this is really not easy to implement at the app level. It's doable but to do it right takes a lot of effort. I'm suggesting that if it's possible to do this natively inside of the control and pass this forward in the same way to old controls did, that would avoid a lot of pain for a lot of developers. Any application that works with input inside of the Web Browser is likely going to need this functionality to truly integrate into a desktop application.

The other issue I see here with what you suggest is that intercepting every keystroke inside of .NET is likely going to cause a bit of overhead. I'm using a sophisticated editor (ACE Editor) in JavaScript and typing speed is rather important - I suspect sending every key event into .NET and checking for special keys is going to introduce some typing latency. Handled at the control (Win32 level) this is likely to cause much less overhead than the marshalled calls into .NET I suspect.

I think I've made my point here (I'm starting to repeat myself) - I get it if you don't want to implement this but I think there's a lot of value for a lot of developers in doing so even if it's kind of a hidden feature. When you need it, self-implementing is going to waste days of dev and testing to get it right.

[champnic]

Thanks for the added context @RickStrahl. Sounds like the base case of keyboard shortcuts/accelerators are working, but we're missing support for the mnemonics for menus (alt and then w, for example). I'll update our backlog scenario to reflect that and make it more clear. Does that capture the core of your ask?

[RickStrahl]

@champnic To be clear - I'm asking for key forwarding behavior similar to the old Web Browser control which forwards everything that isn't handled inside of the browser DOM to the shell. Take a look at how the old WB control works - there it 'just works'. Control shortcuts, menus etc. The only tricky ones are the multi-key mnemonics and key chords, but ultimately I'd like to see those just forwarded to the host if not handled in the DOM.

[mediabuff]

Webview2 needs to implement IKeyboardInputSink

https://docs.microsoft.com/en-us/dotnet/api/system.windows.interop.ikeyboardinputsink to interoperate with wpf

https://docs.microsoft.com/en-us/dotnet/desktop/wpf/advanced/windows-forms-and-wpf-interoperability-input-architecture?view=netframeworkdesktop-4.8