Client crashes coinciding with Edge 117 Rollout

[kevin--]

Description
Coinciding with the release of Edge 117 to the Stable Channel (September 15th), we are observing an increased error rate in our deployed customers, some are experiencing crashes with this or very similar callstack

Report Version: 104

Crashed Thread: 17392

Application Specific Information:
Fatal Error: EXCEPTION_ACCESS_VIOLATION_READ / 0x83

Thread 17392 Crashed:
0   EmbeddedBrowserWebView.dll      0x7ff94775d7c0      embedded_browser_webview::internal::WebViewFactory::EnsureConnection
1   EmbeddedBrowserWebView.dll      0x7ff9477fa659      ipcz::Message::Serialize
2   EmbeddedBrowserWebView.dll      0x7ff947acb725      _tailMerge_api-ms-win-shcore-scaling-l1-1-1.dll
3   EmbeddedBrowserWebView.dll      0x7ff9478642e0      base::sequence_manager::Task::task_order
4   EmbeddedBrowserWebView.dll      0x7ff947863598      base::sequence_manager::internal::WorkQueue::GetFrontTaskOrder
5   EmbeddedBrowserWebView.dll      0x7ff947868fde      base::sequence_manager::internal::WorkQueueSets::OnQueuesFrontTaskChanged
6   EmbeddedBrowserWebView.dll      0x7ff9477fd730      ipcz::msg::RouteClosed::RouteClosed
7   EmbeddedBrowserWebView.dll      0x7ff947818edf      base::subtle::TimeTicksNowIgnoringOverride
8   EmbeddedBrowserWebView.dll      0x7ff9477ef20d      [inlined] allocator_shim::internal::PartitionFree
9   EmbeddedBrowserWebView.dll      0x7ff9477ef20d      allocator_shim::internal::PartitionFree
10  EmbeddedBrowserWebView.dll      0x7ff94785eff4      absl::inlined_vector_internal::Storage<T>::DestroyContents
11  EmbeddedBrowserWebView.dll      0x7ff947922222      base::sequence_manager::Task::IsCanceled
12  EmbeddedBrowserWebView.dll      0x7ff947863ba7      base::sequence_manager::internal::WorkQueue::RemoveAllCanceledTasksFromFront
13  EmbeddedBrowserWebView.dll      0x7ff947ab8081      _tailMerge_api-ms-win-shcore-scaling-l1-1-1.dll
14  EmbeddedBrowserWebView.dll      0x7ff947989477      mojo::core::`anonymous namespace'::ChannelWin::ShutDownImpl
15  EmbeddedBrowserWebView.dll      0x7ff947acbcff      _tailMerge_api-ms-win-shcore-scaling-l1-1-1.dll
16  EmbeddedBrowserWebView.dll      0x7ff947acbcbd      _tailMerge_api-ms-win-shcore-scaling-l1-1-1.dll
17  EmbeddedBrowserWebView.dll      0x7ff947acbe71      _tailMerge_api-ms-win-shcore-scaling-l1-1-1.dll
18  EmbeddedBrowserWebView.dll      0x7ff947ab461f      _tailMerge_api-ms-win-shcore-scaling-l1-1-1.dll
19  EmbeddedBrowserWebView.dll      0x7ff947ab45ff      _tailMerge_api-ms-win-shcore-scaling-l1-1-1.dll
20  EmbeddedBrowserWebView.dll      0x7ff947ab45df      _tailMerge_api-ms-win-shcore-scaling-l1-1-1.dll
21  EmbeddedBrowserWebView.dll      0x7ff947ab4b28      _tailMerge_api-ms-win-shcore-scaling-l1-1-1.dll
22  EmbeddedBrowserWebView.dll      0x7ff947ab6fe7      _tailMerge_api-ms-win-shcore-scaling-l1-1-1.dll
23  EmbeddedBrowserWebView.dll      0x7ff9477ebeb2      malloc
24  EmbeddedBrowserWebView.dll      0x7ff947939b6e      new
25  EmbeddedBrowserWebView.dll      0x7ff947922620      base::sequence_manager::internal::TaskQueueImpl::NotifyWillProcessTask
26  EmbeddedBrowserWebView.dll      0x7ff947712e33      std::__Cr::__tree<T>::__construct_node<T>
27  EmbeddedBrowserWebView.dll      0x7ff947762313      std::__Cr::__tree<T>::__emplace_unique_key_args<T>
28  EmbeddedBrowserWebView.dll      0x7ff94771a7aa      embedded_browser_webview_current::EmbeddedBrowserWebView::CreateWebViewOnIOThread
29  EmbeddedBrowserWebView.dll      0x7ff94771a74f      embedded_browser_webview_current::EmbeddedBrowserWebView::RunOnIOThread
30  EmbeddedBrowserWebView.dll      0x7ff94773013b      base::internal::FunctorTraits<T>::Invoke<T>
31  EmbeddedBrowserWebView.dll      0x7ff94786438e      base::sequence_manager::internal::SequencedTaskSource::SelectedTask::SelectedTask
32  EmbeddedBrowserWebView.dll      0x7ff94771a74f      embedded_browser_webview_current::EmbeddedBrowserWebView::RunOnIOThread
33  EmbeddedBrowserWebView.dll      0x7ff94772ffda      base::internal::Invoker<T>::RunOnce
34  EmbeddedBrowserWebView.dll      0x7ff94771a74f      embedded_browser_webview_current::EmbeddedBrowserWebView::RunOnIOThread
35  EmbeddedBrowserWebView.dll      0x7ff947922848      base::TaskAnnotator::RunTaskImpl
36  EmbeddedBrowserWebView.dll      0x7ff947719fdd      embedded_browser_webview_current::EmbeddedBrowserWebView::RuntimeClassInitialize
37  EmbeddedBrowserWebView.dll      0x7ff9477e02d3      base::sequence_manager::internal::ThreadControllerWithMessagePumpImpl::DoWorkImpl
38  EmbeddedBrowserWebView.dll      0x7ff94781a67f      base::MessagePumpForIO::GetIOItem
39  EmbeddedBrowserWebView.dll      0x7ff9477cfb3c      base::`anonymous namespace'::QPCNow
40  EmbeddedBrowserWebView.dll      0x7ff94786bad6      base::MetadataRecorder::Remove
41  EmbeddedBrowserWebView.dll      0x7ff9478206e4      base::TimeTicks::Now
42  EmbeddedBrowserWebView.dll      0x7ff947b582bf      [inlined] _tailMerge_api-ms-win-shcore-scaling-l1-1-1.dll
43  EmbeddedBrowserWebView.dll      0x7ff947b582bf      [inlined] _tailMerge_api-ms-win-shcore-scaling-l1-1-1.dll
44  EmbeddedBrowserWebView.dll      0x7ff947b582bf      _tailMerge_api-ms-win-shcore-scaling-l1-1-1.dll
45  EmbeddedBrowserWebView.dll      0x7ff94785f5f9      base::sequence_manager::internal::ThreadControllerWithMessagePumpImpl::DoWork
46  EmbeddedBrowserWebView.dll      0x7ff9479a3ced      base::sequence_manager::internal::ThreadController::RunLevelTracker::RunLevel::RunLevel
47  EmbeddedBrowserWebView.dll      0x7ff947b582bf      _tailMerge_api-ms-win-shcore-scaling-l1-1-1.dll
48  EmbeddedBrowserWebView.dll      0x7ff94781a40f      base::MessagePumpForIO::DoRunLoop
49  EmbeddedBrowserWebView.dll      0x7ff9479260dc      base::MessagePumpWin::Run
50  EmbeddedBrowserWebView.dll      0x7ff947ab6fe7      _tailMerge_api-ms-win-shcore-scaling-l1-1-1.dll
51  EmbeddedBrowserWebView.dll      0x7ff947920fc3      base::sequence_manager::internal::ThreadControllerWithMessagePumpImpl::Run
52  EmbeddedBrowserWebView.dll      0x7ff947939b6e      new
53  EmbeddedBrowserWebView.dll      0x7ff947b582bf      _tailMerge_api-ms-win-shcore-scaling-l1-1-1.dll
54  EmbeddedBrowserWebView.dll      0x7ff9478be767      base::RunLoop::Run
55  EmbeddedBrowserWebView.dll      0x7ff947b4c4cf      _tailMerge_api-ms-win-shcore-scaling-l1-1-1.dll
56  EmbeddedBrowserWebView.dll      0x7ff947b2a440      _tailMerge_api-ms-win-shcore-scaling-l1-1-1.dll
57  combase.dll                     0x7ff9a109803f      CActivationStore::GetActivatableClassRegistration (activationstoreapi.cpp:90)
58  EmbeddedBrowserWebView.dll      0x7ff94791ab75      base::sequence_manager::internal::ThreadControllerPowerMonitor::BindToCurrentThread
59  EmbeddedBrowserWebView.dll      0x7ff9477ebeb2      malloc
60  KERNEL32.DLL                    0x7ff9a0192edf      GetVolumeNameForVolumeMountPointW
61  EmbeddedBrowserWebView.dll      0x7ff9478ba653      base::Thread::Run
62  EmbeddedBrowserWebView.dll      0x7ff947ab422a      _tailMerge_api-ms-win-shcore-scaling-l1-1-1.dll
63  EmbeddedBrowserWebView.dll      0x7ff947ac3178      _tailMerge_api-ms-win-shcore-scaling-l1-1-1.dll
64  EmbeddedBrowserWebView.dll      0x7ff947acaec9      _tailMerge_api-ms-win-shcore-scaling-l1-1-1.dll
65  EmbeddedBrowserWebView.dll      0x7ff9478ba648      base::Thread::Run
66  EmbeddedBrowserWebView.dll      0x7ff9478ba779      base::Thread::ThreadMain
67  EmbeddedBrowserWebView.dll      0x7ff9478afc40      base::`anonymous namespace'::ThreadFunc
68  KERNEL32.DLL                    0x7ff9a01853df      BaseThreadInitThunk
69  ntdll.dll                       0x7ff9a14e485a      RtlUserThreadStart

Version
SDK: 1.0.1210.39 / (with 1.0.992.28 WebView2Loader.dll our mistake)
Runtime: 117.0.2045.31
Framework: Win32
OS: Windows 10 -- no incidents seen on Windows 11

Regression
Was this working before but has regressed?
YES
If yes, what version did this last work on?
116 presumably

Repro Steps
Based on crash data it looks like this happens when our GUI is being instantiated, which is also when the WebView2 component would be created.

• Haven't been able to reproduce in house yet.
• no customer reports from the field

AB#46637401

[victorhuangwq]

this is the WebView2 repository - is this bug regarding the Microsoft Edge browser?

If it's directed at the Edge Browser, click on the three dots, click help and feedback, and then send feedback to file an issue with the Edge Browser team

[kevin--]

this is the WebView2 repository - is this bug regarding the Microsoft Edge browser?

If it's directed at the Edge Browser, click on the three dots, click help and feedback, and then send feedback to file an issue with the Edge Browser team

Indeed we have an application that uses WebView2. Our crash reporting tool started showing that our customers began experiencing crashes when Edge 117 was rolled out on September 15th. My understanding is that the evergreen runtime would consume this change.

Since it is our application crashing and it consumes WebView2 (and that is the crashing binary EmbeddedBrowserWebView.dll) I assumed it was a WebView2 issue. I can file in both if that makes sense?

[victorhuangwq]

I see, thank you for the clarification.

[fabiorocha]

Thanks for reporting @kevin--. We have an engineer looking at this with high priority.

[vbryh-msft]

@kevin-- could you share how many wv2 objects do you create and destroy and the timing of creation/deletion - is it simultaneously? Do you know if runtime crashes when you observe the issue?

[Zhattis4]

@kevin-- could you share how many wv2 objects do you create and destroy and the timing of creation/deletion - is it simultaneously? Do you know if runtime crashes when you observe the issue?

jumping in here as I work with @kevin--

We use wv2 in our audio plugin.

Per instance of our plugin, we often create 2 wv2 objects simultaneously, delete them both, then create them again.
This can happen many times and depending on user behavior, could happen quite rapidly.

Additionally, multiple instances of our plugin can run at once, so we may have say, 3 instances of our plugin, each with 2 instances of wv2 objects. Each instance of our plugin gui could be closed/opened at any time and with it the wv2 objects will be created/destroyed whenever our gui is closed/opened.

Do you know if runtime crashes when you observe the issue? - yes we are seeing crash reports occur in our plugin -- which instantiates 2 wbv2 simultaneously every time its run

[Zhattis4]

Hi there, we have seen an increase in crash reports from our users regarding this. Is there any update here regarding a fix? Many thanks,
Zev

[beninada]

@vbryh-msft @lflores-ms could you please provide an update on this issue? Thank you very much.

[lflores-ms]

We have a potential fix for next version 118.

This should only happen on webview creation after closing all webviews previously created with the same User Data Folder.
edit: hacky workarounds are not recommended as they could lead to other issues.

[beninada]

Thank you, @lflores-ms!

[kevin--]

We have seen a significant decrease in error rate as our users seem to be transitioning to 118 successfully.