Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DolphinQt: Fix the panic alert deadlock, dual core edition #8763

Merged
merged 2 commits into from May 16, 2022

Conversation

JosJuice
Copy link
Member

The fix in PR #8715 worked for panic alerts from the CPU thread, but there were still problems with panic alerts from the GPU thread in dual core mode. This change attempts to fix those.

I have assumed that the reason we use RunAsCPUThread when calling SetFullscreen is because we want to wait for the GPU thread to be inactive. (Because it can't be the CPU thread itself that we're waiting for, right? The CPU thread shouldn't directly interact with g_renderer.) However, I'm not entirely sure about this. If my assumption was wrong, then this PR is probably not the best solution.

@JosJuice JosJuice changed the title DolphinQt: Fix the panic alert deadlock, GPU thread edition DolphinQt: Fix the panic alert deadlock, dual core edition Apr 25, 2020
@JosJuice JosJuice force-pushed the panic-alert-deadlock-gpu branch 2 times, most recently from aea039a to ce5e5be Compare September 15, 2020 16:11
@Pokechu22
Copy link
Contributor

This fix doesn't seem to be sufficient; I still get deadlocks with the dual-core FIFO player when stopping playback (though I don't have a way of consistently reproducing it; it just happens occasionally).

22:43:762 Core\Core.cpp:272 I[CONSOLE]: Stop [Main Thread]		---- Shutting down ----
22:43:762 Core\Core.cpp:275 I[CONSOLE]: Stop [Main Thread 4924]	Stop CPU
22:43:784 Common\MsgHandler.cpp:116 E[MASTER]: Question: FIFO is overflowed by GatherPipe !
CPU thread is too fast!

Note that this is with the changes from this PR cherry-picked onto a bunch of other FIFO-related changes made by me (mostly those from #9540, but there are a few others). I don't think those would reintroduce this deadlock, though (especially since I cherry-picked these commits atop my original changes without any merge conflicts).

Stacktrace

Main thread

Core::tls_is_gpu_threadfalse
Core::tls_is_cpu_threadfalse
 	ntdll.dll!NtWaitForAlertByThreadId�()	Unknown
 	ntdll.dll!RtlSleepConditionVariableSRW()	Unknown
 	KernelBase.dll!SleepConditionVariableSRW�()	Unknown
 	[Inline Frame] msvcp140.dll!Concurrency::details::stl_condition_variable_win7::wait_for(Concurrency::details::stl_critical_section_interface *) Line 167	C++
 	msvcp140.dll!Concurrency::details::stl_condition_variable_win7::wait(Concurrency::details::stl_critical_section_interface * lock) Line 161	C++
 	msvcp140.dll!_Cnd_wait(_Cnd_internal_imp_t * cond=0x00007ff6bf78be50, _Mtx_internal_imp_t * mtx=0x00007ff6bf78be00) Line 60	C++
>	[Inline Frame] Dolphin.exe!std::condition_variable::wait(std::unique_lock<std::mutex> &) Line 600	C++
 	Dolphin.exe!CPU::Stop() Line 184	C++
 	Dolphin.exe!Core::Stop() Line 278	C++
 	[Inline Frame] Dolphin.exe!MainWindow::ForceStop() Line 881	C++
 	Dolphin.exe!MainWindow::RequestStop() Line 874	C++
 	Qt5Core.dll!00007ffe1c469264()	Unknown
 	Qt5Core.dll!00007ffe1c467bf1()	Unknown
 	Qt5Widgets.dll!00007ffe1c8687c5()	Unknown
 	Qt5Widgets.dll!00007ffe1c844a80()	Unknown
 	Qt5Widgets.dll!00007ffe1c843a28()	Unknown
 	Qt5Core.dll!00007ffe1c4411da()	Unknown
 	Qt5Core.dll!00007ffe1c442f55()	Unknown
 	qwindows.dll!00007ffe2d540aef()	Unknown
 	Qt5Core.dll!00007ffe1c48a0fa()	Unknown
 	qwindows.dll!00007ffe2d540ac9()	Unknown
 	Qt5Core.dll!00007ffe1c43d64c()	Unknown
 	Qt5Core.dll!00007ffe1c4401a4()	Unknown
 	Dolphin.exe!wWinMain(HINSTANCE__ * hInstance=0x0000000000000000, HINSTANCE__ * hPrevInstance=0x0000000000000000, wchar_t * pCmdLine=0x000000000000000a, int nCmdShow=0x00000000) Line 295	C++
 	[Inline Frame] Dolphin.exe!invoke_main() Line 118	C++
 	Dolphin.exe!__scrt_common_main_seh() Line 288	C++
 	kernel32.dll!BaseThreadInitThunk�()	Unknown
 	ntdll.dll!RtlUserThreadStart�()	Unknown

Video Thread

Core::tls_is_gpu_threadtrue
Core::tls_is_cpu_threadfalse
>	[Inline Frame] Dolphin.exe!std::_Atomic_storage<bool,1>::compare_exchange_strong(bool &) Line 736	C++
 	[Inline Frame] Dolphin.exe!Common::Flag::TestAndSet(bool) Line 37	C++
 	[Inline Frame] Dolphin.exe!Common::Flag::TestAndClear() Line 40	C++
 	Dolphin.exe!Common::BlockingLoop::Run<`Fifo::RunGpuLoop'::`2'::<lambda_1>>(Fifo::RunGpuLoop::__l2::<lambda_1> payload={...}, __int64 timeout=0x0000000000000064) Line 165	C++
 	Dolphin.exe!Fifo::RunGpuLoop() Line 393	C++
 	Dolphin.exe!Core::EmuThread(std::unique_ptr<BootParameters,std::default_delete<BootParameters>> boot={...}, WindowSystemInfo wsi) Line 635	C++
 	[Inline Frame] Dolphin.exe!std::invoke(void(*)(std::unique_ptr<BootParameters,std::default_delete<BootParameters>>, WindowSystemInfo) &&) Line 1596	C++
 	Dolphin.exe!std::thread::_Invoke<std::tuple<void (__cdecl*)(std::unique_ptr<BootParameters,std::default_delete<BootParameters>>,WindowSystemInfo),std::unique_ptr<BootParameters,std::default_delete<BootParameters>>,WindowSystemInfo>,0,1,2>(void * _RawVals=0x000002aacbc9e190) Line 55	C++
 	ucrtbase.dll!thread_start<unsigned int (__cdecl*)(void *),1>()	Unknown
 	kernel32.dll!BaseThreadInitThunk�()	Unknown
 	ntdll.dll!RtlUserThreadStart�()	Unknown

FIFO player thread

Core::tls_is_gpu_threadfalse
Core::tls_is_cpu_threadtrue
 	ntdll.dll!NtWaitForAlertByThreadId�()	Unknown
 	ntdll.dll!RtlSleepConditionVariableSRW()	Unknown
 	KernelBase.dll!SleepConditionVariableSRW�()	Unknown
 	[Inline Frame] msvcp140.dll!Concurrency::details::stl_condition_variable_win7::wait_for(Concurrency::details::stl_critical_section_interface *) Line 167	C++
 	msvcp140.dll!Concurrency::details::stl_condition_variable_win7::wait(Concurrency::details::stl_critical_section_interface * lock) Line 161	C++
 	msvcp140.dll!_Cnd_wait(_Cnd_internal_imp_t * cond=0x000000b9570fec38, _Mtx_internal_imp_t * mtx=0x000000b9570fec80) Line 60	C++
>	[Inline Frame] Dolphin.exe!std::condition_variable::wait(std::unique_lock<std::mutex> &) Line 600	C++
 	[Inline Frame] Dolphin.exe!std::condition_variable::wait(std::unique_lock<std::mutex> &) Line 606	C++
 	[Inline Frame] Dolphin.exe!Common::Event::Wait() Line 56	C++
 	[Inline Frame] Dolphin.exe!RunOnObject(QObject *) Line 69	C++
 	Dolphin.exe!QtMsgAlertHandler(const char * caption=0x00007ff6bf78bcc8, const char * text=0x000000b9570fed80, bool yes_no=true, Common::MsgType style=Warning) Line 47	C++
 	Dolphin.exe!Common::MsgAlert(bool yes_no, Common::MsgType style=Warning, const char * format=0x00007ff6bee92328, ...) Line 126	C++
 	Dolphin.exe!CommandProcessor::GatherPipeBursted() Line 347	C++
 	Dolphin.exe!GPFifo::UpdateGatherPipe() Line 93	C++
 	[Inline Frame] Dolphin.exe!GPFifo::CheckGatherPipe() Line 113	C++
 	Dolphin.exe!GPFifo::Write8(unsigned char value) Line 123	C++
 	Dolphin.exe!FifoPlayer::WriteFifo(const unsigned char * data=0x000002aaf3b06060, unsigned int start, unsigned int end=0x00123ad8) Line 406	C++
 	Dolphin.exe!FifoPlayer::WriteFramePart(unsigned int dataStart=0x00000000, unsigned int dataEnd=0x00123ad8, unsigned int & nextMemUpdate=0x00000000, const FifoFrameInfo & frame={...}, const AnalyzedFrameInfo & info={...}) Line 354	C++
 	Dolphin.exe!FifoPlayer::WriteFrame(const FifoFrameInfo & frame={...}, const AnalyzedFrameInfo & info={...}) Line 297	C++
 	[Inline Frame] Dolphin.exe!FifoPlayer::AdvanceFrame() Line 154	C++
 	Dolphin.exe!FifoPlayer::CPUCore::Run() Line 110	C++
 	Dolphin.exe!PowerPC::RunLoop() Line 368	C++
 	Dolphin.exe!CPU::Run() Line 122	C++
 	Dolphin.exe!Core::FifoPlayerThread(const std::optional<std::string> & savestate_path, bool delete_savestate=false) Line 409	C++
 	[Inline Frame] Dolphin.exe!std::invoke(void(*)(const std::optional<std::string> &, bool) &&) Line 1596	C++
 	Dolphin.exe!std::thread::_Invoke<std::tuple<void (__cdecl*)(std::optional<std::string> const &,bool),std::optional<std::string>,bool>,0,1,2>(void * _RawVals=0x000002aacf285340) Line 55	C++
 	ucrtbase.dll!thread_start<unsigned int (__cdecl*)(void *),1>()	Unknown
 	kernel32.dll!BaseThreadInitThunk�()	Unknown
 	ntdll.dll!RtlUserThreadStart�()	Unknown

I don't actually see any window labeled "Question" at all; it only shows up in the log. Also, the panic alert is actually on the CPU thread, so maybe this is a different issue?

@JosJuice
Copy link
Member Author

This specific PR is only trying to solve issues when the panic alert is from the GPU thread. I had hoped that I had cleared out all the issues with panic alerts from the CPU thread in previous PRs, though...

@JosJuice
Copy link
Member Author

I suppose the problem you're having is that the main thread is blocking the Qt event loop (or something like it?) while trying to stop emulation, and so when the panic alert code comes and tries to synchronize with the main thread, it deadlocks. This is a bit different from the issues I've tried to solve in past PRs and this PR – I've been dealing with deadlocks when trying to synchronize with the CPU thread or GPU thread, but you're dealing with a deadlock when synchronizing with the main thread.

The fix in ef77872 worked for panic alerts from
the CPU thread, but there were still problems with
panic alerts from the GPU thread in dual core mode.
This change attempts to fix those.
If the purpose of calling SetFullscreen using RunAsCPUThread is
to make sure that the GPU thread is paused, the fix in ef77872
is faulty when dual core is used and a panic alert comes from
the CPU thread. This change re-adds synchronization for that case.
@dolphin-emu-bot
Copy link
Contributor

FifoCI detected that this change impacts graphical rendering. Here are the behavior differences detected by the system:

  • DKCR-Char on mvk-osx-m1: diff
  • ea-pink on mvk-osx-m1: diff
  • inverted-depth-range on mvk-osx-m1: diff
  • rs3-bumpmapping on mvk-osx-m1: diff
  • spyro-depth on mvk-osx-m1: diff

automated-fifoci-reporter

@PatrickFerry
Copy link
Contributor

PatrickFerry commented Jan 28, 2022

I tested this and it resolves the issue when using dual core and fullscreen and a panic alert appears.

It worked already on OpenGL for me but this PR also makes it work on the other hardware backends too.

@dreamsyntax
Copy link
Contributor

Bumping this, also confirming solves some of my issues. Would be good to get this in asap.

@@ -85,6 +88,44 @@ void Host::SetMainWindowHandle(void* handle)
m_main_window_handle = handle;
}

static void RunWithGPUThreadInactive(std::function<void()> f)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe this could be a template to avoid the overhead of the std function:

template <typename Func>
static void RunWithGPUThreadInactive(Func&& f)
{
  if (Core::IsGPUThread())
  {
    f();
  }
  else if (Core::IsCPUThread())
  {
    const bool was_running = Core::GetState() == Core::State::Running;
    Fifo::PauseAndLock(true, was_running);
    f();
    Fifo::PauseAndLock(false, was_running);
}
else
{
  Core::RunAsCPUThread(std::forward(f));
}
}

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The most common case is the RunAsCPUThread case, in which f has to be turned into an std::function anyway. Would this be beneficial despite that?

Copy link
Contributor

@iwubcode iwubcode Apr 16, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah I missed that. Probably not then. We may want to consider updating RunAsCPUThread in the future (if it's even possible), as there is some overhead for std::function

@AdmiralCurtiss
Copy link
Contributor

This PR hurts because all it really tells me is that our threading concept is poorly designed. =/

Code looks sensible enough but I can't even pretend to understand all possible deadlock causes here. A few people claim this fixes their issues, so maybe we should just merge and hope for the best?

@JosJuice
Copy link
Member Author

This PR hurts because all it really tells me is that our threading concept is poorly designed. =/

Yeah, the whole concept of having the core threads wait for the user to click OK on a panic alert is something that inherently carries a risk of deadlocks, unfortunately. I can't think of any way to completely avoid that without making it so that execution of the core threads at least sometimes continues without the user acknowledging the panic alert.

@AdmiralCurtiss
Copy link
Contributor

It's been a month since the last comment and I don't think anything changed, I'm just gonna merge this...

@AdmiralCurtiss AdmiralCurtiss merged commit b10808d into dolphin-emu:master May 16, 2022
10 checks passed
@JosJuice JosJuice deleted the panic-alert-deadlock-gpu branch June 6, 2022 07:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
7 participants