Driver deadlock

[michael-fadely]

Describe the bug
With a virtual Xbox 360 controller enabled, programs that poll controller input will be unable to launch. They deadlock attempting to open a handle to the device. This happens with feeder applications as well.

It also prevents any real devices from connecting to or disconnecting from my system.

To Reproduce
I've been unable to consistently reproduce it. It happens fairly frequently, but with no specific steps that I could identify. If applications begin failing to launch with a feeder and Xbox 360 endpoint, however:

1. Launch an application that polls for controller input (a game, Uplay, Steam, Battle.net).
2. The application will stall or fail to launch.
3. Attempt to disable the device with Device Manager/etc, and it will also stall.
4. When the feeder is killed, the Xbox 360 endpoint device will enter an error state, displayed in Device Manager.

Expected behavior
Some kind of error handling or error logging. ViGEmClient has error logging, but it doesn't appear to encounter any errors when this deadlock occurs.

Screenshots

I attempted to disable the device, Device Manager stalled, and both the Game Controllers dialog and Battle.net immediately launched. Opening another instance of Device Manager reveals that the device is still enabled:

System details (please complete the following information):

• OS: Windows 10 Pro 1903
• Feeder software: ds4wizard
• Driver Version: 1.14.3.0

Additional context
It very well could be something I'm doing with my feeder, but I've been unable to figure it out.

[nefarius]

Thanks for the report but could you please monitor the situation with the latest driver version as well? What you run is very outdated.

[michael-fadely]

...Wow, that IS really old! I missed the latest stable release somehow. Will do!

[michael-fadely]

Same issue with latest stable. I will do some experimenting to see if I can narrow down the issue.

[MtDewFella]

I have been getting the same problem, wasn't sure if it was steam, glosc or vigem, I assume it's vigem since trying to disable it causes driver manager to freeze up, only fix I have found is to reboot the pc, oddly it never happens during play only after (not all the time only some times)

[nefarius]

Dammit 😅

[MtDewFella]

I completely uninstalled vigem and glosc, and reinstalled vigem first then glosc and so far I have had no issues, the last time I installed I just installed glosc like normal, which installed vigem (or was supposed to) but when I first used glosc that time I got the couldn't initialize vigem error, I then installed vigem without uninstalling anything, perhaps that caused the issue, I will keep and eye on it

Edit: I can also disable and enable vigem in device manager without issue now

[nefarius]

Any news @michael-fadely ?

[michael-fadely]

Still exhibiting the exact same behavior, unfortunately. Just having the virtual Xbox 360 device present when attempting to launch an application that uses XInput will sometimes cause this deadlock and require a hard reboot to allow any other physical devices to connect to, or disconnect from, the system.

[michael-fadely]

Could this be due to my program's multi-threading? I am using ViGEmClient. I call vigem_alloc and vigem_connect (as well as free and disconnect) on the main thread, but vigem_target_add, vigem_target_x360_register_notification, etc. are called on a secondary thread.

[michael-fadely]

In my program, everything appears to continue working. I simply unplug the DualShock 4 and it calls vigem_remove_target seemingly without error. I will implement some more robust error checking in this instance to be certain.

[Corrodias]

This seems very similar to a problem I'm experiencing. I'm using DS4Windows, on Windows 10 1909 and the latest version of the driver. After about 3 days of uptime, during which I leave a controller connected by USB at all times, Monster Hunter World will stall on its black startup screen and not proceed to the dev logo, but Special K's overlay is drawn, and Chrome can still start a new, GPU-accelerated video, so it's not a video driver failure. Attempting to "stop" and "start" DS4Windows may appear to work but results in a CPU-spinning thread. Attempting to reboot Windows causes it to hang on "Restarting" for several minutes, then BSOD.

I have not yet narrowed this down to being caused by ViGEmBus. It does seem like a likely culprit, especially after reading this thread. The next time this happens, I'll try disabling the device to see if it hangs, like it does for MtDewFella, and I'll try uninstalling it after the reboot. I will also attempt to get an xperf trace on the shutdown process that includes stacks.

If ViGEmBus does wind up being the culprit, is there anything else I can do to help diagnose this? It seems reliably reproducible for me; I just have to wait 3 days each time.

[Corrodias]

In fact, if it's going to do it again, today's the day. I have about 7 or 8 hours before I'm going to commence playing games, at which time I'll be doing the trace and so on if it did deadlock. If you have any advice or suggestions on helping diagnose this, providing them now would be timely.

[nefarius]

Make sure kernel memory dumps are enabled and keep the dump file somewhere. It shouldn't be shared publicly but is the best start to get the culprit.

[Corrodias]

Roger. Seems I'll have to reboot to enable that, so I'll get it ready for next time. I wouldn't get my hopes up, though; it didn't record the bugcheck event to the log the last time, despite the BSOD. (And it rebooted to BIOS setup instead of Windows for some reason; I was able to just exit with no changes and go right back to Windows.) I could only guess as to why. It wrote one of those "Dump file creation failed" events after booting up.

Since the last time it happened, I removed an old Logitech driver I had and replaced it with a newer one. If that was the actual culprit, I won't be seeing this situation again.

[Corrodias]

It happened again today, so I took some traces. I'll link them to you in an email.

I launched MHW, and it froze on its black startup screen. I ran a trace for maybe 30 seconds and produced "out.etl" and "kernel.etl".

Upon unplugging the controller, DS4Windows reported losing connection to it, and MHW suddenly continued as normal. I was also able to exit and launch it again normally. Plugging the controller back in, DS4Windows did not log any new messages, such as connecting to it.

Attempting to restart got stuck, as before. I took a trace during that restart, and I hit reset after it was clear that it wasn't going anywhere, rather than let it play out for 10 minutes. Unfortunately, xbootmgr was not able to assemble a finalized trace file once Windows started up; it claimed it couldn't load something from the registry. Therefore, you have "shutdown_BASE+DIAG+LATENCY+PROFILE+INTERRUPT+DPC_1_um_premerge.etl", and "shutdown_BASE+DIAG+LATENCY+PROFILE+INTERRUPT+DPC_1_km_premerge.etl".

There's also the less-inclusive "shutdown_BASE+DIAG+LATENCY_1.etl" that is from a previous attempt at a restart trace during this stuck condition.

If it happens again in a few days, I now have kernel memory dumping enabled, so I'll force a bugcheck in the hopes that'll actually dump it, following Microsoft's instructions for doing so.

edit: The "kernel.etl" file is actually related to the trace while MHW was running, not the restart trace. The comment has been corrected to reflect this.

[Corrodias]

The deadlock has not yet triggered again, but I'm still being vigilant and and ready to catch a memory dump when it does.

[Corrodias]

Damn, failure. It finally happened again, and I initiated a crash, but Windows once again failed to write a dump file due to an unspecified error. It looks like the only other option for getting a kernel memory dump is enabling kernel debugging mode and dumping it with windbg. I'll try to get that set up for next time.

[nefarius]

@Corrodias shame, but was it "random" or while a game accessed the controller? Because I was thinking about writing a small "fuzzing" application which may get us to the goal faster. Like rapid (dis-)connects, calling XInput APIs automated many many times etc. Might be with the try other than waiting and hoping 😅

[Corrodias]

I am hesitant to enable kernel debugging for several days at a time because I'm wary of a performance impact, and Monster Hunter: World is already a very demanding game. Having a way of reproducing it that doesn't require waiting upward of 3 days would be excellent. I'd be eager to try such an application and wouldn't have a reason to avoid the debugging mode then.

I only notice it happening right as the game's initializing, while the window is black and Special K has drawn its banner, but before it displays the splash screens, most likely during xinput initialization. That said, I can't guarantee that the truth isn't that the state is being triggered before that and the game is merely encountering it. I'm using DS4Windows with "exclusive mode" enabled, so the game cannot see the original DS4.

This may be important: my DS4 is actually connected through a third-party adapter called the Titan Two. That's connected to the PC by USB at all times, and I merely swap the controller between bluetooth and USB. It's supposed to perfectly emulate a DualShock 4 and by all appearances is successful in doing so. However, if it deviates in some small way, perhaps that could be triggering this issue? I could try connecting the controller directly by USB for a couple of weeks to see if that prevents the problem. However, if the Titan Two is misbehaving in some small way, I think it would benefit both its devs and you to understand how and why it's causing a problem, so one or both ends can mitigate it.

Thinking back, unfortunately, I see no pattern between things I do before a gaming session that works and things I do before one that doesn't. It's all the same stuff. I connect and disconnect my USB headset on occasion. I swap the controller between bluetooth and USB so it can charge.

edit: Of course it could be DS4Windows doing something wrong. Somewhere among the Titan Two, DS4Windows, and ViGEmBus, one of them is doing something wrong. Now, I haven't been running the very latest version of DS4Windows. The last few versions don't have anything the changelog that seem obviously related to this, but I can update it anyway. But I'd really like to find a method of reproducing the problem quickly, first, so I can tell whether it goes away just from doing that.

[Corrodias]

I have the XinputTest program from x360ce, so I can try opening and closing that 30 times under various conditions. I did mange to trigger it now but not on a second attempt. Maybe I have to have Steam open. Maybe I have to swap to bluetooth once first. I have to wait until after my pirate anime stream with friends before I can continue testing, but I'll see if I can get this thing to trigger manually afterward.

[Corrodias]

Things that don't trigger it:
Running XinputTest 60 times in a row, while Steam's running.
Running MHW 14 times.
Switching profiles in DS4Windows 20 times.
Running MHW while there are 15 instances of XinputTest open.

There really seems to be a time component to this, somehow. It just won't do it for me on demand.

[Corrodias]

I gave in and enabled kernel debugging. Surprisingly, it seems to have no appreciable effect on performance, at least when the debugger isn't attached. Nice!

My plan is to run livekd64.exe and run a full dump from kernel mode in kd (with /f), while the system's running, the next time it happens. That'll be 32 GB, since that's how much RAM I have, and apparently when you tell it you want a "full" dump, you mean full.

Would you like me to use any other options instead? Unfortunately, the documentation says you specifically can't make a "kernel memory dump" from a debugger in kernel mode; it has to be a full dump or a small one, which would have precious little information in it.

edit: Of course, it hasn't happened again, yet. It has to draw this out as long as possible. :) I'm using the game in DX12 mode now, but that should have no effect on this part.

[Corrodias]

Starting to think that the game running in DX12 mode just doesn't trigger it, so I'm going to switch back to DX11 mode. Shrug emoji.

[michael-fadely]

A combination that tends to have this issue a lot for me is uplay + Rainbow Six: Siege, if you've got that handy.

[Corrodias]

@michael-fadely I don't, unfortunately. If you can reproduce it as well, then you could submit a memory dump.

I'm still waiting for it to happen, and I fear it won't with DX12 mode, but for unrelated reasons, I can't play in DX11 mode right now.

There are 2 ways:

1. The simple way, if it works. Configure Windows to do a kernel memory dump when it crashes, then manually crash it when the condition occurs. Instructions here: DS4Windows(?) continuously glitches my PC Ryochan7/DS4Windows#1049 (comment) - The downside is that on some systems (including mine), Windows never writes memory dumps. Seems to be a common thing when you're booting from an SSD.
2. Install the Windows debugging kit, enable local kernel debugging with bcdedit, reboot, wait for the condition to occur, and use livekd to do a full memory dump, then send that to nefarius. This is a bit more work, but I have confirmed that there seems to be zero performance impact from enabling debugging. The main downsides here are uploading a gigantic dump file and giving nefarius access to anything secret you have in memory. I trust him with it.

References for method number 2:
https://docs.microsoft.com/en-us/windows-hardware/drivers/debugger/setting-up-local-kernel-debugging-of-a-single-computer-manually
https://docs.microsoft.com/en-us/sysinternals/downloads/livekd
https://docs.microsoft.com/en-us/windows-hardware/drivers/debugger/-dump--create-dump-file-

[AugmentedAntics]

Well, after several weeks of thinking I found a way around it, the deadlock started happening again. And this is the fastest record it's had, less than 1 day after the last reboot and it happens after a few hours in Dolphin. I do have a memory dump after waiting for this to happen again, so if it helps, I can send it.

[Corrodias]

Please do send it to nefarius. I assume the email address in their profile is accurate.

@nefarius
I, too, have finally reproduced the issue, and will be sending an S3 download link. While MHW never triggered it again, restarting Steam did. As usual, the Steam executable was simply stuck at an early stage, and unplugging the controller allowed it to continue, but attempting to restart Windows got stuck as well.

edit: Took long enough to upload, but it's finally done and a link sent.

[nefarius]

Well, unfortunately, it's happened again anyway. I'll have to fall back to just reinstalling the old 2009 360 driver, since that seemed to ward off the deadlocks completely for some reason.

How do you install the old driver? Is it possible in Windows 10? Where can i download that 2009 driver?
Thanks

Should be available here.

[rolibra]

Hi,

My pc is also getting stuck on reboot.
I'm using steamlink + playnite +gloSC to play pc games on the tv.

Running driver version 1.17.333.

[nefarius]

Send me your PC, I wanna finally find out what's wrong here 😅

[KerrmexRPG]

Send me your PC, I wanna finally find out what's wrong here 😅

After a couple of days with the 2009 driver installed I can confirm that the deadlock issue is gone. Everything is working perfectly fine now.

[rolibra]

Send me your PC, I wanna finally find out what's wrong here 😅

After a couple of days with the 2009 driver installed I can confirm that the deadlock issue is gone. Everything is working perfectly fine now.

Any specific version of the driver?

Send me your PC, I wanna finally find out what's wrong here 😅

Haha.

Sorry, please let me know of any info/logs i can send as evidence.

[KerrmexRPG]

Send me your PC, I wanna finally find out what's wrong here 😅

After a couple of days with the 2009 driver installed I can confirm that the deadlock issue is gone. Everything is working perfectly fine now.

Any specific version of the driver?

I'm using a DualShock3 + FireShock + Shibari

[Corrodias]

So far, my PC hasn't misbehaved, with the default Win 10 driver for xbox gamepads, version 10.0.19041.1 from 2019-12-06. But that build date is barely before I reported the problem myself, so I may have had an older version at the time. Hmm! And I haven't been using the gamepad, and I'm not sure whether that makes a difference. I also believe I upgraded to Win 10 version 20H2 between then (version 1908, if I'm right) and now. I did share a couple of memory dumps with Nefarius and Mega back then, but as I understand it, they didn't find anything illuminating.

A couple of days may not be enough; I've had it work for a week before failing, in the past. That said, I remember people saying back then that using the older xbox driver helped then, too.

My old PC that I haven't used in a year still has Windows 7 and has some kind of fan problem causing vibration. I could try to fix it up and update the OS, then see if it experiences the issue and ship it if so. 😆

[rolibra]

Ok, i had the issue again, and just noticed conflicting devices on the device manager:

[nefarius]

That's not conflicting, that means the device is in error state, double-click to see the error message and code it reports.

[KerrmexRPG]

No more deadlocks for me since I installed the 2009 driver, I'm just saying... I've been using my controller often these days.

[rolibra]

I think i've identified a pattern on the issue for me.

If I play using 1 controller only, no issues. But If i play a co-op game using 2 controllers (Streets of rafe 4, tetris), then i get the issue when shutting down or rebooting.

[Corrodias]

I played Horizon Zero Dawn on and off this week without issue. Finally rebooted today for updates, which went fine.

For those of you still experiencing the problem, what driver version for the Xbox 360 Controller do you have? Looking at that properties dialog, the date and version number are the useful information. I already said what mine is, and I don't seem to be experiencing the issue any more.

[Corrodias]

Finally experienced the deadlock again, so this combination with the newest 360 driver is unfortunately still a problem.

[gjmv]

I have installed the new version since December, and it only happened to me 2 times the first days. During January and February I used it a lot, and I didn't have any deadlocks.
Thanks for the update!
Regards

[nefarius]

Even once is too much though, thanks for the feedback nonetheless!

[Corrodias]

Pity, I left it running overnight ONE DAY and it happened again. 😆

Is there any thing further I can do to help you diagnose this, other than shipping you my PC? I provided some (unfortunately massive) memory dumps, but evidently those weren't enough on their own.

[nefarius]

Pity, I left it running overnight ONE DAY and it happened again. 😆

Is there any thing further I can do to help you diagnose this, other than shipping you my PC? I provided some (unfortunately massive) memory dumps, but evidently those weren't enough on their own.

I'll just scratch everything and start anew 👻

[valentyn-l]

I am STILL getting a driver deadlock issue with the 2021 Xbox 360 controller driver. Not an issue with the 2009 driver. This is a problem though because the 2009 driver prevents me from activating Core Isolation in the Device Security settings. Tested this on both Windows 10 and 11 leaked build.

[nefarius]

I've pretty much given up on this to be honest. There's simply no resources I or any potential contributor could come up with to tackle this mystery and I'm tired of it and will move on. Sorry.

[andrewklamut]

Hi, I'm facing a 100% reproducible issue which might be relevant? The issue is that when I start to use my Xbox 360 controller over Parsec, the host system hangs and must be forced shut down.

Steps:

1. Using Windows Hyper V, create a Windows 10 VM

2. Install Parsec on VM guest and VM host

3. Plug Xbox 360 controller into VM host

4. Start the Windows 10 VM guest using Hyper-V -> Start (don't do Hyper-V connect) and connect via Parsec

5. Press any button on the Xbox 360 controller

6. The VM guest hangs and must be force shut down. If I open Device Manager on the VM guest before step 4, I can see the Xbox Peripheral device is added as step 4 happens, and the Device Manger refreshes very slowly (I see the tree collapse, and then expand).

It kind of seems like a deadlock and I came across this thread and noticed you were stuck, maybe same root cause?

[ankomen2]

I tried the 2009 driver and my pc still freezes when I use my controller. Do you guys have any suggestions?

[wingfixer]

Found this topic by accident, realized it was the same issue I had with (as I thought) DS4Windows forever ago, ended up giving up on it and just setting a hotkey to disable the whole thing when not needed.
Appears to be completely random, could run for hours or days before it happens, so people that don't leave their PC running for long may be susceptible but will never notice it.

[nefarius]

It will be solved by the rewrite I am working on in the dev dungeon ⚒️

[nefarius]

Time to close this. Can't fix something I can not replicate. Sorry.

[gjmv]

I have the driver installed again in windows 11, in PC and laptop, and never again get deadlock.

[nefarius]

This issue isn't about a bluescreen.

[gjmv]

This issue isn't about a bluescreen.

Sorry, deadlock

I have installed the new version since December, and it only happened to me 2 times the first days. During January and February I used it a lot, and I didn't have any deadlocks. Thanks for the update! Regards

[starjet]

I know this repo is no longer being updated in favour of whatever new one nefarius is working on, but I still want to comment on this
I faced this problem myself and finally managed to isolate the issue to the controller
This was pretty hard to troubleshoot as there's so little info on the web about this problem. As someone above said, only the few of us that keep our PCs running for a long time are bound to notice this most likely

Anyway, two things:
What drove me towards this being related to the Xbox controller driver is this post in Microsoft Answers: https://answers.microsoft.com/en-us/windows/forum/all/video-games-wont-launch-problems-with-restarting/6b139719-470d-4658-860c-68f3224e0a40

When this problem starts, the game seems to stop the process during the launch process (loading dlls, etc.). So I tried using ProcessMonitor, thinking that if I could find the last dll that was trying to load, it might help solve the problem.
...
Then I found that it stopped after loading the "xinput" dll file... Yes, I always have an Xbox360 (very old) controller plugged into my PC. I pulled it out and plugged it in. My games are now up and running!

This suggests that the problem could be tied to the basic Xbox 360 controller driver itself, and not something that originates in Vigembus (Though Vigembus somehow likely exacerbates the problem, leading to the driver being stuck even after you pull out the controller)

I will be following the recommendation to downgrade to the 2009 Xbox 360 controller driver. Will report back if I face this problem again. Though it's going to be hard to tell if it's fixed per se, as this problem only pops up very randomly (Sometimes it may happen twice in a week, but sometimes it may not happen for three to four months at a stretch)