bash won't start after hang during intensive compilation & zombie processes remain after reboots

[bsdz]

• Your Windows build number: Microsoft Windows [Version 10.0.16232.1000]

• What you're doing and what's happening:
  Was attempting to compile haskell as suggested here. After several hours of compilation bash windows froze post gcc stage. Was unable to start a new process using an existing separate bash window. Attempted to run a sudo command but froze.

• What's wrong / what should be happening instead:
  After rebooting several times, unable to start bash process. Console window opens but bash does not start. In Process Hacker/Task Manager I can see ghost processes that had initially stalled. These zombie processes do not die after rebooting (several times) or by terminating in process explorer/task manager. Here is a screenshot of my process explorer highlighting zombie processes after several reboots.
  
  This is what I see if I try to start bash shell.
  

• Strace of the failing command, if applicable:
  Unfortunately, i can't get to that stage but I am able to PM you a log from process monitor while running bash.exe --login from cmd.exe if that helps.

Is there any way to reset the subsystem's memory/process cache manually?

[fpqc]

Wait, even after a hard boot they don't clear? But probably don't try it, because the devs will likely ask you tomorrow for ETL traces.

Here is how to turn on logging for traces:

https://github.com/Microsoft/BashOnWindows/blob/master/CONTRIBUTING.md#8-detailed-logs

I do know that you might be able to kill them with kProcessHacker (need to install process hacker's kernel mode driver) by force-killing the offending svchost.exe hosting the lxssmanager service, but this shouldn't be necessary. It looks like you've hit a bug, and if you can produce logs, it will likely be useful to the dev team (since another person reported something similar but didn't log it, and whatever caused his problem is therefore unfixed).

[bsdz]

Yep I'll hold back from any hard reset. Although I will need my system back soonish I'm happy to wait for now. I'll see if I can set suggested trace options once I'm back on that laptop in around 10hrs :)

[bsdz]

Unfortunately, after setting up debugging as described here and rebooting the problem resolved the issue. I attach the debug logs just in case there's something useful in them.

wsl-log.zip

[benhillis]

@bsdz - The issue you are experiencing sounds like a deadlock somewhere in our driver so the logs are unlikely to have any useful breadcrumbs. I will follow through the instructions you linked and see if I can reproduce the hang.

[bsdz]

That's a shame. I did manage to record a Process Monitor log during the hang. Perhaps in there might be some more useful breadcrumbs in the underlying call stacks. Good luck! Logfile.zip

[fpqc]

@benhillis Is there any way to purposely cause the system to crash and do a kernel memory dump in a case where the driver deadlocks? It might be useful if we find rare or intermittent deadlocks like this in the future.

At least when there's a bugcheck rather than a deadlock, we have something to give you.

[benhillis]

@fpqc - Indeed, a memory dump would be much more useful since it would have stacks of all the lxcore threads. Here are some instructions on how to enable on-demand memory dumps (CrashOnCtrlScroll):

https://blogs.technet.microsoft.com/askpfeplat/2015/04/05/how-to-force-a-diagnostic-memory-dump-when-a-computer-hangs/

[fpqc]

@benhillis I have this enabled, but do you need the full kernel memory dump, the full memory dump, or just the minidump? I would assume it's the full kernel memory dump, since otherwise it will just save a minidump of the 'crasher' driver thing

Edit: oops It's in the article.

Anyway, it would make sense to add this to the same page that has the ETL stuff, contributing.md, iirc.

[sunilmut]

@fpqc

Anyway, it would make sense to add this to the same page that has the ETL stuff, contributing.md, iirc.

Good idea. I documented the steps here.

[fpqc]

@sunilmut A complete memory dump will definitely not be something one can send by email, even if it is compressed..?

Does Microsoft have a secure website for uploading large memory dumps manually?

[sunilmut]

@fpqc

A complete memory dump will definitely not be something one can send by email, even if it is compressed..?

No, they won't be. The suggestion was to put it on a share somewhere and share the location to that. I will update the instructions to make it more clear.

Does Microsoft have a secure website for uploading large memory dumps manually?

I am not aware of one. But, I will ask.

[sunilmut]

@fpqc - So, I heard back from MSRC (owners of secure@microsoft.com) and here is what they said:
We currently do not have an official repository, but do recommend our finders to use OneDrive and share these resources with us privately.

[fpqc]

Neat! Sounds good!

[benhillis]

Cleaning up old threads.