All *nix (cygwin) tools crash as of 15082014

[MartinSGill]

As of this latest update for every unix-like tool I use (e.g. ssh included with git) I get crash reports popping up.

The programs themselves do appear to work, so I'm guessing this is somehow related to the new hook module.

bash.sh
(crash), when I click away the report I get the prompt
ls
directory is listed, (crash) when I click away the report I get the prompt back
when I call something like git pull (which call many unix commands I end up with a desktop full of errors.

This only happens in ConEmu, regardless of which shell (e.g. powershell, cmd etc) I use.

[Maximus5]

I do not see any crashes. So, you need to either describe step-by-step instruction, or provide crash dump.
https://conemu.github.io/en/MemoryDump.html

[MartinSGill]

How I reproduce it:

1. Start ConEmu (x86, or x64 both crash), with powershell (admin) console
2. Change to a git repository folder
3. run "git pull" (git/bin is in path)

Crash dump: https://www.dropbox.com/s/44ygkgqh6nvk4uw/WER3F44.tmp.mdmp?dl=0

Everything was working fine last Friday, and ConEmu is the only thing I've updated.

[nileflowers]

+1

Seeing it all git-bash commands on Windows 7 on 150816 alpha. None of these tools are updated except ConEmu.

[Maximus5]

@MartinSGill
I see two other detouring apps: NVIDIA CoProcManager and Sophos Anti-Virus. I don't have any of them and can't check. But one of them may raise this crash.

[Maximus5]

@nileflowers
Same here. Crash dump.

[nileflowers]

@Maximus5
Here are the related crash dump files

https://github.com/nileflowers/conemu-crash-files/raw/master/conemu-crash.zip

[MartinSGill]

I've been reverting my install. Build 150813 is the last build that works for me. Whatever the issue is it seems to have been introduced in 150814.

[kailiu42]

I have the same problem here. And it does not only affect cygwin tools... The tools so far I know are affected include:

• msysgit shell
• rustc compiler - rustc itself runs OK but when it calls the gcc.exe, gcc.exe will fail
• HP RESTful API tool - a command line tool provided by HP for their server management

Uncheck Inject ConEmuHK in Features can workaround this issue.

[Maximus5]

@Kraml Crash dump

[Maximus5]

1. Both supplied crash dumps has either C:\Program Files (x86)\NVIDIA Corporation\CoProcManager\_etoured.dll or C:\Program Files (x86)\NVIDIA Corporation\CoProcManager\detoured.dll. As I've said before, this detours may harm normal behavior of applications.
2. Even I have installed NVIDIA drivers this CoProcManager is not installed on my PC. So I can't check anything. What is that CoProcManager??? Also, google saw a lot of issues with it.
3. Both crash dumps do not describe the first crash. Pleas follow these steps: http://conemu.github.io/en/CrashDump.html#manual-dumps And I'm insterested in full memory dumps of crashed process and its direct parent process.

[Maximus5]

One more thing. Because I don't have CoProcManager installed on my PC, I need to know exact Video Card models and drivers (version, installation file name, url) you are using. And, because I'm not a end-point user of this product, seems like I can't file an issue to NVIDIA.

[kailiu42]

@Maximus5 here is the dump for the hprest.exe command as a HP tool.
https://dl.dropboxusercontent.com/u/16148499/CEDump-150816-56060-150818180008.7z

I do have a nVIDIA card, Quadro K2100m, on my laptop. The driver version is 352.86. DL link:
http://www.nvidia.com/download/driverResults.aspx/85053/en-us

I'm also downloading the latest video driver and will feedback whether it changes anything.

Update regarding video driver:
Since my laptop support hybrid graphics, I just tested uninstall nVIDIA driver completely and leave only the Intel integrated video. The problem was gone. After installed back the latest nVIDIA driver, it's back. So it definitely is related to nVIDIA driver, but only happens after a certain ConEmu revision...

[Maximus5]

There are a lot of page found by google by simple query "coprocmanager detoured.dll problem".
www.google.com/search?...

[Maximus5]

And the expected result.

So, at least for tests, I need:

• This mysterios CoProcManager folder (well, I belive the files are somewhere in the distro folder, but I need exact list of them)
• Exact command which initialize detours. I believe it somewhere in the startup items and most probably it infiltrates explorer.exe.

Till than I can't tell who exactly is responsible for these crashes. And may be that is NVIDIA problem, especially because a lot of another compains exist on the internet. ConEmu sources are open, but NVIDIA sources seems like to be closed. So I even can't check or fix the bug, if the bug is inside CoProcManager.

[Maximus5]

And I think this simple app will crash too on reported workstations. Please check.
https://github.com/Maximus5/test-callexit/releases/tag/v1.0

[kailiu42]

Here are the files under the coprocomanager folder:
C:\Program Files\NVIDIA Corporation\coprocmanager\detoured.dll
C:\Program Files\NVIDIA Corporation\coprocmanager\nvd3d9wrapx.dll
C:\Program Files\NVIDIA Corporation\coprocmanager\nvdxgiwrapx.dll
C:\Program Files (x86)\NVIDIA Corporation\coprocmanager\detoured.dll
C:\Program Files (x86)\NVIDIA Corporation\coprocmanager\nvd3d9wrap.dll
C:\Program Files (x86)\NVIDIA Corporation\coprocmanager\nvdxgiwrap.dll

The two detoured.dll are in the driver distro, the two nv*.dll fiiles are as .dl_ so need to be uncompressed, 7-zip can do it.

Checked by sysinternal autorun tool none of these files are referenced in common startup location. However they are referenced in the registry.

I've uploaded an archive of the above dll files and the registry export.
https://dl.dropboxusercontent.com/u/16148499/ConEmu%20Crash%20nVIDIA%20files.7z

I can also confirm that after renamed the two coprocmanager folders the problem was gone.

I would like to try the simple callexit app but lacks of two dlls on my system. Although downloaded them from opendll.com still no luck as the dependencies have dependencies. Could you upload a static linked version?

[Maximus5]

Shame on me. Try this: https://github.com/Maximus5/test-callexit/releases/tag/v1.1

[Maximus5]

Is detoured.dll loaded in explorer.exe? Check with ProcessExplorer.

[Maximus5]

Try to filter by nvidia and show what autorun finds. Here is mine:

[kailiu42]

No binary download for the CallExit 1.1 release...

detoured.dll is not loaded in any process. The other two nv*.dll files under the same folder are loaded in many processes but not explorer.exe. detoured.dll are their dependencies though.

[Maximus5]

https://github.com/Maximus5/test-callexit/releases/tag/v1.1

[Maximus5]

In the latest dump from this comment I see a lot of another detours: AVG, MacrType, Sizer, HP ProtectTools.
And only removing NVIDIA's CoProcManager fixes the crash? From my point of view, this fact side by side with this search query is an evidence that problems must be fixed by NVIDIA.

[kailiu42]

CallExit 1.1 crashed as expected, if Inject ConEmuHK is checked and nVIDIA coprocmanager is in place.

[Maximus5]

It would be interesting to find minimal reproduction code. It may help to find a workaround or to submit bugreport to NVIDIA. I prefer last option of course...

So, can anyone from this topic build ConEmuHk with VisualStudio or MinGW? To avoid a lot of test iterations...

[javahaxxor]

happens to me too. Uncheck Inject ConEmuHK works as a workaround, thanks kraml

[Maximus5]

Remove CoProcManager as workaround... Without ConEmuHk you may face another crashes and lack of features.

[kailiu42]

@Maximus5 I can build with MinGW, if you can provide instructions.

[Maximus5]

Take latest daily branch.
Cd into ./src/ConEmuHk
Supposing you have 32bit mingw toolchain:

mingw32-make -f makefile_gcc DIRBIT=32

Take binary from ./Release folder and update your test location (it may be this ./Release folder).
Run ConEmu

ConEmu.exe -basic -cmd {cmd-32}

Post here when you have success or crash after that.

[Maximus5]

Build 150908 was releases. Most probably, crash will repeat... If so, I need new dumps with this version.

[kailiu42]

Yes it crashed as expected. Here are the dumps from the same process, similar to my previous comment #272 (comment)

https://dl.dropboxusercontent.com/u/16148499/CEDump-20150907%201359.7z

[Maximus5]

Please check the following patch and create new dumps. Two dumps is quite enough.
https://sourceforge.net/projects/conemu/files/temp/ConEmuHk.150908b.dll.7z/download

[kailiu42]

Here are the dumps:

https://dl.dropboxusercontent.com/u/16148499/CEDump-20150907%202126.7z

[Maximus5]

How about this one?
https://sourceforge.net/projects/conemu/files/temp/ConEmuHk.150908c.dll.7z/download

[kailiu42]

Crash the same. Need dump?

[Maximus5]

Yes, please

[kailiu42]

Here comes the dump.

https://dl.dropboxusercontent.com/u/16148499/CEDump-20150907%202311.7z

[Maximus5]

Please, check this one. Dumps if crashes.
https://sourceforge.net/projects/conemu/files/temp/ConEmuHk.150908d.dll.7z/download

[kailiu42]

Yes, still crashing.

https://dl.dropboxusercontent.com/u/16148499/CEDump-20150908%200857.7z

[Maximus5]

How about this patch?
http://sourceforge.net/projects/conemu/files/temp/ConEmuHk.150908e.dll.7z/download

[kailiu42]

Crash.

https://dl.dropboxusercontent.com/u/16148499/CEDump-20150909%201343.7z

[Maximus5]

Can you, please, run CallExit.exe outside of ConEmu and create full dump at the moment of Press Enter to continue appears? Please, ensure that after Enter it will not crash (outside of ConEmu).
You may create dump with ProcessExplorer.

[Maximus5]

And try this patch: http://sourceforge.net/projects/conemu/files/temp/ConEmuHk.150908f.dll.7z/download

[kailiu42]

No crash with ConEmuHk.150908f.dll.7z now. I've test callexit.exe, rustc, hprest and some msys2 commands, with both 64-bit and 32-bit ConEmu, no more crash now.

As for the dump you requested in this comment #272 (comment) , I would like to provide one but when I create dump from PE, I got a dialog that says "Error configuring dump resources: system cannot found the designated file", and the dump file is 0 byte. I never used PE to dump before so not sure whether I missed any required configuration on my system.

[Maximus5]

Well, you may use ConEmuC to create dump if running process.

[kailiu42]

Here is the dump for callexit from cmd.exe instead of ConEmu.

https://dl.dropboxusercontent.com/u/16148499/CEDump-150908-46504-150909214707.7z

I created it by start callexit, find its PID from PE, then run comemuc /debugpid= and press ctrl+break to generate a full dump when the press enter message is on the screen.

[Maximus5]

Weird... Last dump has no CoProcManager dll's loaded. What do NVIDIA think about?

OK, this is last test before next release. I'm not happy with workaround, but it's working at least. Anyway, I still believe the problem is inside NVIDIA driver.

https://sourceforge.net/projects/conemu/files/temp/ConEmuHk.150908g.dll.7z/download

[kailiu42]

The last g dll works ok, no crash.

I do notice that when started from cmd.exe there are no NVIDIA coprocmanager DLLs attach with callexit.exe. But if start it directly from explorer windows by double click, NVIDIA is there. Here is the dump in such case, if you would like to look into it.

https://dl.dropboxusercontent.com/u/16148499/CEDump-150908-71028-150910202156.7z

[Maximus5]

I do notice that when started from cmd.e­xe there are no NVIDIA coprocmanager DLL­s attach with callexit.exe.
That's interesting... How did you start cmd.exe and does that cmd.exe have NVIDIA dlls loaded?

[kailiu42]

I started cmd.exe by click the start menu button, type cmd.exe in the search box and press enter. I just tried find the cmd.exe file in explorer window under C:\Windows\System32 and double click it. In neither case no NVIDIA dll was loaded.

[Maximus5]

Weird logic...

[nileflowers]

This issue is no longer occurring after updating to 150910. I have the Settings | Features | In-console options | Inject ConEmuHk checkbox enabled, so no need for the previous semi-workaround.

[kailiu42]

yep 150910 works.

[Maximus5]

Can you please create debug log files for Shell mode during execution of CallExit.exe?

http://conemu.github.io/en/SettingsDebug.html

[Maximus5]

How about build 151006?

Related: gh-341, gh-323

[MartinSGill]

I've not noticed any issues since latest 151006.