New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unable to launch Bash on Ubuntu on Windows on Windows Insider Build 15002 #1012

Closed
asser-dk opened this Issue Jan 11, 2017 · 18 comments

Comments

Projects
None yet
5 participants
@asser-dk

asser-dk commented Jan 11, 2017

Versions

ConEmu build: 161206 x64
OS version: Windows 10 Pro x64 Insider Preview 15002.rs_prerelease.170102-1700
Used shell version (Far Manager, git-bash, cmd, powershell, cygwin, whatever): Bash on Ubuntu on Windows

Problem description

I updated my Windows Installation to Insider build 15002 this morning and now I'm unable to open bash with ConEmu. When I try to open bash I get the following error: Exception 0xC0000005 (Write x0000000001D26BEE) was occurred (ConEmu64.exe, PID=3944) ConEmu build 161206 64

I'm able to open Bash outside ConEmu just fine and I can open cmd, Powershell and git-bash (bundled with git from git-scm.com) in ConEmu just fine.

I tried to reinstall conemu and I tried to delete the conemu.xml file from AppDataRoaming but the issue persists.

Steps to reproduce

  1. Update to Windows Insider build 15002
  2. Open Bash on Ubuntu on Windows using ConEmu

Repro rate: 5/5

Actual results

An error popup appears (see screenshot "Capture").
When dismissing the popup ConEmu shuts down and Bash opens in a new non-ConEmu window (see screenshot "Capture2")

Expected results

Bash should open normally inside ConEmu

Additional files

Memory dump, settings, screenshots, logs

@asser-dk

This comment has been minimized.

Show comment
Hide comment
@asser-dk

asser-dk Jan 11, 2017

I forgot to add the file to the shared folder >.< fixed now I hope

asser-dk commented Jan 11, 2017

I forgot to add the file to the shared folder >.< fixed now I hope

@Maximus5

This comment has been minimized.

Show comment
Hide comment
@Maximus5

Maximus5 Jan 11, 2017

Owner

Check the permissions? Download forbidden...

2017-01-11_13-01-16

Owner

Maximus5 commented Jan 11, 2017

Check the permissions? Download forbidden...

2017-01-11_13-01-16

@asser-dk

This comment has been minimized.

Show comment
Hide comment
@Maximus5

This comment has been minimized.

Show comment
Hide comment
@Maximus5

Maximus5 Jan 11, 2017

Owner

It works!

Owner

Maximus5 commented Jan 11, 2017

It works!

@Maximus5

This comment has been minimized.

Show comment
Hide comment
@Maximus5

Maximus5 Jan 11, 2017

Owner

The code raises crash in the Windows kernel. @miniksa is that expected?

DWORD dwType, dwSize;
RegQueryValueEx(hKeyNames, L"Counters", 0, &dwType, NULL, &dwSize);
Owner

Maximus5 commented Jan 11, 2017

The code raises crash in the Windows kernel. @miniksa is that expected?

DWORD dwType, dwSize;
RegQueryValueEx(hKeyNames, L"Counters", 0, &dwType, NULL, &dwSize);
@Maximus5

This comment has been minimized.

Show comment
Hide comment
@asser-dk

This comment has been minimized.

Show comment
Hide comment
@asser-dk

asser-dk Jan 11, 2017

I got this new error instead:

Exception 0xC0000005 (Write x0000000001D26BFD) was occurred (ConEmu64.exe, PID=16536) ConEmu build 170110 64

logs and memory dump: https://drive.google.com/open?id=0B9uKIrTjhg2NWXFYMmJ3UkJVSG8

Just to be clear this wasn't me doing something wrong. I just replaced the ConEmu64.exe in my installation with the one provided in the zip. Was this correct?

asser-dk commented Jan 11, 2017

I got this new error instead:

Exception 0xC0000005 (Write x0000000001D26BFD) was occurred (ConEmu64.exe, PID=16536) ConEmu build 170110 64

logs and memory dump: https://drive.google.com/open?id=0B9uKIrTjhg2NWXFYMmJ3UkJVSG8

Just to be clear this wasn't me doing something wrong. I just replaced the ConEmu64.exe in my installation with the one provided in the zip. Was this correct?

@Maximus5

This comment has been minimized.

Show comment
Hide comment
@Maximus5

Maximus5 Jan 11, 2017

Owner

Was this correct?

Yep.

Things are worse that I can imagine. The crash occurs at

>	advapi32.dll!PerflibciSetObjectsValidityState()
 	advapi32.dll!PerflibciEnsureCounterSetList()
 	advapi32.dll!PerflibciEnsurePerflibV2StringTable()
 	advapi32.dll!PerfGetNames()
 	advapi32.dll!_guard_dispatch_icall_nop()
 	KERNELBASE.dll!LocalBaseRegQueryValue()
 	KERNELBASE.dll!RegQueryValueExW()

That means, the Insider build 15002 is broken completely. ConEmu can't do anything with that.

Owner

Maximus5 commented Jan 11, 2017

Was this correct?

Yep.

Things are worse that I can imagine. The crash occurs at

>	advapi32.dll!PerflibciSetObjectsValidityState()
 	advapi32.dll!PerflibciEnsureCounterSetList()
 	advapi32.dll!PerflibciEnsurePerflibV2StringTable()
 	advapi32.dll!PerfGetNames()
 	advapi32.dll!_guard_dispatch_icall_nop()
 	KERNELBASE.dll!LocalBaseRegQueryValue()
 	KERNELBASE.dll!RegQueryValueExW()

That means, the Insider build 15002 is broken completely. ConEmu can't do anything with that.

@asser-dk

This comment has been minimized.

Show comment
Hide comment
@asser-dk

asser-dk Jan 11, 2017

Bummer. Would you know if it makes sense that I pass this bug along to the BashOnWindows team? And if so, do you have any information from the crash I could include in the issue?

asser-dk commented Jan 11, 2017

Bummer. Would you know if it makes sense that I pass this bug along to the BashOnWindows team? And if so, do you have any information from the crash I could include in the issue?

@Maximus5

This comment has been minimized.

Show comment
Hide comment
@Maximus5

Maximus5 Jan 11, 2017

Owner

The bug doesn't relate to BashOnWindows. It's native Windows API bug.

Owner

Maximus5 commented Jan 11, 2017

The bug doesn't relate to BashOnWindows. It's native Windows API bug.

@Maximus5

This comment has been minimized.

Show comment
Hide comment
@Maximus5

Maximus5 Jan 11, 2017

Owner

Perhaps it depends on BashOnWindows, but only MSDT devs may check that.

Owner

Maximus5 commented Jan 11, 2017

Perhaps it depends on BashOnWindows, but only MSDT devs may check that.

@miniksa

This comment has been minimized.

Show comment
Hide comment
@miniksa

miniksa Jan 11, 2017

I'm not able to reproduce this issue on a clean x64 VM of 15002.rs_prerelease.170102-1700.

I also couldn't find any records of bugs/issues logged against the performance counters code behind RegQueryValueExW nor do I see any recent changes to that code.

miniksa commented Jan 11, 2017

I'm not able to reproduce this issue on a clean x64 VM of 15002.rs_prerelease.170102-1700.

I also couldn't find any records of bugs/issues logged against the performance counters code behind RegQueryValueExW nor do I see any recent changes to that code.

Maximus5 added a commit that referenced this issue Jan 18, 2017

Internal. Ensure variables are initialized before RegQueryValueEx
  RegQueryValueEx crashed on Windows Insider Build 15002 while it's trying
  to acquire the performance counters.

  Seems like it's a race in kernel.

  Ref: gh-1012
@Maximus5

This comment has been minimized.

Show comment
Hide comment
@Maximus5

Maximus5 Jan 19, 2017

Owner

@asser-dk I see BitDefender in the loaded modules. I'm almost sure it's not a problem, but can you run Bash/ConEmu without it to ensure?

avcuf64.dll	*C:\Program Files\Bitdefender\Endpoint Security\Signatures\AVC\AVC3_00565_030\avcuf64.dll	3.12.17122.6492	25.11.2016 14:46	000000006C0B0000-000000006C15E000

@miniksa Can we do anything to troubleshoot or report the problem? At the moment it looks like a race in advapi32.dll (PerflibciSetObjectsValidityState). Two crash dump are available.

advapi32.dll		10	10.00.15002.1001	13.09.1936 0:19	00007FF93D9D0000-00007FF93DA78000
>	advapi32.dll!PerflibciSetObjectsValidityState() 	
 	advapi32.dll!PerflibciEnsureCounterSetList()  + 0xb7 bytes	
 	advapi32.dll!PerflibciEnsurePerflibV2StringTable()  + 0xd6 bytes	
 	advapi32.dll!PerfGetNames()  + 0x40a bytes	
 	advapi32.dll!_guard_dispatch_icall_nop()  + 0x5c0f bytes	
 	KERNELBASE.dll!LocalBaseRegQueryValue()  + 0x43f bytes	
 	KERNELBASE.dll!RegQueryValueExW()  + 0xf6 bytes	
Owner

Maximus5 commented Jan 19, 2017

@asser-dk I see BitDefender in the loaded modules. I'm almost sure it's not a problem, but can you run Bash/ConEmu without it to ensure?

avcuf64.dll	*C:\Program Files\Bitdefender\Endpoint Security\Signatures\AVC\AVC3_00565_030\avcuf64.dll	3.12.17122.6492	25.11.2016 14:46	000000006C0B0000-000000006C15E000

@miniksa Can we do anything to troubleshoot or report the problem? At the moment it looks like a race in advapi32.dll (PerflibciSetObjectsValidityState). Two crash dump are available.

advapi32.dll		10	10.00.15002.1001	13.09.1936 0:19	00007FF93D9D0000-00007FF93DA78000
>	advapi32.dll!PerflibciSetObjectsValidityState() 	
 	advapi32.dll!PerflibciEnsureCounterSetList()  + 0xb7 bytes	
 	advapi32.dll!PerflibciEnsurePerflibV2StringTable()  + 0xd6 bytes	
 	advapi32.dll!PerfGetNames()  + 0x40a bytes	
 	advapi32.dll!_guard_dispatch_icall_nop()  + 0x5c0f bytes	
 	KERNELBASE.dll!LocalBaseRegQueryValue()  + 0x43f bytes	
 	KERNELBASE.dll!RegQueryValueExW()  + 0xf6 bytes	
@miniksa

This comment has been minimized.

Show comment
Hide comment
@miniksa

miniksa Jan 19, 2017

I looked again at the dump and it's very strange.

0:006> uf advapi32!PerflibciSetObjectsValidityState
advapi32!PerflibciSetObjectsValidityState:
 2194 00007ff9`3d9d6d18 0000            add     byte ptr [rax],al
 2194 00007ff9`3d9d6d1a 50              push    rax
 2194 00007ff9`3d9d6d1b c3              ret

Having only 3 assembly instructions for this function (after looking at the source it is generated from) doesn't make any sense and makes me suspect something on @asser-dk's system has tampered with advapi32.

In comparison on my 15014 system, calling uf on the same function gives me 50+ assembly instructions, not 3. And it doesn't start with add byte ptr[rax], al, it starts with the more sensible retrieval of the stack variable with mov instructions to retrieve the single TRUE/FALSE parameter to this function.

0:007> uf advapi32!PerflibciSetObjectsValidityState
ADVAPI32!PerflibciSetObjectsValidityState:
 2194 00007fff`84ba6d18 488b0424        mov     rax,qword ptr [rsp]
 2194 00007fff`84ba6d1c 6448890424      mov     qword ptr fs:[rsp],rax
 2194 00007fff`84ba6d21 53              push    rbx
 2194 00007fff`84ba6d22 4883ec20        sub     rsp,20h
 2194 00007fff`84ba6d26 8ad9            mov     bl,cl
.... (more lines, I clipped them out) ....

Additionally, the exception itself...

0:006> .ecxr
rax=0000000001d26bee rbx=0000000000000000 rcx=0000000000000000
rdx=00007ff93d9e180c rsi=0000000000000000 rdi=0000000000001000
rip=00007ff93d9d6d18 rsp=0000003ffb1fe168 rbp=0000003ffb1fe1d0
 r8=0000003ffb1fe0c8  r9=0000003ffb1fe1d0 r10=0000000000000000
r11=0000003ffb1fe168 r12=0000000000000063 r13=0000000000000000
r14=0000000000000000 r15=0000000000000000
iopl=0         nv up ei pl zr na po nc
cs=0033  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00010246
advapi32!PerflibciSetObjectsValidityState:
00007ff9`3d9d6d18 0000            add     byte ptr [rax],al ds:00000000`01d26bee=??

The exception is a write access error to the memory at 1d26bee. But that's not where the heaps are with !heap and it's not inside any library or valid address space with !address. So if it's trying to write to invalid memory for no particularly good reason, that's why we hit an access violation.

But reading the source code for this function... there is no write operation that should be occurring immediately on entry to PerflibciSetObjectsValidityState. It should have grabbed its parameters off the stack, checked that the lock variable was not null (read access), and then jumped immediately to kernel32!WaitForSingleObject on the lock value it just checked.

So I further suspect something has tampered with advapi32.dll.

BitDefender could be a good suspect. AntiVirus applications commonly place hooks inside system DLLs and detour them. Perhaps this was a bad detouring of advapi32 or a hook/detour that can't compensate for some compiler/linker update that adjusted the layout of advapi32.dll.

So as of now, I don't really have anything to file unfortunately as it doesn't look like a race condition at all to me. It looks like some other piece of software on @asser-dk's system is tampering with system DLLs and that's not something we support. If it happens on a clean machine or another machine where we understand the full state of everything installed to reproduce it (so it can be proven that the system is at fault), I'm happy to file an internal bug and pass it along. @Maximus5, you're not seeing this at all on your machines, correct?

miniksa commented Jan 19, 2017

I looked again at the dump and it's very strange.

0:006> uf advapi32!PerflibciSetObjectsValidityState
advapi32!PerflibciSetObjectsValidityState:
 2194 00007ff9`3d9d6d18 0000            add     byte ptr [rax],al
 2194 00007ff9`3d9d6d1a 50              push    rax
 2194 00007ff9`3d9d6d1b c3              ret

Having only 3 assembly instructions for this function (after looking at the source it is generated from) doesn't make any sense and makes me suspect something on @asser-dk's system has tampered with advapi32.

In comparison on my 15014 system, calling uf on the same function gives me 50+ assembly instructions, not 3. And it doesn't start with add byte ptr[rax], al, it starts with the more sensible retrieval of the stack variable with mov instructions to retrieve the single TRUE/FALSE parameter to this function.

0:007> uf advapi32!PerflibciSetObjectsValidityState
ADVAPI32!PerflibciSetObjectsValidityState:
 2194 00007fff`84ba6d18 488b0424        mov     rax,qword ptr [rsp]
 2194 00007fff`84ba6d1c 6448890424      mov     qword ptr fs:[rsp],rax
 2194 00007fff`84ba6d21 53              push    rbx
 2194 00007fff`84ba6d22 4883ec20        sub     rsp,20h
 2194 00007fff`84ba6d26 8ad9            mov     bl,cl
.... (more lines, I clipped them out) ....

Additionally, the exception itself...

0:006> .ecxr
rax=0000000001d26bee rbx=0000000000000000 rcx=0000000000000000
rdx=00007ff93d9e180c rsi=0000000000000000 rdi=0000000000001000
rip=00007ff93d9d6d18 rsp=0000003ffb1fe168 rbp=0000003ffb1fe1d0
 r8=0000003ffb1fe0c8  r9=0000003ffb1fe1d0 r10=0000000000000000
r11=0000003ffb1fe168 r12=0000000000000063 r13=0000000000000000
r14=0000000000000000 r15=0000000000000000
iopl=0         nv up ei pl zr na po nc
cs=0033  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00010246
advapi32!PerflibciSetObjectsValidityState:
00007ff9`3d9d6d18 0000            add     byte ptr [rax],al ds:00000000`01d26bee=??

The exception is a write access error to the memory at 1d26bee. But that's not where the heaps are with !heap and it's not inside any library or valid address space with !address. So if it's trying to write to invalid memory for no particularly good reason, that's why we hit an access violation.

But reading the source code for this function... there is no write operation that should be occurring immediately on entry to PerflibciSetObjectsValidityState. It should have grabbed its parameters off the stack, checked that the lock variable was not null (read access), and then jumped immediately to kernel32!WaitForSingleObject on the lock value it just checked.

So I further suspect something has tampered with advapi32.dll.

BitDefender could be a good suspect. AntiVirus applications commonly place hooks inside system DLLs and detour them. Perhaps this was a bad detouring of advapi32 or a hook/detour that can't compensate for some compiler/linker update that adjusted the layout of advapi32.dll.

So as of now, I don't really have anything to file unfortunately as it doesn't look like a race condition at all to me. It looks like some other piece of software on @asser-dk's system is tampering with system DLLs and that's not something we support. If it happens on a clean machine or another machine where we understand the full state of everything installed to reproduce it (so it can be proven that the system is at fault), I'm happy to file an internal bug and pass it along. @Maximus5, you're not seeing this at all on your machines, correct?

@Maximus5

This comment has been minimized.

Show comment
Hide comment
@Maximus5

Maximus5 Jan 19, 2017

Owner

Thanks for investigations! Unfortunately I have not yet updated any of my PC to 15002 build, so I can't say yes or no.
I believe you are right and the function was detoured.
ConEmu doesn't detour its own executable, so the problem might be in BitDefender.

Owner

Maximus5 commented Jan 19, 2017

Thanks for investigations! Unfortunately I have not yet updated any of my PC to 15002 build, so I can't say yes or no.
I believe you are right and the function was detoured.
ConEmu doesn't detour its own executable, so the problem might be in BitDefender.

@asser-dk

This comment has been minimized.

Show comment
Hide comment
@asser-dk

asser-dk Jan 19, 2017

@asser-dk I see BitDefender in the loaded modules. I'm almost sure it's not a problem, but can you run Bash/ConEmu without it to ensure?

Sorry for the late reply. I've uninstalled the insider update and I'm currently on insider build 14986 where everything seems to work fine. I had a bunch of other applications that failed as well so it could be possible that the installation of the update was somehow flawed on my machine.

asser-dk commented Jan 19, 2017

@asser-dk I see BitDefender in the loaded modules. I'm almost sure it's not a problem, but can you run Bash/ConEmu without it to ensure?

Sorry for the late reply. I've uninstalled the insider update and I'm currently on insider build 14986 where everything seems to work fine. I had a bunch of other applications that failed as well so it could be possible that the installation of the update was somehow flawed on my machine.

@nwykes

This comment has been minimized.

Show comment
Hide comment
@nwykes

nwykes Jan 19, 2017

As another datapoint, bash is working for me fine in insider build 15007 with 161206 and 170118 on two different machines. No Bitdefender.

nwykes commented Jan 19, 2017

As another datapoint, bash is working for me fine in insider build 15007 with 161206 and 170118 on two different machines. No Bitdefender.

@t-anjan

This comment has been minimized.

Show comment
Hide comment
@t-anjan

t-anjan Jan 28, 2017

I can confirm that the problem is indeed Bitdefender.

I am on 15019 Insider build. I had the same problem as the OP. Reading the posts here, I disabled BitDefender. And it just worked. I am able to reproduce the error every single time with Bitdefender turned on.

So, should we be reporting this to Bitdefender?

I should probably try replacing Bitdefender with another antivirus and check if the issue persists.

t-anjan commented Jan 28, 2017

I can confirm that the problem is indeed Bitdefender.

I am on 15019 Insider build. I had the same problem as the OP. Reading the posts here, I disabled BitDefender. And it just worked. I am able to reproduce the error every single time with Bitdefender turned on.

So, should we be reporting this to Bitdefender?

I should probably try replacing Bitdefender with another antivirus and check if the issue persists.

Maximus5 added a commit that referenced this issue Apr 25, 2017

gh-1012, gh-1121: Dirty workaround for AVDefender/Bitdefender bug rai…
…sing a crash in the RegQueryValueEx.

  Warning! ConEmu physically can't fix 3rd-paty bugs!
  This commit just skips process bitness detection if we already may be sure about it.

  But the bug/crash may appear in some other situations,
  so it's better to report the problem to the authors.

@Maximus5 Maximus5 closed this Apr 25, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment