IIS crashes with AV time to time after installing latest ANCM

[AndreiGorlov]

ASP.NET Core 2.0 app, IIS 8.5 on Windows Server 2012 R2 with latest updates, aspnetcore.dll file version 7.1.1982.0. IIS crashes every 1-3 hours.

After rollback to previous version of ANCM (uninstalling DotNetCore.2.0.0-WindowsHosting.exe and installing DotNetCore.1.0.7_1.1.4-WindowsHosting.exe (aspnetcore.dll file version 7.1.1972.0) - no crashes for 20+ hours.

Event log error example:

Faulting application name: w3wp.exe, version: 8.5.9600.16384, time stamp: 0x5215df96
Faulting module name: aspnetcore.dll, version: 7.1.1982.0, time stamp: 0x594ab904
Exception code: 0xc0000005
Fault offset: 0x000000000000f8a9
Faulting process id: 0x1670
Faulting application start time: 0x01d335123bc5b2f6
Faulting application path: c:\windows\system32\inetsrv\w3wp.exe
Faulting module path: C:\Windows\system32\inetsrv\aspnetcore.dll
Report Id: 9a753da7-a107-11e7-80ec-00155d000301
Faulting package full name: 
Faulting package-relative application ID: 

DebugDiag Analysis Report with 3 crashes: MultipleDumps_CrashHangAnalysis (5).zip

I can share DebugDiag .dmp files if necessary.

[jkotalik]

@pan-wang @shirhatti Seems to be a lot of issues with ANCM on Win Server 2012, could be platform specific?

[pan-wang]

@AndreiGorlov Could please provide a dump? There was a known AV for Websocket scenario when client disconnected without handshake.

[AndreiGorlov]

@pan-wang dumps: https://drive.google.com/open?id=0B6nsyqRBUOTkbEstOF82ZGJmYlk

There was a known AV for Websocket scenario when client disconnected without handshake.

My app uses signalr (and websockets).

Now I'm trying to build minimal repro. Already cut off everything related to signalr/websockets, installed ANCM 2.0 - IIS started crashing again.
Here is new dumps (app without websockets stuff): https://drive.google.com/open?id=0B6nsyqRBUOTkejAyVi03eEtUT1U

[pan-wang]

@AndreiGorlov I looked at the dump. It seems for a post request, somehow IIS notification context which is used to track the request state in IIS pipeline became NULL and thus caused AV. Based on the dump, it is impossible to tell which module cleaned the notification context. Do you have other custom IIS module installed? Could you please share a repro app so that I can debug it in-house. Otherwise, we may need your help to collect idna trace.

[AndreiGorlov]

@pan-wang I'm have repro. Аfraid to publish because of denial-of-service capabilities. How can I contact you privately?

@jkotalik Not platform specific, reproduced on WinSrv 2016/IIS 10.0 as well.

[pan-wang]

@AndreiGorlov you can send me email at panwang @ Microsoft . com. Thank you so much!

[AndreiGorlov]

@pan-wang Mail sent.

[sepehr1014]

We had the exact same issue & downgrading to 7.1.1972.0 fixed our problem. This is a severe bug requiring immediate action.

[mtinnes]

Same issue with the same server configuration:

Faulting application name: w3wp.exe, version: 8.5.9600.16384, time stamp: 0x5215df96
Faulting module name: aspnetcore.dll, version: 7.1.1988.0, time stamp: 0x5a2eeca2
Exception code: 0xc0000005
Fault offset: 0x000000000000d990
Faulting process id: 0x4a7c
Faulting application start time: 0x01d378f6e76fa57f
Faulting application path: c:\windows\system32\inetsrv\w3wp.exe
Faulting module path: C:\Windows\system32\inetsrv\aspnetcore.dll
Report Id: 2b46a68a-e4ea-11e7-80e0-a0369fbe0782
Faulting package full name: 
Faulting package-relative application ID: 


Fault bucket , type 0
Event Name: APPCRASH
Response: Not available
Cab Id: 0

Problem signature:
P1: w3wp.exe
P2: 8.5.9600.16384
P3: 5215df96
P4: aspnetcore.dll
P5: 7.1.1988.0
P6: 5a2eeca2
P7: c0000005
P8: 000000000000d990
P9: 
P10: 

These files may be available here:
C:\ProgramData\Microsoft\Windows\WER\ReportQueue\AppCrash_w3wp.exe_e3c0a787166295e9ef249eac737b0f57d9466d5_9e3fd63b_cab_17018a43

Analysis symbol: 
Rechecking for solution: 0
Report Id: 2ab06f4f-e4ea-11e7-80e0-a0369fbe0782
Report Status: 4
Hashed bucket: 

[OracPrime]

If it helps, it's not Server 2012 specific - I'm getting it on Windows 10.

And sadly on Windows 10 (admittedly with a site making heavy use of signalr) it's happening several times an hour [often when the site isn't even being used]

[OracPrime]

I'm also finding it hard to work around. The new VS2017 publish process expects to find the 2.0 store on the hosting machine for the app. If I roll back the hosting to 1.1 (by removing 2.0.3 hosting, I'm assuming) then it removes the 2.0 store and the application won't start up.

[AndreiGorlov]

@OracPrime try to install 2.0.3 hosting and replace aspnetcore.dll in %SystemRoot%\system32\inetsrv with 1.1 one.

[OracPrime]

@AndreiGorlov thanks for the tip. However I'm finding it hard to find the 1.1 version of aspnetcore.dll anywhere. Even if I uninstall the 2.0.3 hosting and reinstall 1.1 I end up with the 7.1.1982.0 build.

[sepehr1014]

@OracPrime When can we except this to be fixed? Are you aware of the team's plans?