Getting HTTP Error 503. with 0x800704e7 randomly

[zulander1]

Is there an existing issue for this?

• I have searched the existing issues

Describe the bug

I am have the exact same issue as this; The application is in a heavy traffic area and does a lot of network call... Randomly the app stop's with HTTP Error 503. The service is unavailable..

Any idea how I can find the root cause of this or fix it ?

the error log ends with this in the output:
info: Microsoft.Hosting.Lifetime[0]
Application is shutting down...

aspnetcore-debug.txt

[2023-01-06T04:09:44.557Z, PID: 15204] [aspnetcorev2.dll] Initializing logs for 'C:\Program Files\IIS\Asp.Net Core Module\V2\aspnetcorev2.dll'. Process Id: 15204. File Version: 16.0.22055.3. Description: IIS ASP.NET Core Module V2. Commit: c911002ab43b7b989ed67090f2a48d9073d5118d.
[2023-01-06T04:09:44.561Z, PID: 15204] [aspnetcorev2.dll] Resolving hostfxr parameters for application: 'dotnet' arguments: '.\app1.dll' path: 'c:\inetpub\wwwroot\site1\'
[2023-01-06T04:09:44.562Z, PID: 15204] [aspnetcorev2.dll] Known dotnet.exe location: ''
[2023-01-06T04:09:44.563Z, PID: 15204] [aspnetcorev2.dll] Process path 'dotnet.exe' is dotnet, treating application as portable
[2023-01-06T04:09:44.564Z, PID: 15204] [aspnetcorev2.dll] Resolving absolute path to dotnet.exe from 'dotnet.exe'
[2023-01-06T04:09:44.565Z, PID: 15204] [aspnetcorev2.dll] Invoking where.exe to find dotnet.exe
[2023-01-06T04:09:44.666Z, PID: 15204] [aspnetcorev2.dll] where.exe invocation returned: 'C:\Program Files\dotnet\dotnet.exe
C:\Program Files (x86)\dotnet\dotnet.exe
'
[2023-01-06T04:09:44.667Z, PID: 15204] [aspnetcorev2.dll] Current process bitness type detected as isX64=1
[2023-01-06T04:09:44.668Z, PID: 15204] [aspnetcorev2.dll] Processing entry 'C:\Program Files\dotnet\dotnet.exe'
[2023-01-06T04:09:44.669Z, PID: 15204] [aspnetcorev2.dll] Binary type 6
[2023-01-06T04:09:44.670Z, PID: 15204] [aspnetcorev2.dll] Found dotnet.exe via where.exe invocation at 'C:\Program Files\dotnet\dotnet.exe'
[2023-01-06T04:09:44.670Z, PID: 15204] [aspnetcorev2.dll] Resolving absolute path to hostfxr.dll from 'C:\Program Files\dotnet\dotnet.exe'
[2023-01-06T04:09:44.671Z, PID: 15204] [aspnetcorev2.dll] hostfxr.dll located at 'C:\Program Files\dotnet\host\fxr\6.0.3\hostfxr.dll'
[2023-01-06T04:09:44.672Z, PID: 15204] [aspnetcorev2.dll] Converted argument '.\app1.dll' to 'c:\inetpub\wwwroot\site1\.\app1.dll'
[2023-01-06T04:09:44.672Z, PID: 15204] [aspnetcorev2.dll] Parsed hostfxr options: dotnet location: 'C:\Program Files\dotnet\dotnet.exe' hostfxr path: 'C:\Program Files\dotnet\host\fxr\6.0.3\hostfxr.dll' arguments:
[2023-01-06T04:09:44.673Z, PID: 15204] [aspnetcorev2.dll] Argument[0] = 'C:\Program Files\dotnet\dotnet.exe'
[2023-01-06T04:09:44.674Z, PID: 15204] [aspnetcorev2.dll] Argument[1] = 'c:\inetpub\wwwroot\site1\.\app1.dll'
[2023-01-06T04:09:44.674Z, PID: 15204] [aspnetcorev2.dll] Loading hostfxr from location C:\Program Files\dotnet\host\fxr\6.0.3\hostfxr.dll
[2023-01-06T04:09:44.712Z, PID: 15204] [aspnetcorev2.dll] Canceling standard stream pipe reader
[2023-01-06T04:09:44.713Z, PID: 15204] [aspnetcorev2.dll] Loading request handler:  'C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\6.0.3\aspnetcorev2_inprocess.dll'
[2023-01-06T04:09:44.717Z, PID: 15204] [aspnetcorev2.dll] Creating handler application
[2023-01-06T04:09:44.719Z, PID: 15204] [aspnetcorev2_inprocess.dll] Initializing logs for 'C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\6.0.3\aspnetcorev2_inprocess.dll'. Process Id: 15204. File Version: 16.0.22055.3. Description: IIS ASP.NET Core Module V2 Request Handler. Commit: c911002ab43b7b989ed67090f2a48d9073d5118d.
[2023-01-06T04:09:44.720Z, PID: 15204] [aspnetcorev2_inprocess.dll] Waiting for initialization
[2023-01-06T04:09:44.721Z, PID: 15204] [aspnetcorev2_inprocess.dll] Starting app_offline monitoring in application 'c:\inetpub\wwwroot\site1\'
[2023-01-06T04:09:44.722Z, PID: 15204] [aspnetcorev2_inprocess.dll] Starting file watcher thread
[2023-01-06T04:09:44.722Z, PID: 15204] [aspnetcorev2_inprocess.dll] Starting in-process worker thread
[2023-01-06T04:09:44.723Z, PID: 15204] [aspnetcorev2_inprocess.dll] Resolving hostfxr parameters for application: 'dotnet' arguments: '.\app1.dll' path: 'c:\inetpub\wwwroot\site1\'
[2023-01-06T04:09:44.724Z, PID: 15204] [aspnetcorev2_inprocess.dll] Known dotnet.exe location: 'C:\Program Files\dotnet\dotnet.exe'
[2023-01-06T04:09:44.724Z, PID: 15204] [aspnetcorev2_inprocess.dll] Process path 'dotnet.exe' is dotnet, treating application as portable
[2023-01-06T04:09:44.725Z, PID: 15204] [aspnetcorev2_inprocess.dll] Resolving absolute path to hostfxr.dll from 'C:\Program Files\dotnet\dotnet.exe'
[2023-01-06T04:09:44.726Z, PID: 15204] [aspnetcorev2_inprocess.dll] hostfxr.dll located at 'C:\Program Files\dotnet\host\fxr\6.0.3\hostfxr.dll'
[2023-01-06T04:09:44.726Z, PID: 15204] [aspnetcorev2_inprocess.dll] Converted argument '.\app1.dll' to 'c:\inetpub\wwwroot\site1\.\app1.dll'
[2023-01-06T04:09:44.727Z, PID: 15204] [aspnetcorev2_inprocess.dll] Parsed hostfxr options: dotnet location: 'C:\Program Files\dotnet\dotnet.exe' hostfxr path: 'C:\Program Files\dotnet\host\fxr\6.0.3\hostfxr.dll' arguments:
[2023-01-06T04:09:44.728Z, PID: 15204] [aspnetcorev2_inprocess.dll] Argument[0] = 'C:\Program Files\dotnet\dotnet.exe'
[2023-01-06T04:09:44.728Z, PID: 15204] [aspnetcorev2_inprocess.dll] Argument[1] = 'c:\inetpub\wwwroot\site1\.\app1.dll'
[2023-01-06T04:09:44.729Z, PID: 15204] [aspnetcorev2_inprocess.dll] Setting environment variable ASPNETCORE_IIS_HTTPAUTH=anonymous;
[2023-01-06T04:09:44.729Z, PID: 15204] [aspnetcorev2_inprocess.dll] Setting environment variable ASPNETCORE_IIS_PHYSICAL_PATH=c:\inetpub\wwwroot\site1\
[2023-01-06T04:09:44.730Z, PID: 15204] [aspnetcorev2_inprocess.dll] Loading hostfxr from location C:\Program Files\dotnet\host\fxr\6.0.3\hostfxr.dll
[2023-01-06T04:09:44.731Z, PID: 15204] [aspnetcorev2_inprocess.dll] Initial Dll directory: '', current directory: 'c:\windows\system32\inetsrv'
[2023-01-06T04:09:44.732Z, PID: 15204] [aspnetcorev2_inprocess.dll] Setting dll directory to c:\windows\system32\inetsrv
[2023-01-06T04:09:44.732Z, PID: 15204] [aspnetcorev2_inprocess.dll] Setting current directory to c:\inetpub\wwwroot\site1\
[2023-01-06T04:09:47.846Z, PID: 15204] [aspnetcorev2_inprocess.dll] In-process callbacks set
[2023-01-06T04:09:47.847Z, PID: 15204] [aspnetcorev2_inprocess.dll] Event Log: 'Application 'c:\inetpub\wwwroot\site1\' started successfully.' 
End Event Log Message.
[2023-01-07T02:23:21.737Z, PID: 15204] [aspnetcorev2.dll] ASPNET_CORE_GLOBAL_MODULE::OnGlobalStopListening
[2023-01-07T02:23:21.740Z, PID: 15204] [aspnetcorev2.dll] Stopping application '/LM/W3SVC/1093447982/ROOT'
[2023-01-07T02:23:21.741Z, PID: 15204] [aspnetcorev2_inprocess.dll] Stopping file watching.
[2023-01-07T02:23:21.742Z, PID: 15204] [aspnetcorev2_inprocess.dll] Stopping file watcher thread
[2023-01-07T02:23:21.742Z, PID: 15204] [aspnetcorev2_inprocess.dll] Stopping CLR
[2023-01-07T02:23:21.744Z, PID: 15204] [aspnetcorev2_inprocess.dll] Starting shutdown sequence 0
[2023-01-07T02:23:21.818Z, PID: 15204] [aspnetcorev2.dll] Failed HRESULT returned: 0x800704e7 at D:\a\_work\1\s\src\Servers\IIS\AspNetCoreModuleV2\AspNetCore\proxymodule.cpp:96 
[2023-01-07T02:23:21.818Z, PID: 15204] [aspnetcorev2.dll] Failed HRESULT returned: 0x800704e7 at D:\a\_work\1\s\src\Servers\IIS\AspNetCoreModuleV2\AspNetCore\proxymodule.cpp:96 
[2023-01-07T02:23:21.820Z, PID: 15204] [aspnetcorev2.dll] Failed HRESULT returned: 0x800704e7 at D:\a\_work\1\s\src\Servers\IIS\AspNetCoreModuleV2\AspNetCore\proxymodule.cpp:120 
[2023-01-07T02:23:21.819Z, PID: 15204] [aspnetcorev2.dll] Failed HRESULT returned: 0x800704e7 at D:\a\_work\1\s\src\Servers\IIS\AspNetCoreModuleV2\AspNetCore\proxymodule.cpp:120 
[2023-01-07T02:23:21.818Z, PID: 15204] [aspnetcorev2.dll] Failed HRESULT returned: 0x800704e7 at D:\a\_work\1\s\src\Servers\IIS\AspNetCoreModuleV2\AspNetCore\proxymodule.cpp:96 
[2023-01-07T02:23:21.822Z, PID: 15204] [aspnetcorev2.dll] Failed HRESULT returned: 0x800704e7 at D:\a\_work\1\s\src\Servers\IIS\AspNetCoreModuleV2\AspNetCore\proxymodule.cpp:120 
[2023-01-07T02:23:21.980Z, PID: 15204] [aspnetcorev2.dll] Failed HRESULT returned: 0x800704e7 at D:\a\_work\1\s\src\Servers\IIS\AspNetCoreModuleV2\AspNetCore\proxymodule.cpp:96 
[2023-01-07T02:23:21.981Z, PID: 15204] [aspnetcorev2.dll] Failed HRESULT returned: 0x800704e7 at D:\a\_work\1\s\src\Servers\IIS\AspNetCoreModuleV2\AspNetCore\proxymodule.cpp:120 
[2023-01-07T02:23:22.009Z, PID: 15204] [aspnetcorev2.dll] Failed HRESULT returned: 0x800704e7 at D:\a\_work\1\s\src\Servers\IIS\AspNetCoreModuleV2\AspNetCore\proxymodule.cpp:96 
[2023-01-07T02:23:22.010Z, PID: 15204] [aspnetcorev2.dll] Failed HRESULT returned: 0x800704e7 at D:\a\_work\1\s\src\Servers\IIS\AspNetCoreModuleV2\AspNetCore\proxymodule.cpp:120 
[2023-01-07T02:23:22.027Z, PID: 15204] [aspnetcorev2.dll] Failed HRESULT returned: 0x800704e7 at D:\a\_work\1\s\src\Servers\IIS\AspNetCoreModuleV2\AspNetCore\proxymodule.cpp:96 
[2023-01-07T02:23:22.028Z, PID: 15204] [aspnetcorev2.dll] Failed HRESULT returned: 0x800704e7 at D:\a\_work\1\s\src\Servers\IIS\AspNetCoreModuleV2\AspNetCore\proxymodule.cpp:120 
[2023-01-07T02:23:22.325Z, PID: 15204] [aspnetcorev2.dll] Failed HRESULT returned: 0x800704e7 at D:\a\_work\1\s\src\Servers\IIS\AspNetCoreModuleV2\AspNetCore\proxymodule.cpp:96 
[2023-01-07T02:23:22.326Z, PID: 15204] [aspnetcorev2.dll] Failed HRESULT returned: 0x800704e7 at D:\a\_work\1\s\src\Servers\IIS\AspNetCoreModuleV2\AspNetCore\proxymodule.cpp:120 
[2023-01-07T02:23:22.367Z, PID: 15204] [aspnetcorev2.dll] Failed HRESULT returned: 0x800704e7 at D:\a\_work\1\s\src\Servers\IIS\AspNetCoreModuleV2\AspNetCore\proxymodule.cpp:96 
[2023-01-07T02:23:22.368Z, PID: 15204] [aspnetcorev2.dll] Failed HRESULT returned: 0x800704e7 at D:\a\_work\1\s\src\Servers\IIS\AspNetCoreModuleV2\AspNetCore\proxymodule.cpp:120 
[2023-01-07T02:23:22.455Z, PID: 15204] [aspnetcorev2.dll] Failed HRESULT returned: 0x800704e7 at D:\a\_work\1\s\src\Servers\IIS\AspNetCoreModuleV2\AspNetCore\proxymodule.cpp:96 
[2023-01-07T02:23:22.456Z, PID: 15204] [aspnetcorev2.dll] Failed HRESULT returned: 0x800704e7 at D:\a\_work\1\s\src\Servers\IIS\AspNetCoreModuleV2\AspNetCore\proxymodule.cpp:120 
[2023-01-07T02:23:22.491Z, PID: 15204] [aspnetcorev2.dll] Failed HRESULT returned: 0x800704e7 at D:\a\_work\1\s\src\Servers\IIS\AspNetCoreModuleV2\AspNetCore\proxymodule.cpp:96 
[2023-01-07T02:23:22.492Z, PID: 15204] [aspnetcorev2.dll] Failed HRESULT returned: 0x800704e7 at D:\a\_work\1\s\src\Servers\IIS\AspNetCoreModuleV2\AspNetCore\proxymodule.cpp:120 
[2023-01-07T02:23:22.527Z, PID: 15204] [aspnetcorev2.dll] Failed HRESULT returned: 0x800704e7 at D:\a\_work\1\s\src\Servers\IIS\AspNetCoreModuleV2\AspNetCore\proxymodule.cpp:96 
[2023-01-07T02:23:22.528Z, PID: 15204] [aspnetcorev2.dll] Failed HRESULT returned: 0x800704e7 at D:\a\_work\1\s\src\Servers\IIS\AspNetCoreModuleV2\AspNetCore\proxymodule.cpp:120 

Further technical details
ASP.NET Core version: 6.0.3
IIS version 10.0.xx, Windows Server 2016
Include the output of dotnet --info:

Expected Behavior

No response

Steps To Reproduce

No response

Exceptions (if any)

No response

.NET Version

Version: 6.0.3

Anything else?

Host (useful for support):
  Version: 6.0.3
  Commit:  c24d9a9c91

.NET SDKs installed:
  No SDKs were found.

.NET runtimes installed:
  Microsoft.AspNetCore.All 2.2.4 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  [Microsoft.AspNetCore.App](http://microsoft.aspnetcore.app/) 2.2.4 [C:\Program Files\dotnet\shared\[Microsoft.AspNetCore.App](http://microsoft.aspnetcore.app/)]
  [Microsoft.AspNetCore.App](http://microsoft.aspnetcore.app/) 6.0.3 [C:\Program Files\dotnet\shared\[Microsoft.AspNetCore.App](http://microsoft.aspnetcore.app/)]
  [Microsoft.NETCore.App](http://microsoft.netcore.app/) 2.0.7 [C:\Program Files\dotnet\shared\[Microsoft.NETCore.App](http://microsoft.netcore.app/)]
  [Microsoft.NETCore.App](http://microsoft.netcore.app/) 2.2.4 [C:\Program Files\dotnet\shared\[Microsoft.NETCore.App](http://microsoft.netcore.app/)]
  [Microsoft.NETCore.App](http://microsoft.netcore.app/) 6.0.3 [C:\Program Files\dotnet\shared\[Microsoft.NETCore.App](http://microsoft.netcore.app/)]

[adityamandaleeka]

It looks like the logs you pasted might be missing some stuff between

[2023-01-06T04:09:47.847Z, PID: 15204] [aspnetcorev2_inprocess.dll] Event Log: 'Application 'c:\inetpub\wwwroot\site1\' started successfully.' 
End Event Log Message.
[2023-01-07T02:23:21.737Z, PID: 15204] [aspnetcorev2.dll] ASPNET_CORE_GLOBAL_MODULE::OnGlobalStopListening

Perhaps there's more there that could explain why it's stopping the server?

[zulander1]

It looks like the logs you pasted might be missing some stuff between

[2023-01-06T04:09:47.847Z, PID: 15204] [aspnetcorev2_inprocess.dll] Event Log: 'Application 'c:\inetpub\wwwroot\site1\' started successfully.' 
End Event Log Message.
[2023-01-07T02:23:21.737Z, PID: 15204] [aspnetcorev2.dll] ASPNET_CORE_GLOBAL_MODULE::OnGlobalStopListening

Perhaps there's more there that could explain why it's stopping the server?

No there is no other log, double checked

[zulander1]

I was thinking, can this caused by the IIS pool Queue Length?
When the maximum number of requests in the queue has been reached, the service returns an out-of-service message (503: Service Unavailable) to any additional requests to the current application pool.

[BrennanConroy]

There are two things here, 0x800704e7 errors and your app shutting down.

The 0x800704e7 errors are expected, they represent incoming requests during the app shutting down. These aren't interesting for this issue.

What you want is to figure out why the app is shutting down in the first place. From the logs it looks like a graceful close caused by something unrelated to ASP.NET. Which makes me think it's likely an app pool recycle. These can be triggered for various reasons, there is a detailed article that describes all the different ways a recycle can be triggered, as well as ways to log events on recycle:
https://learn.microsoft.com/previous-versions/windows/it-pro/windows-server-2008-R2-and-2008/cc753179(v=ws.10)

Hi @zulander1. We have added the "Needs: Author Feedback" label to this issue, which indicates that we have an open question for you before we can take further action. This issue will be closed automatically in 7 days if we do not hear back from you by then - please feel free to re-open it if you come back to this issue after that time.

[zulander1]

@BrennanConroy I will follow your advice, and look into the log to see why the IIS pool is being recycled. Thanks 🙏