Updating from one version of hosting bundle to next does not work.

[NXTwoThou]

Every single release has been unable to properly update the hosting bundle. After updating, no Core/5 sites function, they return http 500. I have to uninstall the components and reinstall for my sites to function. This has been an ongoing issue but I've bit my teeth and done the process of fixing it with each update and complained in the blog posts(with the 5.0 release, it was suggested I fill this out, since I'd already gone through the process of updating all three of our servers, I couldn't give you a "before" dotnet --info). Now that we're at 5.0 and going to be automatically updated, it absolutely needs to be fixed so Microsoft Update doesn't break our servers.

Before running dotnet-hosting-5.0.1-win.exe on Windows Server 2016 1607 14393.4104

C:\Windows\system32>dotnet --info

Host (useful for support):
Version: 5.0.0
Commit: cf258a14b7

.NET SDKs installed:
No SDKs were found.

.NET runtimes installed:
Microsoft.AspNetCore.App 5.0.0 [C:\Program Files (x86)\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.NETCore.App 5.0.0 [C:\Program Files (x86)\dotnet\shared\Microsoft.NETCore.App]

Blazor site works as expected.

After installing, site has a HTTP ERROR 500. IIS indicates site is running. New dotnet --info

Host (useful for support):
Version: 5.0.1
Commit: b02e13abab

.NET SDKs installed:
No SDKs were found.

.NET runtimes installed:
Microsoft.AspNetCore.App 5.0.1 [C:\Program Files (x86)\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.NETCore.App 5.0.1 [C:\Program Files (x86)\dotnet\shared\Microsoft.NETCore.App]

I try and run a repair, it completes, my site is still dead. Restarting site, restarting IIS, restarting server, nothing fixes it except for uninstalling and reinstalling.

I've only used .NET Core 2.2, 3.0, 3.1, and 5.0 and every single one has done this exact same broken behavior.

Here's dotnet --info AFTER uninstalling the three components and reinstalling the hosting bundle.

Host (useful for support):
Version: 5.0.1
Commit: b02e13abab

.NET SDKs installed:
No SDKs were found.

.NET runtimes installed:
Microsoft.AspNetCore.App 5.0.1 [C:\Program Files (x86)\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.NETCore.App 5.0.1 [C:\Program Files (x86)\dotnet\shared\Microsoft.NETCore.App]

Everything is back up and running.

EventLogs.zip

[BrennanConroy]

cc @shirhatti Any thoughts?

[shirhatti]

@NXTwoThou The only relevant event in the eventlogs indicates that the web.config might have been malformed during an attempted restart. Are you able get your machine into a bad state again? Or was this a one-off issue?

[NXTwoThou]

@NXTwoThou The only relevant event in the eventlogs indicates that the web.config might have been malformed during an attempted restart. Are you able get your machine into a bad state again? Or was this a one-off issue?

"Every single release has been unable to properly update the hosting bundle." represents me slapping together the site back with v2.2 and it being upgraded with each new release. So, whenever v5.0.2 comes out, it'll likely do it again.

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <location path="." inheritInChildApplications="false">
    <system.webServer>
      <handlers>
        <add name="aspNetCore" path="*" verb="*" modules="AspNetCoreModuleV2" resourceType="Unspecified" />
      </handlers>
      <aspNetCore processPath="dotnet" arguments=".\WinVoiceLanding.dll" stdoutLogEnabled="false" stdoutLogFile=".\logs\stdout" hostingModel="inprocess" />
    </system.webServer>
  </location>
</configuration>
<!--ProjectGuid: 0774a412-d5e6-459f-aeef-9fa32e6a7f8e-->

[shirhatti]

<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
    <System>
        <Provider Name="IIS AspNetCore Module V2" />
        <EventID Qualifiers="0">1034</EventID>
        <Level>2</Level>
        <Task>0</Task>
        <Keywords>0x80000000000000</Keywords>
        <TimeCreated SystemTime="2020-12-09T12:13:12.8571464Z" />
        <EventRecordID>63767</EventRecordID>
        <Channel>Application</Channel>
        <Computer>WEBSERVER3</Computer>
        <Security />
    </System>
    <EventData>
        <Data>Could not load configuration. Exception message: Unable to get required configuration section 'system.webServer/aspNetCore'. Possible reason is web.config authoring error.</Data>
        <Data>Process Id: 12616.</Data>
        <Data>File Version: 15.0.20336.1. Description: IIS ASP.NET Core Module V2. Commit: fc93e595ceffbb1e3e85532bf454e92a6a80dd6b</Data>
    </EventData>
</Event>

This is the only failure in the logs you're sharing. There are some known edge cases around installer upgrades. I've seen it manifest in either the ANCM schema being removed or .NET no longer being present on your PATH.

I'd love to get to the bottom of this, I just don't have much to go by.

👀 @joeloff @jamshedd

[shirhatti]

I don't have a hypothesis yet, but a few things that might help once your machine is in a bad state:

• applicationHost.config from C:\Windows\System32\inetsrv\config\applicationHost.config
• The system PATH
• The process environment block from WAS to see environment variables as seen by WAS when it started
• The contents of C:\Program Files\IIS\Asp.Net Core Module\
• More EventLogs (in case there are other failures)

[NXTwoThou]

I don't have a hypothesis yet, but a few things that might help once your machine is in a bad state:

• applicationHost.config from C:\Windows\System32\inetsrv\config\applicationHost.config
• The system PATH
• The process environment block from WAS to see environment variables as seen by WAS when it started
• The contents of C:\Program Files\IIS\Asp.Net Core Module\
• More EventLogs (in case there are other failures)

If you can give me instructions on "the process environment block from WAS", the other things are pretty clear. When v5.0.2(or whatever subversion) comes out, I'll upgrade, verify system is in a bad state, and collect all the info(then I need to get the servers functional again, I can't leave in a broken state).

As a fyi, these servers(3) only have one .NET(Core/v5) site. All the rest are v4.8 or old static sites. None of the other sites have any issue while the Core one is down.

[shirhatti]

I use procexp (from the SysInternals suite). Look for the parent process of w3wp.exe.

And then you can find the environment variables from there

We're mostly interested in the PATH variable from there

[NXTwoThou]

Path C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0;C:\Program Files\Mellanox\WinMFT;C:\Program Files\Microsoft\Web Platform Installer;C:\Program Files (x86)\dotnet;C:\Program Files\dotnet;C:\Users\Administrator\AppData\Local\Microsoft\WindowsApps;C:\Users\Administrator.dotnet\tools

Here's what it is now. During the next update I'll grab it wihile in the broken state.

[BrennanConroy]

Closing as there is nothing actionable right now, but please reopen next patch if you can repro!