self-contained publish app crashes with missing files after update .net7.0.3 -> 7.0.4

[schhwork]

First off since this error realtes to the runtime files from the selfcontained publish this seemd like the appropriate place to post my issue. If this is wrong pls direct me to the right place. Also its my first issue so apologies if im missing any mandatory information i should have included.

Problem: After azuredevops latest windows image and a got updated my asp.net app crashed with the following exceptions:

System.IO.FileNotFoundException: Could not load file or assembly 'Microsoft.Net.Http.Headers, Version=7.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'. The system cannot find the file specified.
File name: 'Microsoft.Net.Http.Headers, Version=7.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.DateHeaderValueManager.SetDateValues(DateTimeOffset value)
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.Heartbeat.OnHeartbeat()

System.IO.FileNotFoundException: Could not load file or assembly 'System.Net.Security, Version=7.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'. The system cannot find the file specified.
File name: 'System.Net.Security, Version=7.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'
at Microsoft.AspNetCore.Server.Kestrel.KestrelConfigurationLoader.Reload()
at Microsoft.AspNetCore.Server.Kestrel.KestrelConfigurationLoader.Load()
at Microsoft.AspNetCore.Server.Kestrel.Core.KestrelServerImpl.BindAsync(CancellationToken cancellationToken)
at Microsoft.AspNetCore.Server.Kestrel.Core.KestrelServerImpl.StartAsync[TContext](IHttpApplication`1 application, CancellationToken cancellationToken)
at Microsoft.AspNetCore.Hosting.GenericWebHostService.StartAsync(CancellationToken cancellationToken)
at Microsoft.Extensions.Hosting.Internal.Host.StartAsync(CancellationToken cancellationToken)
at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.Run(IHost host)
at EDP.Mobile.Server.Program.Main(String[] args)

I can reproduce this with a selfcontained pusblish on my local machine aswell. If i do a framework dependant publish everything works.

The build from 2 days prior with the old image works fine. No changes to any nuget packages/dependencies has been changed.
If i look in my publish output the files in questions are there but with different versions.

Old Microsoft.Net.Http.Headers file version: 7.0.323.8009, New version: 7.0.4.423.11903
Same versions with System.net.Security.

Environment:

Its a kestrel grpc windowsservice project targetting .net7-windows

dotnet --info

.NET SDK:
Version: 7.0.202
Commit: 6c74320bc3

Runtime Environment:
OS Name: Windows
OS Version: 10.0.19043
OS Platform: Windows
RID: win10-x64
Base Path: C:\Program Files\dotnet\sdk\7.0.202\

Host:
Version: 7.0.4
Architecture: x64
Commit: 0a396ac

.NET SDKs installed:
2.1.526 [C:\Program Files\dotnet\sdk]
2.1.818 [C:\Program Files\dotnet\sdk]
3.1.116 [C:\Program Files\dotnet\sdk]
3.1.421 [C:\Program Files\dotnet\sdk]
5.0.214 [C:\Program Files\dotnet\sdk]
5.0.301 [C:\Program Files\dotnet\sdk]
5.0.408 [C:\Program Files\dotnet\sdk]
7.0.202 [C:\Program Files\dotnet\sdk]

.NET runtimes installed:
Microsoft.AspNetCore.All 2.1.28 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
Microsoft.AspNetCore.All 2.1.30 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
Microsoft.AspNetCore.App 2.1.28 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 2.1.30 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 3.1.16 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 3.1.27 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 5.0.7 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 5.0.17 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 6.0.15 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 7.0.4 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.NETCore.App 2.1.28 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.1.30 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 3.1.16 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 3.1.27 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 5.0.7 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 5.0.17 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 6.0.15 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 7.0.4 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.WindowsDesktop.App 3.1.16 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Microsoft.WindowsDesktop.App 3.1.27 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Microsoft.WindowsDesktop.App 5.0.7 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Microsoft.WindowsDesktop.App 5.0.17 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Microsoft.WindowsDesktop.App 6.0.15 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Microsoft.WindowsDesktop.App 7.0.4 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]

Other architectures found:
arm64 [C:\Program Files\dotnet]
registered at [HKLM\SOFTWARE\dotnet\Setup\InstalledVersions\arm64\InstallLocation]
x86 [C:\Program Files (x86)\dotnet]
registered at [HKLM\SOFTWARE\dotnet\Setup\InstalledVersions\x86\InstallLocation]

Visual Studio 17.5.3

[jeffschwMSFT]

@schh can you share the project file for the self contained app and publish settings?

Tagging subscribers to this area: @vitek-karas, @agocke, @VSadov
See info in area-owners.md if you want to be subscribed.

Issue Details

---

First off since this error realtes to the runtime files from the selfcontained publish this seemd like the appropriate place to post my issue. If this is wrong pls direct me to the right place. Also its my first issue so apologies if im missing any mandatory information i should have included.

Problem: After azuredevops latest windows image and a got updated my asp.net app crashed with the following exceptions:

System.IO.FileNotFoundException: Could not load file or assembly 'Microsoft.Net.Http.Headers, Version=7.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'. The system cannot find the file specified.
File name: 'Microsoft.Net.Http.Headers, Version=7.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.DateHeaderValueManager.SetDateValues(DateTimeOffset value)
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.Heartbeat.OnHeartbeat()

System.IO.FileNotFoundException: Could not load file or assembly 'System.Net.Security, Version=7.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'. The system cannot find the file specified.
File name: 'System.Net.Security, Version=7.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'
at Microsoft.AspNetCore.Server.Kestrel.KestrelConfigurationLoader.Reload()
at Microsoft.AspNetCore.Server.Kestrel.KestrelConfigurationLoader.Load()
at Microsoft.AspNetCore.Server.Kestrel.Core.KestrelServerImpl.BindAsync(CancellationToken cancellationToken)
at Microsoft.AspNetCore.Server.Kestrel.Core.KestrelServerImpl.StartAsync[TContext](IHttpApplication`1 application, CancellationToken cancellationToken)
at Microsoft.AspNetCore.Hosting.GenericWebHostService.StartAsync(CancellationToken cancellationToken)
at Microsoft.Extensions.Hosting.Internal.Host.StartAsync(CancellationToken cancellationToken)
at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.Run(IHost host)
at EDP.Mobile.Server.Program.Main(String[] args)

I can reproduce this with a selfcontained pusblish on my local machine aswell. If i do a framework dependant publish everything works.

The build from 2 days prior with the old image works fine. No changes to any nuget packages/dependencies has been changed.
If i look in my publish output the files in questions are there but with different versions.

Old Microsoft.Net.Http.Headers file version: 7.0.323.8009, New version: 7.0.4.423.11903
Same versions with System.net.Security.

Environment:

Its a kestrel grpc windowsservice project targetting .net7-windows

dotnet --info

.NET SDK:
Version: 7.0.202
Commit: 6c74320bc3

Runtime Environment:
OS Name: Windows
OS Version: 10.0.19043
OS Platform: Windows
RID: win10-x64
Base Path: C:\Program Files\dotnet\sdk\7.0.202\

Host:
Version: 7.0.4
Architecture: x64
Commit: 0a396ac

.NET SDKs installed:
2.1.526 [C:\Program Files\dotnet\sdk]
2.1.818 [C:\Program Files\dotnet\sdk]
3.1.116 [C:\Program Files\dotnet\sdk]
3.1.421 [C:\Program Files\dotnet\sdk]
5.0.214 [C:\Program Files\dotnet\sdk]
5.0.301 [C:\Program Files\dotnet\sdk]
5.0.408 [C:\Program Files\dotnet\sdk]
7.0.202 [C:\Program Files\dotnet\sdk]

.NET runtimes installed:
Microsoft.AspNetCore.All 2.1.28 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
Microsoft.AspNetCore.All 2.1.30 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
Microsoft.AspNetCore.App 2.1.28 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 2.1.30 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 3.1.16 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 3.1.27 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 5.0.7 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 5.0.17 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 6.0.15 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 7.0.4 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.NETCore.App 2.1.28 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.1.30 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 3.1.16 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 3.1.27 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 5.0.7 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 5.0.17 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 6.0.15 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 7.0.4 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.WindowsDesktop.App 3.1.16 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Microsoft.WindowsDesktop.App 3.1.27 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Microsoft.WindowsDesktop.App 5.0.7 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Microsoft.WindowsDesktop.App 5.0.17 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Microsoft.WindowsDesktop.App 6.0.15 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Microsoft.WindowsDesktop.App 7.0.4 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]

Other architectures found:
arm64 [C:\Program Files\dotnet]
registered at [HKLM\SOFTWARE\dotnet\Setup\InstalledVersions\arm64\InstallLocation]
x86 [C:\Program Files (x86)\dotnet]
registered at [HKLM\SOFTWARE\dotnet\Setup\InstalledVersions\x86\InstallLocation]

Visual Studio 17.5.3

Author:	schh
Assignees:	-
Labels:	area-Host, untriaged, needs-area-label
Milestone:	-

[schhwork]

Publishcommand:

"C:\Program Files\dotnet\dotnet.exe" publish D:\a\1\s\src\Server\Server.csproj -c release -r win-x64 --self-contained --output D:\a\1\s\output\publish\Server

Attached projfile as .txt.

Server.txt

Also of note is that both my WPF projects that are also .net7-windows both crash now. No logs from the apps just a less than helpful eventlog:

Faulting application name: Mobile.exe, version: 2.11.0.0, time stamp: 0x63ed82c1
Faulting module name: KERNELBASE.dll, version: 10.0.19041.2364, time stamp: 0x5b7d4d22
Exception code: 0xe0434352
Fault offset: 0x000000000002cd29
Faulting process id: 0x37a8
Faulting application start time: 0x01d967db99ae7fac
Faulting application path: C:\git\EDP.Mobile\output\client\fordon\publish\win-x64\Mobile.exe
Faulting module path: C:\WINDOWS\System32\KERNELBASE.dll
Report Id: 4c80bd39-cd4a-4dba-bfae-7d5647c3ea73
Faulting package full name:
Faulting package-relative application ID:

[agocke]

Strange... System.Net.Security should be included in the framework and published to your output directory when publishing as self-contained.

Can you check to see if that DLL exists next to your app and what version it is?

Also, it would be great if we could get a binlog. The binlog will contain environment variables, so double check that there's nothing confidential in your environment variables (see https://msbuildlog.com/#security for the info it contains). You can also send it to me directly if you don't want to upload it.

[vitek-karas]

GIven the package references in the project file I have a theory, but I still don't know why it would fail.

The app carries Microsoft.Extensions.Hosting version 7.0.1 which is the exact same file/assembly version as that file shipping in 7.0.4 ASP.NET Framework. So on 7.0.3 the app should us the assemblies it carries with itself, but when executed on 7.0.4 it should switch to the file from the framework.

So to validate that what would help is the host tract of the FDD app (COREHOST_TRACE=1, COREHOST_TRACEFILE=host.txt, run the app - the host.txt file will contain detailed logs). The interesting part would be where did the host resolve the Microsoft.Extensions.Hosting from (framework or app) - this would be visible in the TRUSTED_PLATFORM_ASSEMBLIES property (written out near the very end of that trace).

But that still doesn't tell why it would fail to load at runtime. For that we would probably need a binder trace - https://learn.microsoft.com/en-us/dotnet/core/dependency-loading/collect-details.

Both the host trace and the biner trace may contain environment variable and so on, so please be careful to check before sharing them anywhere.

Or ideally get a repro.

/cc @elinor-fung

[schhwork]

Strange... System.Net.Security should be included in the framework and published to your output directory when publishing as self-contained.

Can you check to see if that DLL exists next to your app and what version it is?

Also, it would be great if we could get a binlog. The binlog will contain environment variables, so double check that there's nothing confidential in your environment variables (see https://msbuildlog.com/#security for the info it contains). You can also send it to me directly if you don't want to upload it.

GIven the package references in the project file I have a theory, but I still don't know why it would fail.

The app carries Microsoft.Extensions.Hosting version 7.0.1 which is the exact same file/assembly version as that file shipping in 7.0.4 ASP.NET Framework. So on 7.0.3 the app should us the assemblies it carries with itself, but when executed on 7.0.4 it should switch to the file from the framework.

So to validate that what would help is the host tract of the FDD app (COREHOST_TRACE=1, COREHOST_TRACEFILE=host.txt, run the app - the host.txt file will contain detailed logs). The interesting part would be where did the host resolve the Microsoft.Extensions.Hosting from (framework or app) - this would be visible in the TRUSTED_PLATFORM_ASSEMBLIES property (written out near the very end of that trace).

But that still doesn't tell why it would fail to load at runtime. For that we would probably need a binder trace - https://learn.microsoft.com/en-us/dotnet/core/dependency-loading/collect-details.

Both the host trace and the biner trace may contain environment variable and so on, so please be careful to check before sharing them anywhere.

Or ideally get a repro.

/cc @elinor-fung

The files are located in my publish output folder for selfcontained so im guessing this is just an issue with the version of the file not being what the app expects.

The TRUSTED_PLATFORM_ASSEMBLIES contains the following for the Microsoft.Extensions.Hosting.dll

The selfcontained app points to the publishfolder for this file and the framework dependant variety point to the installed one in C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\7.0.4. Both files are identical.

I have also created a bindertrace and complied a summary of all assembly loaded events from perfview for both the self-contained app and the framework dependant one. Ill attach as txt again but its in xml format.

assemblyloader_events.txt

[elinor-fung]

The events show that we are successfully finding and opening System.Net.Security from the publish folder when probing for known paths. But we report AssemblyNotFound as the end result of looking in app assemblies. (I suspect there's a separate bug in the tracing where we can do some additional probing that ends up losing the reason we rejected the assembly from the known path.)

I wonder if this could be related to #83526. That is the change I know of that affected assembly loading between 7.0.3 and 7.0.4. There should be a System.Reflection.Metadata.dll in the publish folder for your self-contained app. As a possible check, you try replacing the 7.0.4 one with the 7.0.3 version?

[schhwork]

The events show that we are successfully finding and opening System.Net.Security from the publish folder when probing for known paths. But we report AssemblyNotFound as the end result of looking in app assemblies. (I suspect there's a separate bug in the tracing where we can do some additional probing that ends up losing the reason we rejected the assembly from the known path.)

I wonder if this could be related to #83526. That is the change I know of that affected assembly loading between 7.0.3 and 7.0.4. There should be a System.Reflection.Metadata.dll in the publish folder for your self-contained app. As a possible check, you try replacing the 7.0.4 one with the 7.0.3 version?

Yes replacing the 7.0.4 version of System.Reflection.Metadata.dll with 7.0.3 seems to work. Now my app starts.

[elinor-fung]

Thanks for checking that. I'm going to close this against #83526 as that is main issue tracking the fix and has more discussion. @VSadov - the fix for that should by in 7.0.6 (May release), right?