Issue with System.Globalization.AppLocalIcu-setting suddenly appears

[enkelmedia]

Hi!

We're using Hangfire 1.7.31 on a ASP.NET 6 web app and Umbraco CMS 10.2.0. We're running the workers as background services in the web application.

Umbraco ships with some settings in the csproj file (presumably to unify culture behavior across different hosting environments).

<ItemGroup>
     <PackageReference Include="Microsoft.ICU.ICU4C.Runtime" Version="68.2.0.9" />
      <RuntimeHostConfigurationOption Include="System.Globalization.AppLocalIcu" Value="68.2.0.9" Condition="$(RuntimeIdentifier.StartsWith('linux')) Or $(RuntimeIdentifier.StartsWith('win')) Or ('$(RuntimeIdentifier)' == '' And !$([MSBuild]::IsOSPlatform('osx')))" />
</ItemGroup>

This has worked great up until this morning, it worked yesterday when I left work but now the site wouldn't start.

When I disabled the "Hangfire Background Service" by commenting out services.AddHangfireServer() the site started just fine again. The same site worked fine for my colleague until his Windows Update ran today and now he has the same issue. I'm on Windows 10 and he is on Windows 11.

Not 100% sure but it seems that this update causes some issues:
https://support.microsoft.com/en-us/topic/october-11-2022-kb5018410-os-builds-19042-2130-19043-2130-and-19044-2130-6390f057-28ca-43d3-92ce-f4b79a8378fd

I managed to replicate this issue without having Umbraco installed by adding the lines above to a "vanilla" ASP.NET Core website csproj with Hangfire installed. Without these lines, everything works just fine, after adding them the site won't start. There are no errors in the logs - the site just starts up, the browser is shown and then it dies.

However, Event Viewer logs an error, here is an example from my test project (vanilla asp.net core):

Faulting application name: HangfireIssue.exe, version: 1.0.0.0, time stamp: 0x6331eb0e
Faulting module name: ntdll.dll, version: 10.0.19041.2130, time stamp: 0xb5ced1c6
Exception code: 0xc0000409
Fault offset: 0x000000000008c67f
Faulting process id: 0xa634
Faulting application start time: 0x01d8de1cdf986bfb
Faulting application path: C:\Development\PlayGround\HangfireIssue\HangfireIssue\HangfireIssue\HangfireIssue\bin\Debug\net6.0\HangfireIssue.exe
Faulting module path: C:\WINDOWS\SYSTEM32\ntdll.dll
Report Id: bf195e95-5e8d-4b78-8770-56e61347da5c
Faulting package full name: 
Faulting package-relative application ID: 

After removing this line:

      <RuntimeHostConfigurationOption Include="System.Globalization.AppLocalIcu" Value="68.2.0.9" Condition="$(RuntimeIdentifier.StartsWith('linux')) Or $(RuntimeIdentifier.StartsWith('win')) Or ('$(RuntimeIdentifier)' == '' And !$([MSBuild]::IsOSPlatform('osx')))" />

Things work as expected again.

I'm really not sure if we really need this setting or if this is expected behavior but I wanted to share the problem here so that others that face the same issue might find a workaround and maybe if there is something in the code base that could be adjusted to work with these settings.

Please let me know if you need any further information or if you want the project files to replicate it.

Cheers!

[nul800sebastiaan]

I can report the same behavior, different site, similar setup and removing the AppLocalIcu reference it indeed works again.

[nul800sebastiaan]

For me no browser by the way, it just dies:

[Arjanvanmaren]

I have the same issue and commenting out
<RuntimeHostConfigurationOption Include="System.Globalization.AppLocalIcu" Value="68.2.0.9" Condition="$(RuntimeIdentifier.StartsWith('linux')) or $(RuntimeIdentifier.StartsWith('win')) or ('$(RuntimeIdentifier)' == '' and !$([MSBuild]::IsOSPlatform('osx')))" /> worked :) Thanks

[garpunkal]

Same thing happened to me on a recent Umbraco 10 build using Hangfire.

I've updated to the latest version of Visual Studio 2022 17.3.6 and .net 6.0.10 and the site doesn't run. It falls over in the startup stage.

#2105 (comment) fixed the problem for me.

[nheilbuth]

This issue is NOT related to Umbraco. I can reproduce it in an emty ASP.NET 6 app using ICU

[kieron-mcintyre]

Yep, our local development environments which have all had the Windows 22H2 update have the same issue (Umbraco v10.2.0 and .Net 6). However, it makes sense that it is not Umbraco specific.

We host the vast majority of our sites on Azure App Services and a few on Windows VPS. Is this issue going to be something that affects these production environments because we haven't seen any issues yet?

[kieron-mcintyre]

Also, can I suggest that the title of the issue is altered as it's not clear what the issue is. Can I suggest, "AddHangfireServer() crashes .Net 6 application after Windows 22H2 update"? This would have helped me find this issue hours earlier.

[odinserj]

Just tried to reproduce the issue with .NET 6.0.400 and the following configuration on Windows 10 21H2 (OS Build 19044.2130) with InMemory storage – no success.

<Project Sdk="Microsoft.NET.Sdk.Web">

  <PropertyGroup>
    <TargetFramework>net6.0</TargetFramework>
    <Nullable>enable</Nullable>
    <ImplicitUsings>enable</ImplicitUsings>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Hangfire.Core" Version="1.7.31" />
    <PackageReference Include="Hangfire.AspNetCore" Version="1.7.31" />
    <PackageReference Include="Hangfire.InMemory" Version="0.3.4" />
    <PackageReference Include="Microsoft.ICU.ICU4C.Runtime" Version="68.2.0.9" />

    <RuntimeHostConfigurationOption Include="System.Globalization.AppLocalIcu" Value="68.2.0.9" Condition="$(RuntimeIdentifier.StartsWith('linux')) or $(RuntimeIdentifier.StartsWith('win')) or ('$(RuntimeIdentifier)' == '' and !$([MSBuild]::IsOSPlatform('osx')))" />
  </ItemGroup>

</Project>

using Hangfire;

namespace WebApplication2
{
    public class Program
    {
        public static void Main(string[] args)
        {
            var builder = WebApplication.CreateBuilder(args);

            // Add services to the container.
            builder.Services.AddRazorPages();
            builder.Services.AddHangfire(config => config.UseInMemoryStorage());
            builder.Services.AddHangfireServer();

            var app = builder.Build();

            // Configure the HTTP request pipeline.
            if (!app.Environment.IsDevelopment())
            {
                app.UseExceptionHandler("/Error");
            }
            app.UseStaticFiles();
            app.UseHangfireDashboard();

            app.UseRouting();
            app.UseAuthorization();
            app.MapRazorPages();

            app.Run();
        }
    }
}

info: Hangfire.BackgroundJobServer[0]
      Starting Hangfire Server using job storage: 'Hangfire.InMemory.InMemoryStorage'
info: Hangfire.BackgroundJobServer[0]
      Using the following options for Hangfire Server:
          Worker count: 20
          Listening queues: 'default'
          Shutdown timeout: 00:00:15
          Schedule polling interval: 00:00:15
info: Hangfire.Server.BackgroundServerProcess[0]
      Server desktop-am941qm:5752:07911dfc successfully announced in 2.4287 ms
info: Hangfire.Server.BackgroundServerProcess[0]
      Server desktop-am941qm:5752:07911dfc is starting the registered dispatchers: ServerWatchdog, ServerJobCancellationWatcher, Worker, DelayedJobScheduler, RecurringJobScheduler...
info: Hangfire.Server.BackgroundServerProcess[0]
      Server desktop-am941qm:5752:07911dfc all the dispatchers started
info: Microsoft.Hosting.Lifetime[14]
      Now listening on: http://localhost:5153
info: Microsoft.Hosting.Lifetime[0]
      Application started. Press Ctrl+C to shut down.
info: Microsoft.Hosting.Lifetime[0]
      Hosting environment: Development
info: Microsoft.Hosting.Lifetime[0]
      Content root path: C:\Users\odins\source\repos\WebApplication2\WebApplication2\

[odinserj]

But then switched to Hangfire.SqlServer as a job storage and got process exited after ~1 second after start:

builder.Services.AddHangfire(config =>
    config.UseSqlServerStorage("Server=.;Database=Hangfire.Sample;Integrated Security=true"));

info: Hangfire.Server.BackgroundServerProcess[0]
      Server desktop-am941qm:7472:ca1de140 all the dispatchers started

C:\Users\odins\source\repos\WebApplication2\WebApplication2\bin\Debug\net6.0\WebApplication2.exe (process 7472) exited with code -1073740791.

[odinserj]

Tried to switch to Microsoft.Data.SqlClient – no luck:

builder.Services.AddHangfire(config => config.UseSqlServerStorage(
                () => new Microsoft.Data.SqlClient.SqlConnection("Server=.;Database=Hangfire.Sample;Integrated Security=true;TrustServerCertificate=True")));

[ThomasSkyldahl]

@odinserj disable Microsoft Defender as it kills the process because of the attempt to access restricted memory, then you will get the details, but it's inside the framework, nothing to do with Hangfire but maybe a release with a ccompile on the new SDK could help?

It's either the ICU update or the security patch for .NET 6 that was also released in the same timeframe

KB5018410: https://support.microsoft.com/en-us/topic/october-11-2022-kb5018410-os-builds-19042-2130-19043-2130-and-19044-2130-6390f057-28ca-43d3-92ce-f4b79a8378fd
or
KB5019351: https://github.com/dotnet/core/blob/main/release-notes/README.md

[odinserj]

Was able to minimize the reproduction to the following one, just a regular SQL query with Dapper installed and no Hangfire involved.

<Project Sdk="Microsoft.NET.Sdk.Web">

  <PropertyGroup>
    <TargetFramework>net6.0</TargetFramework>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Dapper" Version="2.0.4" />
    <PackageReference Include="Microsoft.Data.SqlClient" Version="5.0.0" />

    <PackageReference Include="Microsoft.ICU.ICU4C.Runtime" Version="68.2.0.9" />
    <RuntimeHostConfigurationOption Include="System.Globalization.AppLocalIcu" Value="68.2.0.9" Condition="$(RuntimeIdentifier.StartsWith('linux')) or $(RuntimeIdentifier.StartsWith('win')) or ('$(RuntimeIdentifier)' == '' and !$([MSBuild]::IsOSPlatform('osx')))" />
  </ItemGroup>

</Project>

using System;
using System.Data;
using Dapper;
using Microsoft.Data.SqlClient;

namespace WebApplication2
{
    public class Program
    {
        public static void Main(string[] args)
        {
            Console.WriteLine("Execuring stored procedure...");

            var parameters = new DynamicParameters();
            parameters.Add("@Resource", "hello-world");
            parameters.Add("@DbPrincipal", "public");
            parameters.Add("@LockMode", "Exclusive");
            parameters.Add("@LockOwner", "Session");
            parameters.Add("@LockTimeout", 5000);
            parameters.Add("@Result", dbType: DbType.Int32, direction: ParameterDirection.ReturnValue);

            using (var connection = new SqlConnection(@"Server=.\;Database=Hangfire.Sample;Trusted_Connection=True;TrustServerCertificate=True;"))
            {
                connection.Execute(
                    @"sp_getapplock",
                    parameters,
                    commandType: CommandType.StoredProcedure);

                var lockResult = parameters.Get<int>("@Result");

                Console.WriteLine("Result: " + lockResult);
            }
        }
    }
}

UPD. Works the same way with System.Data.SqlClient package

[odinserj]

Need to understand whether this is a Dapper-related issue (initial tests shown that with pure ADO.NET it works fine, but I had no time to test output parameters). If so, we can try to get rid of Dapper. DynamicParameters class looks suspicious here.

[odinserj]

Reduced to the following code:

using System;
using System.Data;
using Dapper;
using Microsoft.Data.SqlClient;

namespace WebApplication2
{
    public class Program
    {
        public static void Main(string[] args)
        {
            using (var connection = new SqlConnection(@"Server=.\;Database=Hangfire.Sample;Trusted_Connection=True;TrustServerCertificate=True;"))
            {
                Console.WriteLine("Execuring stored procedure...");

                // WORKS FINE (ADO.NET)
                using (var command = new SqlCommand("sp_getapplock", connection))
                {
                    command.CommandType = CommandType.StoredProcedure;
                    command.Parameters.Add("@Resource", SqlDbType.NVarChar).Value = "hello-world";
                    command.Parameters.Add("@LockMode", SqlDbType.NVarChar).Value = "Exclusive";

                    connection.Open();
                    command.ExecuteNonQuery();
                }

                // WORKS FINE (Dapper)
                var result = connection.ExecuteScalar("select @hello", new { hello = "world" });

                // FAULTS (Dapper with DynamicParameters)
                var parameters = new DynamicParameters();
                parameters.Add("@Resource", "hello-world");
                parameters.Add("@LockMode", "Exclusive");
                connection.Execute(@"sp_getapplock", parameters, commandType: CommandType.StoredProcedure);

                Console.WriteLine("Execution succeeded");
            }
        }
    }
}

So looks like DynamicParameters usage in Hangfire should be replaced with plain ADO.NET to work around the issue. But perhaps the issue itself should be reported to Dapper's team. Any help is appreciated.

[ThomasSkyldahl]

using System.Globalization;

namespace WebApplication2
{
    public class Program
    {
        public static void Main(string[] args)
        {
            CultureInfo.GetCultureInfo("en-US").CompareInfo.Compare("sample", "Sample", CompareOptions.IgnoreSymbols);
        }
    }
}

The simplest repo I could create

[ThomasSkyldahl]

FYI: I have reported this in the .NET Runtime git repository: dotnet/runtime#77045

[nul800sebastiaan]

For followers here:

The fix will be in the next .NET 6.0 service release.

dotnet/runtime#77045 (comment)

[nul800sebastiaan]

.NET SDK 6.0.403 (aka 6.0.11) is out now and should fix the issue. Haven't tested yet but should be good now!