Recursive resource lookup failure in Kestrel server app for .NET Core 2.0

[sam-piper]

We've recently ported our ASP.NET Core 2.0 application running on .NET Framework 4.7 to use .NET Core 2.0, and we are seeing critical failures in our Kestrel logs now, like this:

Assert Failure
Expression: [Recursive resource lookup bug]
Application is shutting down...
Description: Infinite recursion during resource lookup within System.Private.CoreLib.  This may be a bug in System.Private.CoreLib, or potentially in certain extensibility points such as assembly resolve events or CultureInfo names.  Resource name: ArgumentNull_Generic
Stack Trace:
  at System.SR.InternalGetResourceString(String key)
  at System.SR.GetResourceString(String resourceKey, String defaultString)
  at System.ArgumentNullException..ctor(String paramName)
  at System.Runtime.Loader.AssemblyLoadContext.GetLoadContext(Assembly assembly)
  at System.Reflection.Assembly.LoadFromResolveHandler(Object sender, ResolveEventArgs args)
  at System.AppDomain.OnAssemblyResolveEvent(RuntimeAssembly assembly, String assemblyFullName)
  at System.Reflection.RuntimeAssembly._nLoad(AssemblyName fileName, String codeBase, Evidence assemblySecurity, RuntimeAssembly locationHint, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppressSecurityChecks, IntPtr ptrLoadContextBinder)
  at System.Reflection.RuntimeAssembly.InternalGetSatelliteAssembly(String name, CultureInfo culture, Version version, Boolean throwOnFileNotFound, StackCrawlMark& stackMark)
  at System.Resources.ManifestBasedResourceGroveler.GetSatelliteAssembly(CultureInfo lookForCulture, StackCrawlMark& stackMark)
  at System.Resources.ManifestBasedResourceGroveler.GrovelForResourceSet(CultureInfo culture, Dictionary`2 localResourceSets, Boolean tryParents, Boolean createIfNotExists, StackCrawlMark& stackMark)
  at System.Resources.ResourceManager.InternalGetResourceSet(CultureInfo requestedCulture, Boolean createIfNotExists, Boolean tryParents, StackCrawlMark& stackMark)
  at System.Resources.ResourceManager.InternalGetResourceSet(CultureInfo culture, Boolean createIfNotExists, Boolean tryParents)
  at System.Resources.ResourceManager.GetString(String name, CultureInfo culture)
  at System.SR.InternalGetResourceString(String key)
  at System.SR.GetResourceString(String resourceKey, String defaultString)
  at System.ArgumentNullException..ctor(String paramName)
  at System.Runtime.Loader.AssemblyLoadContext.GetLoadContext(Assembly assembly)
  at System.Reflection.Assembly.LoadFromResolveHandler(Object sender, ResolveEventArgs args)
  at System.AppDomain.OnAssemblyResolveEvent(RuntimeAssembly assembly, String assemblyFullName)
  at System.Reflection.RuntimeAssembly._nLoad(AssemblyName fileName, String codeBase, Evidence assemblySecurity, RuntimeAssembly locationHint, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppressSecurityChecks, IntPtr ptrLoadContextBinder)
   at System.Reflection.RuntimeAssembly.InternalGetSatelliteAssembly(String name, CultureInfo culture, Version version, Boolean throwOnFileNotFound, StackCrawlMark& stackMark)
   at System.Resources.ManifestBasedResourceGroveler.GetSatelliteAssembly(CultureInfo lookForCulture, StackCrawlMark& stackMark)
   at System.Resources.ManifestBasedResourceGroveler.GrovelForResourceSet(CultureInfo culture, Dictionary`2 localResourceSets, Boolean tryParents, Boolean createIfNotExists, StackCrawlMark& stackMark)
   at System.Resources.ResourceManager.InternalGetResourceSet(CultureInfo requestedCulture, Boolean createIfNotExists, Boolean tryParents, StackCrawlMark& stackMark)
   at System.Resources.ResourceManager.InternalGetResourceSet(CultureInfo culture, Boolean createIfNotExists, Boolean tryParents)
   at System.Resources.ResourceManager.GetString(String name, CultureInfo culture)
   at System.SR.InternalGetResourceString(String key)
   at System.SR.GetResourceString(String resourceKey, String defaultString)
   at System.Threading.Tasks.TaskCanceledException..ctor(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

We are using en-AU as our default culture.

This stack trace appears to identify a known issue that has been resolved but not released (dotnet/coreclr/issues/12668.

However, none of the following proposed workarounds in that issue and related issue (#23608) worked for us:

1: Adding new ArgumentException(); into the main method of Program.cs before creating a WebHost.
2. Changing the OS culture to United States, or setting CurrentUICulture to en-US.

Will there be a service release to fix this issue? Any idea when that release will be available? Is there a pre-release build available that I could use to see if the issue is resolved in that build (even though not released yet?)

Thanks,

Sam

[danmoseley]

@tarekgh any idea why new ArgumentException() might not work? Another customer was successful with that workaround.

[tarekgh]

I think what makes this workaround work is setting the CurrentUICulture to en-US. The problem happens when having a different culture and trying to load a satellite assembly for this culture.

@kouvel already fixed this issue in the release/2.0.0 branch. is it possible to use the packages built from this branch?

[mhmd-azeez]

@tarekgh I have the same problem. I am using Asp Net Core 2.0, How can I use the latest package?

[tarekgh]

@weshaggard @kouvel does the package https://dotnet.myget.org/feed/dotnet-core/package/nuget/Microsoft.NETCore.Runtime.CoreCLR/2.0.2-servicing-25712-01 is the latest servicing package includes @kouvel fix?

[kouvel]

Yes based on the commit in version.txt for the package, it should contain the fix for this issue

[kouvel]

I believe the way to target a specific build is described here: https://github.com/dotnet/corefx/blob/master/Documentation/project-docs/dogfooding.md#advanced-scenario---using-a-nightly-build-of-microsoftnetcoreapp

There's a link there to installers/binaries. The latest installer under Release/2.0.X should also contain the fix.

[JamesReate]

Anybody tried changing this with an Azure app service deployment? ie. to get the latest bits of the runtime that contain this fix in Azure?

[tarekgh]

@karelz @danmosemsft do you know who can help with @JamesReate question?

[danmoseley]

@JamesReate I would not expect Azure to offer prerelease binaries. If you're deploying yourself, it should be possible. I'm not famliar with how this is done.

@tarekgh /@kouvel do I understand right that if culture is not en-US in the first place, there is no known workaround?

[tarekgh]

I am not aware of any workaround other than changing the CurrentUICulture to en-US. or try to provide Satellite assembly for the requested language.

[tarekgh]

@karelz this is not globalization issue. This is just happened to interfere with the globalization. Also, I am not sure if we need to keep this issue opened as we already fixed it in the 2.0 servicing.

[JamesReate]

Thanks guys. So what I'm gonna have my team try is follow option 2 (self contained) from these docs: https://github.com/dotnet/corefx/blob/master/Documentation/project-docs/dogfooding.md#advanced-scenario---using-a-nightly-build-of-microsoftnetcoreapp

I'll report back if it works. If it doesn't I guess we'll try mounting it in a Docker container where we have more control.

[JamesReate]

So I can report back that with the self contained option (2) we were able to get this to work.

One tricky thing was getting the package to restore in our build process in VSTS, but we managed that by configuring the nuget.config with the myget feed.

[sam-piper]

I'm trying to use a 2.0.2 nightly runtime build in my local dev environment (VS 15.3) to see if this problem is resolved, but can't get my code building.

These are the steps I've taken:

1. Installed the latest 2.0.2 runtime (https://dotnetcli.blob.core.windows.net/dotnet/Runtime/release/2.0.0/dotnet-runtime-latest-win-x64.exe) which is version 2.0.2-servicing-25728-02.

2. Updated all csproj files containing <TargetFramework>netcoreapp2.0</TargetFramework>, to also contain <RuntimeFrameworkVersion>2.0.2-servicing-25728-02</RuntimeFrameworkVersion>.

NuGet restore fails with the message "Unable to find package Microsoft.NETCore.App with version (>= 2.0.2-servicing-25728-02)", and the build fails completely as well, due to all system types not being available anymore.

I'm guessing that I need to install a matching SDK as well but I can't find a download link for the SDK for the 2.0.X branch here (https://github.com/dotnet/core-setup#daily-builds), only the runtime. The current SDK available from nightly builds is still Preview 2 of .NET Core 2.0.

Alternatively, I can just wait for an official release of 2.0.X - could someone please give a rough indication of when this will happen? EG, within the next few weeks?

[kouvel]

You may need to add the myget.org feed to the project's nuget.config in addition to the nuget.org feed (daily build packages are published to myget.org), see the dotnet-core line here and ignore the local coreclr line:
https://github.com/dotnet/corefx/blob/master/Documentation/project-docs/dogfooding.md#2---add-your-bin-directory-to-the-nuget-feed-list

[kouvel]

could someone please give a rough indication of when this will happen? EG, within the next few weeks?

@leecow, do you happen to know roughly when 2.0.1 would be out? Also it looks like we're using the same branch (release/2.0.0) for 2.0.1 and 2.0.2 so where can I find the latest commit hash for 2.0.1 so that I can verify whether the fix would be in 2.0.1?

[leecow]

Due to some build kerfuffles while working toward the September update, 2.0 will skip to 2.0.2 (ie, there won't be a public 2.0.1). Release expected next month.

[danmoseley]

Fix released https://github.com/dotnet/corefx/releases/tag/v2.0.3