WEBSITE_RUN_FROM_PACKAGE startup issues after platform maintenance

[jonarmstrong]

Hi @PaitoAnderson and @xt0rted, could you please email me with your site names? I'm trying to repro your issue. Thanks!

This is a follow-up to my post on 11/23/2019. I've since dug a little deeper and realize I have the same problem was impacting @PaitoAnderson & @xt0rted. My apps are also using the WEBSITE_RUN_FROM_PACKAGE setting, which seems to be the root cause of this issue.

Was there every a resolution to this? I have three apps which currently have bad instances because of this exact issue. The only way to resolve the issue is to restart the app and hope it comes back up. It has been plaguing my app reliability for months now. Below are stack traces from 2 of the 3 failing web sites

Instance - RD501AC561D51E

/subscriptions/3a2f_______________3e76/resourceGroups/fsp___________rg/providers/Microsoft.Web/sites/merch__________________-mahzuanneuqrm

@ 2020-02-16T09:48:18

Application: *******************.exe
CoreCLR Version: 4.6.28207.3
Description: The process was terminated due to an unhandled exception.
Exception Info: System.IO.FileLoadException: Could not load file or assembly 'Microsoft.AspNetCore.Authentication.Abstractions, Version=2.2.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'. An assertion failure has occurred. (Exception from HRESULT: 0x8007029C)
   at Microsoft.Extensions.DependencyInjection.AuthenticationCoreServiceCollectionExtensions.AddAuthenticationCore(IServiceCollection services)
   at Microsoft.AspNetCore.Hosting.WebHostBuilderIISExtensions.<>c__DisplayClass5_1.<UseIISIntegration>b__0(IServiceCollection services)
   at Microsoft.AspNetCore.Hosting.WebHostBuilder.BuildCommonServices(AggregateException& hostingStartupErrors)
   at Microsoft.AspNetCore.Hosting.WebHostBuilder.Build()
   at ******************.Program.Main(String[] args) in d:\a\1\s\****************\Program.cs:line 17

Instance - RD501AC56B00BE

/subscriptions/3a2f______________3e76/resourceGroups/fsp_________________rg/providers/Microsoft.Web/sites/hosted_____________-bqxfcgnuocrvw

@ 2020-02-16T07:20:21

Application: ***********************.exe
   CoreCLR Version: 4.6.28207.3
   Description: The process was terminated due to an unhandled exception.
   Exception Info: System.IO.FileLoadException: Could not load file or assembly 'System.IO.Pipelines, Version=4.0.0.1, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51'. An assertion failure has occurred. (Exception from HRESULT: 0x8007029C)
      at Microsoft.AspNetCore.Server.Kestrel.Core.KestrelServer.CreateServiceContext(IOptions`1 options, ILoggerFactory loggerFactory)
      at Microsoft.AspNetCore.Server.Kestrel.Core.KestrelServer..ctor(IOptions`1 options, ITransportFactory transportFactory, ILoggerFactory loggerFactory)
   --- End of stack trace from previous location where exception was thrown ---
      at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitConstructor(ConstructorCallSite constructorCallSite, ServiceProviderEngineScope scope)
      at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitScoped(ScopedCallSite scopedCallSite, ServiceProviderEngineScope scope)
      at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService(IServiceProvider provider, Type serviceType)
      at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService[T](IServiceProvider provider)
      at Microsoft.AspNetCore.Hosting.Internal.WebHost.EnsureServer()
      at Microsoft.AspNetCore.Hosting.Internal.WebHost.BuildApplication()
      at Microsoft.AspNetCore.Hosting.Internal.WebHost.StartAsync(CancellationToken cancellationToken)
      at Microsoft.AspNetCore.Hosting.WebHostExtensions.RunAsync(IWebHost host, CancellationToken token, String shutdownMessage)
      at Microsoft.AspNetCore.Hosting.WebHostExtensions.RunAsync(IWebHost host, CancellationToken token)
      at Microsoft.AspNetCore.Hosting.WebHostExtensions.Run(IWebHost host)
      at ***********************.Program.Main(String[] args) in d:\a\2\s\***********************\***********************\Program.cs:line 24

Originally posted by @jonarmstrong in Azure/app-service-announcements-discussions#32 (comment)

[flagbug]

We're seeing the exact same issue, see #3088

Even contacted Azure Support about this, but they were useless, asking me to "attach a profiler" and just ignored my explanation of the root cause 🤷‍♂

[katiep23]

I'm seeing this issue now. Anyone have luck figuring this out?

[jonarmstrong]

@flagbug @katiep23 Regretfully, I've yet to receive any feedback or support about this particular issue.

[suwatch]

@jonarmstrong This is on AppService Environment and only happened to some instances, correct? Wonder if you have a more recent incident? Provide the UTC time, instance and error as you did above - we will take try to correlate with our log.

[jonarmstrong]

@jonarmstrong This is on AppService Environment and only happened to some instances, correct? Wonder if you have a more recent incident? Provide the UTC time, instance and error as you did above - we will take try to correlate with our log.

@suwatch, Yes, this is on an ILB ASE and I don't have a more recent occurrence of this issue as it has not occurred again since then. Perhaps @katiep23 who just had this occur has a more recent log that she can pull for you to cross reference?

[jonarmstrong]

@suwatch Hey, this issue happened again over the weekend and killed a node on two of my app service clusters. Both are I1 skus in an ILB ASE

Errors started ~ @ 5/17/2020 09:40:00, a restart for a 'Platform File Server Upgrade' also occurred at the same time. The impacted instance name was RD501AC561EA94

Errors started ~ @ 5/18/2020 03:00:00 UTC, a restarted for a 'Platform File Server Upgrade' also occurred at the same time. The impacted instance name was RD501AC561D51E

[suwatch]

I suggest to open http://aka.ms/CRI-WebApps. Kudu is more about deployment and for runtime issue, it will be best served from App Service product team.

[jonarmstrong]

@suwatch Do you have a publicly accessible location I can submit the issue to? That goes to an internal microsoft site which I can't sign into.