Deploy to Azure - site dies after 24-48 hours

[bobwiller]

Sorry if this isn't the right place to post this, but I am pulling my hair out: I have a simple Angular app built using ASP.NET 5 rc1 (not using MVC) and then using Visual Studio 2015 to deploy to an Azure Web App.

The site works fine for 24-48 hours and at some point it fails to load/render properly. I sense it is missing something to start or restart itself properly, but the deploy to azure docs are not complete.

My web.config looks like this:

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <system.webServer>
    <handlers>
      <add name="httpPlatformHandler" path="*" verb="*" modules="httpPlatformHandler" resourceType="Unspecified"/>
    </handlers>
    <httpPlatform processPath="%DNX_PATH%" arguments="%DNX_ARGS%" stdoutLogEnabled="false" startupTimeLimit="3600"/>

    <rewrite>
      <rules>
        <!--Redirect selected traffic to index -->
        <rule name="Index Rule" stopProcessing="true">
          <match url=".*" />
          <conditions logicalGrouping="MatchAll">
            <add input="{REQUEST_FILENAME}" matchType="IsFile" negate="true" />
            <!--<add input="{REQUEST_FILENAME}" matchType="IsDirectory" negate="true" />-->
            <add input="{REQUEST_URI}" pattern="^/(blog)" negate="true" />
          </conditions>
          <action type="Rewrite" url="/index.html" />
        </rule>
      </rules>
    </rewrite>
    <staticContent>
      <remove fileExtension=".woff" />
      <mimeMap fileExtension=".woff" mimeType="application/font-woff" />
      <remove fileExtension=".woff2" />
      <mimeMap fileExtension=".woff2" mimeType="application/font-woff2" />
      <remove fileExtension=".ttf" />
      <mimeMap fileExtension=".ttf" mimeType="application/font-ttf" />
    </staticContent>
  </system.webServer>
</configuration>

and my project.json:

{
  "version": "1.0.0-*",
  "compilationOptions": {
    "emitEntryPoint": true
  },

  "dependencies": {
    "Microsoft.AspNet.Mvc": "6.0.0-rc1-final",
    "Microsoft.AspNet.IISPlatformHandler": "1.0.0-rc1-final",
    "Microsoft.AspNet.Server.Kestrel": "1.0.0-rc1-final",
    "Microsoft.AspNet.StaticFiles": "1.0.0-rc1-final"
  },

  "commands": {
    "web": "Microsoft.AspNet.Server.Kestrel"
  },

  "frameworks": {
    "dnx451": { },
    "dnxcore50": { }
  },

  "exclude": [
    "wwwroot",
    "node_modules"
  ],
  "publishExclude": [
    "**.user",
    "**.vspscc",
    "wwwroot/lib/**",
    "wwwroot/scripts/**"
  ]
}

And here is the error:

Event code: 3005
Event message: An unhandled exception has occurred.
Event time: 12/31/2015 9:38:31 PM
Event time (UTC): 12/31/2015 9:38:31 PM
Event ID: f3ac130824e24d9f8b5e858c73ee3ac1
Event sequence: 1
Event occurrence: 1
Event detail code: 0

Application information:
Application domain: /LM/W3SVC/580019606/ROOT-2-130960715112734726
Trust level: 
Application Virtual Path: /
Application Path: D:\home\site\wwwroot\
Machine name: RD00155D84A050

Process information:
Process ID: 9288
Process name: w3wp.exe
Account name: IIS APPPOOL\cardtarget

Exception information:
Exception type: InvalidOperationException
Exception message: Couldn't determine an appropriate version of runtime to run. See http://go.microsoft.com/fwlink/?LinkId=517742 for more information. at AspNet.Loader.RuntimeLocator.LocateRuntime(MapPathHelper mapPathHelper, Boolean& isCoreClr, String& relativeAppBasePath) at AspNet.Loader.Bootstrapper.LoadApplication(String appId, String appConfigPath, IProcessHostSupportFunctions supportFunctions, LoadApplicationData* pLoadAppData, Int32 loadAppDataSize) at System.Runtime.InteropServices.Marshal.ThrowExceptionForHRInternal(Int32 errorCode, IntPtr errorInfo) at System.Web.Hosting.ProcessHost.System.Web.Hosting.IProcessHostLite.ReportCustomLoaderError(String appId, Int32 hr, AppDomain newlyCreatedAppDomain) --- End of stack trace from previous location where exception was thrown --- at System.Web.Hosting.ApplicationManager.CreateAppDomainWithHostingEnvironment(String appId, IApplicationHost appHost, HostingEnvironmentParameters hostingParameters) at System.Web.HttpRuntime.HostingInit(HostingEnvironmentFlags hostingFlags, PolicyLevel policyLevel, Exception appDomainCreationException)


Request information:
Request URL: http://www.cardtarget.com/?cardID=1130407470
Request path: /
User host address: 68.180.230.226
User: 
Is authenticated: False
Authentication Type: 
Thread account name: IIS APPPOOL\cardtarget

Thread information:
Thread ID: 6
Thread account name: IIS APPPOOL\cardtarget
Is impersonating: False
Stack trace: at System.Web.HttpRuntime.HostingInit(HostingEnvironmentFlags hostingFlags, PolicyLevel policyLevel, Exception appDomainCreationException)

------------------------------
Of course the link in the error doesn't point to anything, and all the other searches for locating the appropriate run time haven't helped.

What else is missing?? I am pulling my hair out with this and any help would be appreciated.
Bob

[guardrex]

@bobwiller Yeah ... and what a night to have this happen, right??? I'm sure everyone is out partying their rear-ends off.

There is no reported Azure issue with that exception AFAIK. It's probably a config problem with your app. The critical bit is ...

Couldn't determine an appropriate version of runtime to run ... at 
AspNet.Loader.RuntimeLocator.LocateRuntime(...)

The runtime is specified in global.json, which is attached in VS to the project's properties on the Application tab of the project properties. It also appears in the VS Publish dialog.

How are you publishing your project prior to deployment to Azure? Do you have a global.json file? If so, what's in it? If so, does it match what you have in the project's properties Application tab? If so, does it also match what you have in the VS Publish dialog (if you are using VS Publish)? Finally, do a dnvm list and see which runtime is showing as active (it will have an asterisk next to it), and what is listed as the 'default' runtime?

[davidfowl]

That error looks like Helios is left over (AspNet.Loader.dll) in the bin folder of your deployed application

[guardrex]

Oh, that's a new one!

[bobwiller]

Just had the same issue about an hour ago -

@guardrex - Here is my global.json although it does not appear to get deployed to Azure.

{
"projects": [ "src", "test" ],
"sdk": {
"version": "1.0.0-rc1-update1"
}
}

When I publish, I am simply right-clicking the project, selecting "Publish..." and doing a deployment to my Azure Web app. On the Settings tab, my Configuration is set to "Release - Any CPU" and Target DNX Version is set to "Default (dnx-clr-win-x86.1.0.0.0-rc1-update1)"

if I use the Kudu cmd window, I get 'dnvm' is not recognized as an internal or external command.

@davidfowl: I was hoping it was going to be that easy, but AspNet.Loader.dll does not appear to be present (although this is a project I upgraded from beta 5, and had the same issues, although not as frequently)

Here is my StartUp class if this helps:

public class Startup
{
    // This method gets called by the runtime. Use this method to add services to the container.
    // For more information on how to configure your application, visit http://go.microsoft.com/fwlink/?LinkID=398940
    public void ConfigureServices(IServiceCollection services)
    {
    }

    // This method gets called by the runtime. Use this method to configure the HTTP request pipeline.
    public void Configure(IApplicationBuilder app)
    {
        app.UseIISPlatformHandler();
        app.UseDefaultFiles();
        app.UseStaticFiles();
    }

    // Entry point for the application.
    public static void Main(string[] args) => WebApplication.Run<Startup>(args);
}

}

Thanks for the replies guys, I appreciate the help so far.
Bob

[guardrex]

@bobwiller Is your delete all files checked in the publish profile? If it isn't, that could lead to the AspNet.Loader.dll persisting on the server even though it's not in your project.

[davidfowl]

Yep! AspNet.Loader.dll is totally on your server messing things up 😄 .

[glennc]

@bobwiller I am going to assume that this is resolved once Helios is removed from the server. Closing. Re-open it or create a new issue if you still need help.

[bobwiller]

Sorry @glennc I was traveling last week and wasn't able to respond. This is still causing me headaches:

I went through and deleted all the files under / via FTP, and then published per @guardrex (checking remove additional files and compile files into NuGet packages), and I still go 48-72 hours until the site stops working. Only now, I also no longer get any errors in the event viewer.

The other thing that is different now is that when I try to publish I get the following error:

"An error was encountered when processing operation 'Delete Directory' on 'D:\home\site\wwwroot\App_Data\Jobs'."

So I need to ftp and delete that Jobs folder before republishing.

I just turned on some additional logging in the Azure portal hoping to catch what is going on, but this continues to be a problem (at least for me). Sorry for the delay in responding.

Let me know if there is anything else I should look for, or can do from a troubleshooting perspective.
Thanks,
Bob

[guardrex]

@bobwiller There is a report of the Azure logging causing issues (but only one report of that, so it might be nothing): #364 (comment)

Is this a free tier app? There is a report of running out of space on the free tier: aspnet/dnx#3192