500 on negotiate when deployed to load balanced servers

[DavidMCarek]

Currently I am using Microsoft.Azure.SignalR.AspNet.1.0.0-preview1-10330 and when I deploy to load balanced servers I get a 500 in the browser when the /signalr/negotiate endpoint is hit. The really confusing part of this is that it works locally just fine. I tried a setup with 2 iis app pools running the same website and it worked great. I also was able to get this working on the load balanced servers but only once I downgraded the nuget package to 10275. So something has changed that broke deployment to remote load balanced servers.

[vicancy]

The changes in 10330 is that it throws early when the app server is not connected to the Azure Service. For 10275, even if the app server is not connected to the Azure Service, the client not throw when /negotiate but throw on /connect. Could you make sure that your app server is connected to the Azure SignalR Service?

[DavidMCarek]

I checked the Azure portal and looks like there's no connection to the app server when the site is spun up on the load balanced server. Is there anything in particular that I need to call to make that connection other than app.MapAzureSignalR or do you think its more likely an infrastructure issue since the code seems to work locally?

[vicancy]

Does the load balancer allow web-socket connections? Some proxy need special configuration to enable websocket feature.

[DavidMCarek]

Since the solution was working on 10275 I assume the websocket connections are working just fine on the server but I'll double check tomorrow.

[DavidMCarek]

Just updated to the latest version and I'm still having the same issue. Its working locally but when it gets deployed I get a 500 in my network tab when the request to signalr/negotiate is executed. If I navigate to the url for that request I get the error message "Azure SignalR Service is not connected yet, please try again later.". I also see no connections made to the signalr service on the azure portal when deployed. 10275 still works just fine. This is what my startup.cs looks like.

public class Startup
{
    public void Configuration(IAppBuilder app)
    {
        var client = new HttpClient();
        client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));

        var container = AutofacConfig.Make(client);

        var enableLogging = bool.Parse(ConfigurationManager.AppSettings["Azure.Signalr.EnableLogging"]);
        if(!int.TryParse(ConfigurationManager.AppSettings["Azure.Signalr.PerServerConnectionCount"], out var perServerConnectionCount))
            perServerConnectionCount = 1;

        DependencyResolver.SetResolver(new global::Autofac.Integration.Mvc.AutofacDependencyResolver(container));

        app.UseAutofacMiddleware(container);
        app.UseAutofacMvc();

        app.UseCookieAuthentication(new CookieAuthenticationOptions
        {
            LoginPath = new PathString("/security/login"),
            AuthenticationType = CookieAuthenticationDefaults.AuthenticationType,
            Provider = new CustomCookieAuthenticationProvider(DependencyResolver.Current.GetService<TokenClient>()),
            SlidingExpiration = true,
            CookieName = IdentityServerConstants.Cookie.NAME
        });

        app.UseCors(SignalrCorsOptions);
        app.MapAzureSignalR(GetType().FullName,
            new HubConfiguration
            {
                Resolver = new global::Autofac.Integration.SignalR.AutofacDependencyResolver(container)
            },
            options =>
            {
                options.ConnectionCount = perServerConnectionCount;
            });

        GlobalHost.TraceManager.Switch.Level = enableLogging ? SourceLevels.Verbose : SourceLevels.Off;
    }

    private static readonly CorsOptions SignalrCorsOptions = new CorsOptions
    {
        PolicyProvider = new CorsPolicyProvider
        {
            PolicyResolver = context => Task.FromResult(new CorsPolicy
            {
                AllowAnyOrigin = true,
                AllowAnyMethod = true,
                AllowAnyHeader = true,
                SupportsCredentials = false
            })
        }
    };
}

[Jun2014]

We see exactly same error when we deploy app server to azure cloud service, version 10275 doesn't have this issue. We filed a Microsoft support tick to track this issue.

[vicancy]

Update: after 1.0.0, the config to enable trace is:

  <system.diagnostics>
    <sources>
      <source name="Microsoft.Azure.SignalR" switchName="SignalRSwitch">
        <listeners>
          <add name="ASRS" />
        </listeners>
      </source>
    </sources>
    <!-- Sets the trace verbosity level -->
    <switches>
      <add name="SignalRSwitch" value="Information" />
    </switches>
    <!-- Specifies the trace writer for output -->
    <sharedListeners>
      <add name="ASRS" type="System.Diagnostics.TextWriterTraceListener" initializeData="asrs.log.txt" />
    </sharedListeners>
    <trace autoflush="true" />
  </system.diagnostics>

---

Could you add the following to web.config to enable trace for the underlying server connection to see what's the error connecting to Azure Service from cloud service? @Jun2014 @DavidMCarek

<system.diagnostics>
    <sources> 
      <source name="Microsoft.AspNetCore.Http.Connections.Client.HttpConnection">
        <listeners>
          <add name="ASRS" />
        </listeners>
      </source>
      <source name="Microsoft.AspNetCore.Http.Connections.Client.Internal.WebSocketsTransport">
        <listeners>
          <add name="ASRS" />
        </listeners>
      </source>
      <source name="Microsoft.Azure.SignalR.ServiceConnectionBase">
        <listeners>
          <add name="ASRS" />
        </listeners>
      </source>
    </sources>
    <!-- Sets the trace verbosity level -->
    <switches>
      <add name="SignalRSwitch" value="Verbose" />
    </switches>
    <!-- Specifies the trace writer for output -->
    <sharedListeners>
      <add name="ASRS" type="System.Diagnostics.TextWriterTraceListener" initializeData="asrs.log.txt" />
    </sharedListeners>
    <trace autoflush="true" />
  </system.diagnostics>

[DavidMCarek]

This is the error in the trace.

Microsoft.Azure.SignalR.ServiceConnectionBase Error: 2 : Failed to connect to the service: The type initializer for 'Microsoft.Azure.SignalR.ConnectionFactory' threw an exception. Could not load file or assembly 'System.Runtime.InteropServices.RuntimeInformation, Version=0.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. The located assembly's manifest definition does not match the assembly reference. (Exception from HRESULT: 0x80131040).

[vicancy]

Thanks @DavidMCarek for your help, this looks like the root cause for no server connection connected and this 500 error. Could you try adding the following assembly rebinding to your web.config to see if the issue is mitigated? In the meanwhile, I will check why the new version has this issue, seems similar to https://stackoverflow.com/questions/48520188/could-not-load-file-or-assembly-system-runtime-interopservices-runtimeinformati

<dependentAssembly>
  <assemblyIdentity name="System.Runtime.InteropServices.RuntimeInformation" publicKeyToken="b03f5f7f11d50a3a" culture="neutral" />
  <bindingRedirect oldVersion="0.0.0.0-4.0.2.0" newVersion="4.0.2.0" />
</dependentAssembly>

[DavidMCarek]

Adding the binding redirect fixes the issue, thanks! Will this issue be fixed in the next release?

[vicancy]

Hi @DavidMCarek, what Azure service are you using to deploy the app server? Is it Cloud service (classic)? I'd like to verify the fix.

[DavidMCarek]

Yep we had it set on classic.

[vicancy]

Hi @DavidMCarek what netframework version is your project?

[DavidMCarek]

I'm pretty sure we are using .net 4.6.2

[vicancy]

The latest 1.0.0 should've fixed the issue.