"Remote host closed the connection" unhandled exception polluting Elmah logs

[TechSavvySam]

Note that this issue was previously logged but was closed during the backlog bankruptcy.

#3296

Expected behavior

By default, SignalR should handle remote host unexpectedly closing connections since this is the reality of life in a network environment. Users turn off their systems or close browser windows w/o logging out of web sites.

Actual behavior

Two main exceptions go unhandled and pollute the Elmah logs to the point they are unusable. I would say on my production web site, at least 80% of the errors in Elmah are caused by SignalR.

The request URLs are:

/signalr/start
/signalr/poll/
/signalr/hubs

The exceptions are:

System.Web.HttpException

The client disconnected.

System.Web.HttpException (0x80070040): The client disconnected.
   at System.Web.Hosting.IIS7WorkerRequest.BeginRead(Byte[] buffer, Int32 offset, Int32 count, AsyncCallback callback, Object state)
   at System.Web.HttpBufferlessInputStream.BeginRead(Byte[] buffer, Int32 offset, Int32 count, AsyncCallback callback, Object state)
   at System.IO.Stream.<>c.<BeginEndReadAsync>b__43_0(Stream stream, ReadWriteParameters args, AsyncCallback callback, Object state)
   at System.Threading.Tasks.TaskFactory`1.FromAsyncTrim[TInstance,TArgs](TInstance thisRef, TArgs args, Func`5 beginMethod, Func`3 endMethod)
   at System.IO.Stream.BeginEndReadAsync(Byte[] buffer, Int32 offset, Int32 count)
   at System.IO.Stream.ReadAsync(Byte[] buffer, Int32 offset, Int32 count, CancellationToken cancellationToken)
   at Microsoft.Owin.Host.SystemWeb.CallStreams.DelegatingStream.ReadAsync(Byte[] buffer, Int32 offset, Int32 count, CancellationToken cancellationToken)
   at System.IO.Stream.ReadAsync(Byte[] buffer, Int32 offset, Int32 count)
   at System.IO.StreamReader.<ReadBufferAsync>d__97.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.IO.StreamReader.<ReadToEndAsyncInternal>d__62.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Owin.OwinRequest.<ReadFormAsync>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNet.SignalR.Owin.ServerRequest.<ReadForm>d__20.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Owin.Mapping.MapMiddleware.<Invoke>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Owin.Host.SystemWeb.IntegratedPipeline.IntegratedPipelineContextStage.<RunApp>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Owin.Security.Infrastructure.AuthenticationMiddleware`1.<Invoke>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Owin.Host.SystemWeb.IntegratedPipeline.IntegratedPipelineContextStage.<RunApp>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Owin.Host.SystemWeb.IntegratedPipeline.IntegratedPipelineContext.<DoFinalWork>d__2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Owin.Host.SystemWeb.IntegratedPipeline.StageAsyncResult.End(IAsyncResult ar)
   at System.Web.HttpApplication.AsyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
   at System.Web.HttpApplication.ExecuteStepImpl(IExecutionStep step)
   at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)

System.Web.HttpException

The remote host closed the connection. The error code is 0x80070057.

System.Web.HttpException (0x80070057): The remote host closed the connection. The error code is 0x80070057.
   at System.Web.Hosting.IIS7WorkerRequest.RaiseCommunicationError(Int32 result, Boolean throwOnDisconnect)
   at System.Web.Hosting.IIS7WorkerRequest.ExplicitFlush()
   at System.Web.HttpResponse.Flush(Boolean finalFlush, Boolean async)
   at System.Web.HttpWriter.WriteFromStream(Byte[] data, Int32 offset, Int32 size)
   at Microsoft.Owin.Host.SystemWeb.CallStreams.OutputStream.Write(Byte[] buffer, Int32 offset, Int32 count)
   at Microsoft.AspNet.SignalR.Owin.ServerResponse.Write(ArraySegment`1 data)
   at Microsoft.AspNet.SignalR.Hosting.ResponseExtensions.End(IResponse response, String data)
   at Microsoft.Owin.Mapping.MapMiddleware.<Invoke>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Owin.Host.SystemWeb.IntegratedPipeline.IntegratedPipelineContextStage.<RunApp>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Owin.Security.Infrastructure.AuthenticationMiddleware`1.<Invoke>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Owin.Host.SystemWeb.IntegratedPipeline.IntegratedPipelineContextStage.<RunApp>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Owin.Host.SystemWeb.IntegratedPipeline.IntegratedPipelineContext.<DoFinalWork>d__2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Owin.Host.SystemWeb.IntegratedPipeline.StageAsyncResult.End(IAsyncResult ar)
   at System.Web.HttpApplication.AsyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
   at System.Web.HttpApplication.ExecuteStepImpl(IExecutionStep step)
   at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)

Steps to reproduce

Easiest way to see the issue occurring is to run Elmah and SignalR on a system. That way you can see the errors.

I have yet to figure out a definitive method for reproducing this, but my users have an uncanny ability to make it happen many times per day.

If I figure out a way to consistently reproduce it, I'll update this ticket.

[analogrelay]

Is ELMAH's filtering support not sufficient for suppressing these messages?

In ASP.NET Core, we have indeed suppressed most of these messages because you are correct they aren't terribly useful. Our limitation here is breaking changes. Suppressing these messages would negatively affect users who are expecting them. We could make it a config flag that allows you to suppress the exceptions but a) that isn't cheap in terms of time and resources and b) we don't have plans to ship a new minor version (in which an API change could be made) to do so.

If the ELMAH filtering is able to reduce these logs sufficiently for you, I'd strongly encourage using that. If not, let us know and we can look at our options.

[TechSavvySam]

Thanks--with all my searching, I never used a phrase that pointed me to Elmah's filtering support. I just wrote some filter rules and will deploy them early next week. I'll report back here what I did to help the community and then very likely close the ticket.

[TechSavvySam]

Thanks. Was finally able to deploy code where it could be tested and, YES, this is a reasonable work-around.

For anyone else that stumbles upon this thread, here is what I added to the web.config to exclude my specific errors:

  <elmah>
    <!--
        Other settings omitted for this illustration
    -->
    <errorFilter>
      <test>
        <or>
          <and>
            <equal binding="HttpStatusCode" value="500" type="Int32" />
            <regex binding="Exception.Message" pattern="The remote host closed the connection\. The error code is 0x80070057\..*" />
            <regex binding="Context.Request.ServerVariables['URL']" pattern="/signalr/.*" />
          </and>
          <and>
            <equal binding="HttpStatusCode" value="500" type="Int32" />
            <regex binding="Exception.Message" pattern="The client disconnected\..*" />
            <regex binding="Context.Request.ServerVariables['URL']" pattern="/signalr/.*" />
          </and>
        </or>
      </test>
    </errorFilter>
  </elmah>

[analogrelay]

Glad to hear it!