System.InvalidOperationException: TODO, RESX: net_Websockets_AlreadyOneOutstandingOperation; #207

shravan2x · 2017-10-26T02:01:13Z

This exception occurred during some testing, I'm not 100% sure how to reproduce it. If the TODO below actually means something was not implemented, that may be the cause.

System.InvalidOperationException: TODO, RESX: net_Websockets_AlreadyOneOutstandingOperation; (SendAsync)
   at System.Net.WebSockets.ManagedWebSocket.ThrowIfOperationInProgress(Task operationTask, String methodName)
   at System.Net.WebSockets.ManagedWebSocket.SendAsync(ArraySegment`1 buffer, WebSocketMessageType messageType, Boolean endOfMessage, CancellationToken cancellationToken)
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at EzDrive.ControlChannel.ControlSocketHandler.<BroadcastProgressAsync>d__6.MoveNext() in C:\Users\xx.cs:line 39
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at EZDrive.Controllers.DriveController.<Upload>d__11.MoveNext() in C:\Users\xx.cs:line 117
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.<InvokeActionMethodAsync>d__12.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.<InvokeNextActionFilterAsync>d__10.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Rethrow(ActionExecutedContext context)
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.<InvokeInnerFilterAsync>d__14.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.<InvokeNextResourceFilter>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Rethrow(ResourceExecutedContext context)
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.<InvokeFilterPipelineAsync>d__17.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.<InvokeAsync>d__15.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNetCore.Builder.RouterMiddleware.<Invoke>d__4.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware.<Invoke>d__7.MoveNext()

The text was updated successfully, but these errors were encountered:

shravan2x · 2017-10-26T02:04:33Z

That issue seems to corrupt all of the application for some reason, leading to more of

System.Net.WebSockets.WebSocketException (0x80004005): The remote party closed the WebSocket connection without completing the close handshake. ---> System.ObjectDisposedException: Cannot access a disposed object.
Object name: 'FrameResponseStream'.
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.FrameResponseStream.ValidateState(CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.FrameResponseStream.WriteAsync(Byte[] buffer, Int32 offset, Int32 count, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.FrameDuplexStream.WriteAsync(Byte[] buffer, Int32 offset, Int32 count, CancellationToken cancellationToken)
   at System.Net.WebSockets.ManagedWebSocket.SendFrameLockAcquiredNonCancelableAsync(MessageOpcode opcode, Boolean endOfMessage, ArraySegment`1 payloadBuffer)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at EzDrive.ControlChannel.DriveGroup.<RegisterRunAsync>d__7.MoveNext() in C:\Users\xx.cs:line 37
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at EzDrive.ControlChannel.ControlSocketHandler.<ControlHandlerAsync>d__9.MoveNext() in C:\Users\xx.cs:line 79
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNetCore.Builder.Extensions.MapWhenMiddleware.<Invoke>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNetCore.Builder.RouterMiddleware.<Invoke>d__4.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware.<Invoke>d__7.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware.<Invoke>d__7.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNetCore.Hosting.Internal.RequestServicesContainerMiddleware.<Invoke>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.Frame`1.<ProcessRequestsAsync>d__2.MoveNext()

jkotalik · 2017-10-26T06:32:27Z

The TODO is a duplicate of another issue #198. It isn't an actual issue, its that we forgot to add logging for some websocket scenarios. The exceptions you are getting are more likely because there is a websocket protocol violation between the client and the server. I'd check out the docs for more information https://docs.microsoft.com/en-us/aspnet/core/fundamentals/websockets. Sorry that the exceptions are confusing ☹️
cc/ @anurse regarding the TODO, I think we need to start copying for 2.1. This has to be fixed.

shravan2x · 2017-10-26T06:45:33Z

I think I figured out the issue with the sockets themselves. I was calling SendAsync when a previous SendAsync might have been blocked since the connection closed. I'll confirm by tomorrow.

analogrelay · 2017-10-27T02:27:29Z

@shravan2x Yeah, that makes sense. The exception message here is supposed to be this one: https://github.com/dotnet/corefx/blob/dfb4bf4b7454fdbacbcada8ff8fe344a32fc527e/src/System.Net.WebSockets/src/Resources/Strings.resx#L115 : There is already one outstanding 'SendAsync' call for this WebSocket instance. ReceiveAsync and SendAsync can be called simultaneously, but at most one outstanding operation for each of them is allowed at the same time.

@jkotalik Yeah, plus there's another PR to corefx to copy across: dotnet/corefx#24846 . Since it looks like we'll be stuck with source copying for a little while longer, we may as well update it after that PR is merged.

DjTrilogic · 2017-12-18T11:57:03Z

Hello,
I'm facing the same problem, and my application totally quits when it occurs.
Is there a workaround to prevent this from happening ?
Thank's in advance

davidfowl · 2017-12-18T12:29:20Z

Don’t overlap more than one send or more than one receive in your code.

DjTrilogic · 2017-12-18T12:34:13Z

@davidfowl Alright Thank's a lot.

Any idea how to handle exceptions of this kind, to prevent the application crash ?

davidfowl · 2017-12-18T12:45:48Z

Catch exceptions if you do things off the request thread.

ghost · 2017-12-21T18:24:17Z

Calling SendAsync in parallel is not the only issue here, it just goes nuts after the disposal

OK so i have added breakpoints in EVERY access of WebSocket in my service code.

I started to get these annoying errors in logs for about million times, so i disconnected all the sessions and put those breakpoints while service was still running (receiving no new connections).

Now the situation is connections died 45 minutes ago and these FrameResponseStream errors are logging periodically in batches and NOT hitting ANY breakpoint, Earlier in my code I called closedAsync, Dispoced, marked the ws object to null and not touching that object anywhere and not hitting ANY breakpoint...

@anurse
Glad you mentioned about the copy of corefx code.. Can you please please pleaseee start depending on corefx Websocket package via nuget instead of copying code that copy is now 8 months old in this repo.... Can you PLEASE do that and release a new version of WS? THANK YOU

analogrelay · 2017-12-21T19:56:04Z

Can you please please pleaseee start depending on corefx Websocket package via nuget instead of copying code

Agreed :). Filed #214 to make sure at the very least we update our copy of the code (it's pretty easy to update).

niemyjski · 2017-12-22T13:26:26Z

We hooked up our message bus up to our implementation and push messages right off the hub to the client over a web socket. Is there any plans to add a buffer on top of web sockets to prevent this kind of stuff? I'd really hate to have to add locking around this.

niemyjski · 2017-12-22T13:30:11Z

Would be really nice if a web socket had a state where I could check if it's currently sending data.

Tratcher · 2017-12-22T14:16:51Z

@niemyjski such a state would be too transient, it would be a constant race condition. Any cross-thread access really needs a locking mechanism. SemaphoreSlim is my preferred async lock. We have no plans to add a buffering layer.

stephentoub · 2017-12-22T14:43:32Z

SemaphoreSlim is my preferred async lock.

That's a good option if callers may run in parallel. If the issue is that the caller is serialized but may call while a previous operation is still in fight, you can also just store the previous Send/ReceiveAsync-returned task and await it before sending/receiving the next message. It'll be a nop if there are no attempts at concurrent usage and will force serialization if there is.

davidfowl · 2017-12-24T03:22:20Z

To add to what @stephentoub said, that's basically what the previous version of SignalR did via this TaskQueue abstraction.

https://github.com/SignalR/SignalR/blob/35a44764686f796baf7c4256e5d52ab5f9d1162b/src/Microsoft.AspNet.SignalR.Core/Infrastructure/TaskQueue.cs

niemyjski · 2017-12-27T14:22:41Z

@davidfowl Thanks for that. I had to write a task queue last week for back plane related work (https://github.com/FoundatioFx/Foundatio/blob/feature/taskqueue/src/Foundatio/Queues/TaskQueue.cs).

niemyjski · 2017-12-27T14:28:04Z

@davidfowl I'm curious how you are handling this in the new version of signalr (Will have to dive into the code).

davidfowl · 2017-12-27T20:04:52Z

@niemyjski currently we're usings Channels as our synchronization primitive (later we'll change to pipelines in some places) but it's basically the same idea. There's a single consumer pulling data from the channel and writing to the websocket (and vice versa for websocket reads). There's never any overlapping sends or receives.

aspnet-hello · 2018-01-01T07:12:01Z

This issue was moved to dotnet/aspnetcore#2322

samyhrer mentioned this issue Nov 9, 2017

ReceiveAsync and SendAsync can be called simultaneously, but at most one outstanding operation for each of them is allowed at the same time graphql-dotnet/server#26

Closed

ghost mentioned this issue Dec 21, 2017

The remote party closed the WebSocket connection without completing the close handshake. : Cannot access a disposed object. Object name: 'FrameResponseStream' #212

Closed

niemyjski mentioned this issue Dec 22, 2017

WebSocket connection is closed without proper handshake aspnet/AspNetCoreModule#77

Closed

aspnet locked and limited conversation to collaborators Jan 1, 2018

aspnet-hello closed this as completed Jan 1, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

System.InvalidOperationException: TODO, RESX: net_Websockets_AlreadyOneOutstandingOperation; #207

System.InvalidOperationException: TODO, RESX: net_Websockets_AlreadyOneOutstandingOperation; #207

shravan2x commented Oct 26, 2017 •

edited

Loading

shravan2x commented Oct 26, 2017

jkotalik commented Oct 26, 2017

shravan2x commented Oct 26, 2017

analogrelay commented Oct 27, 2017

DjTrilogic commented Dec 18, 2017

davidfowl commented Dec 18, 2017

DjTrilogic commented Dec 18, 2017

davidfowl commented Dec 18, 2017

ghost commented Dec 21, 2017 •

edited by ghost

Loading

analogrelay commented Dec 21, 2017

niemyjski commented Dec 22, 2017

niemyjski commented Dec 22, 2017

Tratcher commented Dec 22, 2017

stephentoub commented Dec 22, 2017

davidfowl commented Dec 24, 2017 •

edited

Loading

niemyjski commented Dec 27, 2017

niemyjski commented Dec 27, 2017

davidfowl commented Dec 27, 2017

aspnet-hello commented Jan 1, 2018

System.InvalidOperationException: TODO, RESX: net_Websockets_AlreadyOneOutstandingOperation; #207

System.InvalidOperationException: TODO, RESX: net_Websockets_AlreadyOneOutstandingOperation; #207

Comments

shravan2x commented Oct 26, 2017 • edited Loading

shravan2x commented Oct 26, 2017

jkotalik commented Oct 26, 2017

shravan2x commented Oct 26, 2017

analogrelay commented Oct 27, 2017

DjTrilogic commented Dec 18, 2017

davidfowl commented Dec 18, 2017

DjTrilogic commented Dec 18, 2017

davidfowl commented Dec 18, 2017

ghost commented Dec 21, 2017 • edited by ghost Loading

analogrelay commented Dec 21, 2017

niemyjski commented Dec 22, 2017

niemyjski commented Dec 22, 2017

Tratcher commented Dec 22, 2017

stephentoub commented Dec 22, 2017

davidfowl commented Dec 24, 2017 • edited Loading

niemyjski commented Dec 27, 2017

niemyjski commented Dec 27, 2017

davidfowl commented Dec 27, 2017

aspnet-hello commented Jan 1, 2018

shravan2x commented Oct 26, 2017 •

edited

Loading

ghost commented Dec 21, 2017 •

edited by ghost

Loading

davidfowl commented Dec 24, 2017 •

edited

Loading