New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Revert to legacy queued handler. #1280

Merged
merged 1 commit into from Apr 21, 2017

Conversation

5 participants
@pgermishuys
Member

pgermishuys commented Apr 20, 2017

No description provided.

@hayley-jean hayley-jean merged commit eeeecac into release-v4.0.1 Apr 21, 2017

2 checks passed

continuous-integration/appveyor/pr AppVeyor build succeeded
Details
wercker/build-mono4 Wercker pipeline passed
Details

@hayley-jean hayley-jean deleted the revert-mpsc branch Apr 21, 2017

pgermishuys added a commit that referenced this pull request Apr 21, 2017

@Narvalex

This comment has been minimized.

Show comment
Hide comment
@Narvalex

Narvalex Apr 21, 2017

Contributor

Can someone tell me please what is the matter with the QueueHandlerUsingMpsc?

Contributor

Narvalex commented Apr 21, 2017

Can someone tell me please what is the matter with the QueueHandlerUsingMpsc?

@Narvalex

This comment has been minimized.

Show comment
Hide comment
@Narvalex

Narvalex Apr 21, 2017

Contributor

I thought that the @Scooletz queue was better than the legacy one... What happen?

Contributor

Narvalex commented Apr 21, 2017

I thought that the @Scooletz queue was better than the legacy one... What happen?

@gregoryyoung

This comment has been minimized.

Show comment
Hide comment
@gregoryyoung

gregoryyoung Apr 21, 2017

Member

It seems to run into an occasional issue but only on AWS linux, thread getting stuck in enqueue. We root caused an issue on AWS linux to that. Therefore moving back to the old one for now. The overall performance difference is extremely minimal.

Member

gregoryyoung commented Apr 21, 2017

It seems to run into an occasional issue but only on AWS linux, thread getting stuck in enqueue. We root caused an issue on AWS linux to that. Therefore moving back to the old one for now. The overall performance difference is extremely minimal.

@Narvalex

This comment has been minimized.

Show comment
Hide comment
@Narvalex

Narvalex Apr 21, 2017

Contributor

Thanks Greg. But why not just go like this
#if MONO
return DefaultHandler
#else
return MinimalPerformanceBoostHandler
#endif

Contributor

Narvalex commented Apr 21, 2017

Thanks Greg. But why not just go like this
#if MONO
return DefaultHandler
#else
return MinimalPerformanceBoostHandler
#endif

@Scooletz

This comment has been minimized.

Show comment
Hide comment
@Scooletz

Scooletz Apr 21, 2017

Contributor

@gregoryyoung So it's enqueuing thread that is freezing? Is it MONO only?

Contributor

Scooletz commented Apr 21, 2017

@gregoryyoung So it's enqueuing thread that is freezing? Is it MONO only?

@gregoryyoung

This comment has been minimized.

Show comment
Hide comment
@gregoryyoung

gregoryyoung Apr 21, 2017

Member

likely will. There is not a dramatic performance increase. My bigger worry is there is a bug in other environments possibly including windows and we have not run into it yet. Risk management would suggest, revert it out for all platforms, further testing over a week or two then review.

@Scooletz not mono only. AWS linux only. We have not seen the issue in ubuntu etc. It may exist in OSX as well (have not tested enough)

Member

gregoryyoung commented Apr 21, 2017

likely will. There is not a dramatic performance increase. My bigger worry is there is a bug in other environments possibly including windows and we have not run into it yet. Risk management would suggest, revert it out for all platforms, further testing over a week or two then review.

@Scooletz not mono only. AWS linux only. We have not seen the issue in ubuntu etc. It may exist in OSX as well (have not tested enough)

@Scooletz

This comment has been minimized.

Show comment
Hide comment
@Scooletz

Scooletz Apr 21, 2017

Contributor

I'd say so. Revert them all.
As you bumped up .NET there's Volatile accessible now which would greatly simplify this code (no IntPtr magic included)

Contributor

Scooletz commented Apr 21, 2017

I'd say so. Revert them all.
As you bumped up .NET there's Volatile accessible now which would greatly simplify this code (no IntPtr magic included)

@gregoryyoung

This comment has been minimized.

Show comment
Hide comment
@gregoryyoung

gregoryyoung Apr 21, 2017

Member

@Scooletz I can also send over threadstacks (managed + gdbg) showing the problem. Basically processing locks (in this case on rebuilding an index). CPU pegs up threadstacks keep coming back withenqueue on the top. It being there is a while(true) it seems like a reasonable candidate for an access pattern problem that loops forever. We have a reproduction now, a reasonable test would be to try limit that to some large number (say 50k) and see if the issue still occurs.

a perflog shows the enqueue methods using CPU.

Here is a thread stack:

"Finalizer"
"Threadpool worker"
"MainQueue" at EventStore.Core.Bus.MPSCMessageQueue.Enqueue (EventStore.Core.Messaging.Message) <0x00124>
at EventStore.Core.Bus.QueuedHandlerAutoResetWithMpsc.Publish (EventStore.Core.Messaging.Message) <0x0002b>
at EventStore.Core.Services.Transport.Http.HttpService.Handle (EventStore.Core.Messages.HttpMessage/PurgeTimedOutRequests) <0x000cc>
at EventStore.Core.Bus.MessageHandler1<T_REF>.TryHandle (EventStore.Core.Messaging.Message) <0x000af> at EventStore.Core.Bus.InMemoryBus.Publish (EventStore.Core.Messaging.Message) <0x000db> at EventStore.Core.Services.VNode.VNodeFSMHandling1/c__AnonStorey1<TMessage_REF>.<>m__0 (EventStore.Core.Data.VNodeState,EventStore.Core.Messaging.Message) <0x00058>
at EventStore.Core.Services.VNode.VNodeFSM.Handle (EventStore.Core.Messaging.Message) <0x00133>
at EventStore.Core.Services.VNode.ClusterVNodeController.EventStore.Core.Bus.IHandle<EventStore.Core.Messaging.Message>.Handle (EventStore.Core.Messaging.Message) <0x0002b>
at EventStore.Core.Bus.QueuedHandlerAutoResetWithMpsc.ReadFromQueue (object) <0x00243>
at System.Threading.ThreadHelper.ThreadStart_Context (object) <0x000d1>
at System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext,System.Threading.ContextCallback,object,bool) <0x001c6>
at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext,System.Threading.ContextCallback,object,bool) <0x00020>
at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext,System.Threading.ContextCallback,object) <0x00059>
at System.Threading.ThreadHelper.ThreadStart (object) <0x0004c>
at (wrapper runtime-invoke) .runtime_invoke_void__this___object (object,intptr,intptr,intptr) <0x000ed>

Member

gregoryyoung commented Apr 21, 2017

@Scooletz I can also send over threadstacks (managed + gdbg) showing the problem. Basically processing locks (in this case on rebuilding an index). CPU pegs up threadstacks keep coming back withenqueue on the top. It being there is a while(true) it seems like a reasonable candidate for an access pattern problem that loops forever. We have a reproduction now, a reasonable test would be to try limit that to some large number (say 50k) and see if the issue still occurs.

a perflog shows the enqueue methods using CPU.

Here is a thread stack:

"Finalizer"
"Threadpool worker"
"MainQueue" at EventStore.Core.Bus.MPSCMessageQueue.Enqueue (EventStore.Core.Messaging.Message) <0x00124>
at EventStore.Core.Bus.QueuedHandlerAutoResetWithMpsc.Publish (EventStore.Core.Messaging.Message) <0x0002b>
at EventStore.Core.Services.Transport.Http.HttpService.Handle (EventStore.Core.Messages.HttpMessage/PurgeTimedOutRequests) <0x000cc>
at EventStore.Core.Bus.MessageHandler1<T_REF>.TryHandle (EventStore.Core.Messaging.Message) <0x000af> at EventStore.Core.Bus.InMemoryBus.Publish (EventStore.Core.Messaging.Message) <0x000db> at EventStore.Core.Services.VNode.VNodeFSMHandling1/c__AnonStorey1<TMessage_REF>.<>m__0 (EventStore.Core.Data.VNodeState,EventStore.Core.Messaging.Message) <0x00058>
at EventStore.Core.Services.VNode.VNodeFSM.Handle (EventStore.Core.Messaging.Message) <0x00133>
at EventStore.Core.Services.VNode.ClusterVNodeController.EventStore.Core.Bus.IHandle<EventStore.Core.Messaging.Message>.Handle (EventStore.Core.Messaging.Message) <0x0002b>
at EventStore.Core.Bus.QueuedHandlerAutoResetWithMpsc.ReadFromQueue (object) <0x00243>
at System.Threading.ThreadHelper.ThreadStart_Context (object) <0x000d1>
at System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext,System.Threading.ContextCallback,object,bool) <0x001c6>
at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext,System.Threading.ContextCallback,object,bool) <0x00020>
at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext,System.Threading.ContextCallback,object) <0x00059>
at System.Threading.ThreadHelper.ThreadStart (object) <0x0004c>
at (wrapper runtime-invoke) .runtime_invoke_void__this___object (object,intptr,intptr,intptr) <0x000ed>

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment