Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Potential 100% CPU hang in some circumstances #541

Closed
driis opened this issue Sep 28, 2018 · 2 comments
Closed

Potential 100% CPU hang in some circumstances #541

driis opened this issue Sep 28, 2018 · 2 comments
Labels

Comments

@driis
Copy link
Contributor

driis commented Sep 28, 2018

A service we have in production using EventFlow started exhibiting 100% CPU hangs periodically.

We have identified that this happens due to concurrent, not synchronized, access to the dictionaries used in ReadModelSqlGenerator. A single instance of this class is shared between all threads and requests.

This call stack (redacted) was found on multiple threads in a hang dump taken from the process:

0:023> !clrstack
OS Thread Id: 0x4cc0 (23)
        Child SP               IP Call Site
000000033173c9d0 00007ffc27c0669e System.Collections.Generic.Dictionary`2[[System.__Canon, mscorlib],[System.__Canon, mscorlib]].FindEntry(System.__Canon) [f:\dd\ndp\clr\src\BCL\system\collections\generic\dictionary.cs @ 298]
000000033173ca40 00007ffc27c05e64 System.Collections.Generic.Dictionary`2[[System.__Canon, mscorlib],[System.__Canon, mscorlib]].TryGetValue(System.__Canon, System.__Canon ByRef) [f:\dd\ndp\clr\src\BCL\system\collections\generic\dictionary.cs @ 499]
000000033173ca80 00007ffbd3d38c1a EventFlow.Sql.ReadModels.ReadModelSqlGenerator.CreateUpdateSql[[System.__Canon, mscorlib]]()
000000033173cb30 00007ffbd3d244a6 EventFlow.MsSql.ReadStores.MssqlReadModelStore`1+d__7[[System.__Canon, mscorlib]].MoveNext()

Replacing the dictionaries in the ReadModelSqlGenerator with ConcurrentDictionary<,> should fix this issue. I intend to provide a pull request for this shortly.

@driis driis mentioned this issue Sep 28, 2018
@rasmus rasmus added the bug label Sep 28, 2018
@rasmus
Copy link
Member

rasmus commented Sep 28, 2018

There's a blog post on MSDN here on the subject as well
https://blogs.msdn.microsoft.com/tess/2009/12/21/high-cpu-in-net-app-using-a-static-generic-dictionary/

@rasmus
Copy link
Member

rasmus commented Sep 28, 2018

Automatically closed when the PR got merged. Releasing 0.66 with the fix now

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants