Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

System.Web.Caching.Cache data race on LRU list #12218

Closed
dmitryvk opened this issue Dec 28, 2018 · 0 comments · Fixed by #12219

Comments

@dmitryvk
Copy link
Contributor

@dmitryvk dmitryvk commented Dec 28, 2018

During load testing of our web application we noticed that during high load there are intermittent errors related to System.Web.Caching which result in application being unusable (all further requests return HTTP 500). This happens rarely and under prolonged load tests.

I've been able to reduce this problem to data race inside System.Web.Caching.CacheItemLRU.this[string].get which is called from System.Web.Caching.Cache.GetKeyLastChange(string) with only read lock held. But this getter modifies LRU list (contained in list local variable).

Steps to Reproduce

Minimal reproduction that runs as console application (without starting web application)

  1. Get source code from https://gist.github.com/dmitryvk/30d4a3ce117104adafc5138efaceef3c
  2. Compile and run it (as console application)

Current Behavior

Application almost immediately crashes with different exceptions each time. See below for several examples.

Expected Behavior

Applications keeps running and prints infinite amount of interleaving + and * characters

On which platforms did you notice this

[ ] macOS
[x] Linux
[ ] Windows

Version Used:

Mono JIT compiler version 5.16.0.220 (tarball Mon Nov 26 17:05:57 UTC 2018)
Copyright (C) 2002-2014 Novell, Inc, Xamarin Inc and Contributors. www.mono-project.com
        TLS:           __thread
        SIGSEGV:       altstack
        Notifications: epoll
        Architecture:  amd64
        Disabled:      none
        Misc:          softdebug 
        Interpreter:   yes
        LLVM:          yes(3.6.0svn-mono-/)
        GC:            sgen (concurrent by default)

Stacktrace

System.Collections.Generic.KeyNotFoundException: The given key 'System.Collections.Generic.LinkedListNode`1[System.Web.Caching.CacheItem]' was not present in the dictionary.
  at System.Collections.Generic.Dictionary`2[TKey,TValue].get_Item (TKey key) [0x0001e] in <7c6726ddba2042189622013443252667>:0 
  at System.Web.Caching.CacheItemLRU.EvictIfNecessary () [0x00028] in <979c6d3c1df54a4b9abbd8b5ecbe7cbb>:0 
  at System.Web.Caching.Cache.SetItemTimeout (System.Web.Caching.CacheItem ci, System.DateTime absoluteExpiration, System.TimeSpan slidingExpiration, System.Web.Caching.CacheItemRemovedCallback onRemoveCallback, System.Web.Caching.CacheItemUpdateCallback onUpdateCallback, System.String key, System.Boolean doLock) [0x00069] in <979c6d3c1df54a4b9abbd8b5ecbe7cbb>:0 
  at System.Web.Caching.Cache.Insert (System.String key, System.Object value, System.Web.Caching.CacheDependency dependencies, System.DateTime absoluteExpiration, System.TimeSpan slidingExpiration, System.Web.Caching.CacheItemPriority priority, System.Web.Caching.CacheItemRemovedCallback onRemoveCallback, System.Web.Caching.CacheItemUpdateCallback onUpdateCallback, System.Boolean doLock) [0x000b7] in <979c6d3c1df54a4b9abbd8b5ecbe7cbb>:0 
  at System.Web.Caching.Cache.Add (System.String key, System.Object value, System.Web.Caching.CacheDependency dependencies, System.DateTime absoluteExpiration, System.TimeSpan slidingExpiration, System.Web.Caching.CacheItemPriority priority, System.Web.Caching.CacheItemRemovedCallback onRemoveCallback) [0x00033] in <979c6d3c1df54a4b9abbd8b5ecbe7cbb>:0 
  at ConsoleApplication10.Program.Adder (System.Web.Caching.Cache cache) [0x00012] in <74a7b3b5d7d44f67a275f3c44d703ba9>:0 
System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.InvalidOperationException: The LinkedList node does not belong to current LinkedList.
  at System.Collections.Generic.LinkedList`1[T].ValidateNode (System.Collections.Generic.LinkedListNode`1[T] node) [0x00017] in <102ddc49e9884872921866bb2bd1bab6>:0 
  at System.Collections.Generic.LinkedList`1[T].Remove (System.Collections.Generic.LinkedListNode`1[T] node) [0x00000] in <102ddc49e9884872921866bb2bd1bab6>:0 
  at System.Web.Caching.CacheItemLRU.get_Item (System.String key) [0x00028] in <979c6d3c1df54a4b9abbd8b5ecbe7cbb>:0 
  at System.Web.Caching.Cache.GetKeyLastChange (System.String key) [0x0000b] in <979c6d3c1df54a4b9abbd8b5ecbe7cbb>:0 
  at (wrapper managed-to-native) System.Reflection.MonoMethod.InternalInvoke(System.Reflection.MonoMethod,object,object[],System.Exception&)
  at System.Reflection.MonoMethod.Invoke (System.Object obj, System.Reflection.BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) [0x0003b] in <7c6726ddba2042189622013443252667>:0 
   --- End of inner exception stack trace ---
  at System.Reflection.MonoMethod.Invoke (System.Object obj, System.Reflection.BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) [0x00051] in <7c6726ddba2042189622013443252667>:0 
  at System.Reflection.MethodBase.Invoke (System.Object obj, System.Object[] parameters) [0x00000] in <7c6726ddba2042189622013443252667>:0 
  at ConsoleApplication10.Program.CheckGetKeyLastChange (System.Web.Caching.Cache cache) [0x0001f] in <74a7b3b5d7d44f67a275f3c44d703ba9>:0 
System.ArgumentNullException: Value cannot be null.
Parameter name: key
  at System.Collections.Generic.Dictionary`2[TKey,TValue].FindEntry (TKey key) [0x00008] in <7c6726ddba2042189622013443252667>:0 
  at System.Collections.Generic.Dictionary`2[TKey,TValue].get_Item (TKey key) [0x00000] in <7c6726ddba2042189622013443252667>:0 
  at System.Web.Caching.CacheItemLRU.EvictIfNecessary () [0x00028] in <979c6d3c1df54a4b9abbd8b5ecbe7cbb>:0 
  at System.Web.Caching.Cache.SetItemTimeout (System.Web.Caching.CacheItem ci, System.DateTime absoluteExpiration, System.TimeSpan slidingExpiration, System.Web.Caching.CacheItemRemovedCallback onRemoveCallback, System.Web.Caching.CacheItemUpdateCallback onUpdateCallback, System.String key, System.Boolean doLock) [0x00069] in <979c6d3c1df54a4b9abbd8b5ecbe7cbb>:0 
  at System.Web.Caching.Cache.Insert (System.String key, System.Object value, System.Web.Caching.CacheDependency dependencies, System.DateTime absoluteExpiration, System.TimeSpan slidingExpiration, System.Web.Caching.CacheItemPriority priority, System.Web.Caching.CacheItemRemovedCallback onRemoveCallback, System.Web.Caching.CacheItemUpdateCallback onUpdateCallback, System.Boolean doLock) [0x000b7] in <979c6d3c1df54a4b9abbd8b5ecbe7cbb>:0 
  at System.Web.Caching.Cache.Add (System.String key, System.Object value, System.Web.Caching.CacheDependency dependencies, System.DateTime absoluteExpiration, System.TimeSpan slidingExpiration, System.Web.Caching.CacheItemPriority priority, System.Web.Caching.CacheItemRemovedCallback onRemoveCallback) [0x00033] in <979c6d3c1df54a4b9abbd8b5ecbe7cbb>:0 
  at ConsoleApplication10.Program.Adder (System.Web.Caching.Cache cache) [0x00012] in <74a7b3b5d7d44f67a275f3c44d703ba9>:0 
dmitryvk pushed a commit to barsgroup/mono that referenced this issue Dec 28, 2018
System.Web.Caching.CacheItemLRU.this[string].get modifies LRU list so it
must be called under write lock. Cache.GetKeyLastChange violates this
constraint by acquiring only a read lock.

Fixes mono#12218
alexischr added a commit that referenced this issue Dec 31, 2018
System.Web.Caching.CacheItemLRU.this[string].get modifies LRU list so it
must be called under write lock. Cache.GetKeyLastChange violates this
constraint by acquiring only a read lock.

Fixes #12218
jonpryor added a commit to xamarin/xamarin-android that referenced this issue Apr 24, 2019
Bumps to mono/api-snapshot@ae01378
Bumps to mono/reference-assemblies@e5173a5
Bumps to mono/bockbuild@d30329d
Bumps to mono/boringssl@3d87996
Bumps to mono/corefx@72f7d76
Bumps to mono/corert@1b7d4a1
Bumps to mono/helix-binaries@7e893ea
Bumps to mono/illinker-test-assets@f21ff68
Bumps to mono/linker@13d864e
Bumps to mono/llvm@1aaaaa5 [mono]
Bumps to mono/llvm@2c2cffe [xamarin-android]
Bumps to mono/NUnitLite@0029561
Bumps to mono/roslyn-binaries@0bbc9b4
Bumps to mono/xunit-binaries@8f6e62e

	$ git diff --shortstat 886c4901..e66c7667      # mono
        3597 files changed, 350850 insertions(+), 91128 deletions(-)
	$ git diff --shortstat 349752c464c5fc93b32e7d45825f2890c85c8b7d..2c2cffedf01e0fe266b9aaad2c2563e05b750ff4
	 240 files changed, 18562 insertions(+), 6581 deletions(-)

Context: dotnet/coreclr#22046

Fixes: CVE 2018-8292 on macOS
Fixes: http://work.devdiv.io/737323
Fixes: dotnet/corefx#33965
Fixes: dotnet/standard#642
Fixes: mono/mono#6997
Fixes: mono/mono#7326
Fixes: mono/mono#7517
Fixes: mono/mono#7750
Fixes: mono/mono#7859
Fixes: mono/mono#8360
Fixes: mono/mono#8460
Fixes: mono/mono#8766
Fixes: mono/mono#8922
Fixes: mono/mono#9418
Fixes: mono/mono#9507
Fixes: mono/mono#9951
Fixes: mono/mono#10024
Fixes: mono/mono#10030
Fixes: mono/mono#10038
Fixes: mono/mono#10448
Fixes: mono/mono#10735
Fixes: mono/mono#10735
Fixes: mono/mono#10737
Fixes: mono/mono#10743
Fixes: mono/mono#10834
Fixes: mono/mono#10837
Fixes: mono/mono#10838
Fixes: mono/mono#10863
Fixes: mono/mono#10945
Fixes: mono/mono#11020
Fixes: mono/mono#11021
Fixes: mono/mono#11021
Fixes: mono/mono#11049
Fixes: mono/mono#11091
Fixes: mono/mono#11095
Fixes: mono/mono#11123
Fixes: mono/mono#11138
Fixes: mono/mono#11146
Fixes: mono/mono#11202
Fixes: mono/mono#11214
Fixes: mono/mono#11317
Fixes: mono/mono#11326
Fixes: mono/mono#11378
Fixes: mono/mono#11385
Fixes: mono/mono#11478
Fixes: mono/mono#11479
Fixes: mono/mono#11488
Fixes: mono/mono#11489
Fixes: mono/mono#11527
Fixes: mono/mono#11529
Fixes: mono/mono#11596
Fixes: mono/mono#11603
Fixes: mono/mono#11613
Fixes: mono/mono#11623
Fixes: mono/mono#11663
Fixes: mono/mono#11681
Fixes: mono/mono#11684
Fixes: mono/mono#11693
Fixes: mono/mono#11697
Fixes: mono/mono#11779
Fixes: mono/mono#11809
Fixes: mono/mono#11858
Fixes: mono/mono#11895
Fixes: mono/mono#11898
Fixes: mono/mono#11898
Fixes: mono/mono#11965
Fixes: mono/mono#12182
Fixes: mono/mono#12193
Fixes: mono/mono#12218
Fixes: mono/mono#12235
Fixes: mono/mono#12263
Fixes: mono/mono#12307
Fixes: mono/mono#12331
Fixes: mono/mono#12362
Fixes: mono/mono#12374
Fixes: mono/mono#12402
Fixes: mono/mono#12421
Fixes: mono/mono#12461
Fixes: mono/mono#12479
Fixes: mono/mono#12479
Fixes: mono/mono#12552
Fixes: mono/mono#12603
Fixes: mono/mono#12747
Fixes: mono/mono#12831
Fixes: mono/mono#12843
Fixes: mono/mono#12881
Fixes: mono/mono#13030
Fixes: mono/mono#13284
Fixes: mono/mono#13297
Fixes: mono/mono#13455
Fixes: mono/mono#13460
Fixes: mono/mono#13478
Fixes: mono/mono#13479
Fixes: mono/mono#13522
Fixes: mono/mono#13607
Fixes: mono/mono#13610
Fixes: mono/mono#13610
Fixes: mono/mono#13639
Fixes: mono/mono#13672
Fixes: mono/mono#13834
Fixes: mono/mono#13878
Fixes: mono/mono#6352
Fixes: mono/monodevelop#6898
Fixes: xamarin/maccore#1069
Fixes: xamarin/maccore#1407
Fixes: xamarin/maccore#604
Fixes: xamarin/xamarin-macios#4984
Fixes: xamarin/xamarin-macios#5289
Fixes: xamarin/xamarin-macios#5363
Fixes: xamarin/xamarin-macios#5381
Fixes: https://issuetracker.unity3d.com/issues/editor-crashes-with-g-logv-when-entering-play-mode-with-active-flowcanvas-script
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.