System.Web.Caching.Cache data race on LRU list

[dmitryvk]

During load testing of our web application we noticed that during high load there are intermittent errors related to System.Web.Caching which result in application being unusable (all further requests return HTTP 500). This happens rarely and under prolonged load tests.

I've been able to reduce this problem to data race inside System.Web.Caching.CacheItemLRU.this[string].get which is called from System.Web.Caching.Cache.GetKeyLastChange(string) with only read lock held. But this getter modifies LRU list (contained in list local variable).

Steps to Reproduce

Minimal reproduction that runs as console application (without starting web application)

1. Get source code from https://gist.github.com/dmitryvk/30d4a3ce117104adafc5138efaceef3c
2. Compile and run it (as console application)

Current Behavior

Application almost immediately crashes with different exceptions each time. See below for several examples.

Expected Behavior

Applications keeps running and prints infinite amount of interleaving + and * characters

On which platforms did you notice this

[ ] macOS
[x] Linux
[ ] Windows

Version Used:

Mono JIT compiler version 5.16.0.220 (tarball Mon Nov 26 17:05:57 UTC 2018)
Copyright (C) 2002-2014 Novell, Inc, Xamarin Inc and Contributors. www.mono-project.com
        TLS:           __thread
        SIGSEGV:       altstack
        Notifications: epoll
        Architecture:  amd64
        Disabled:      none
        Misc:          softdebug 
        Interpreter:   yes
        LLVM:          yes(3.6.0svn-mono-/)
        GC:            sgen (concurrent by default)

Stacktrace

System.Collections.Generic.KeyNotFoundException: The given key 'System.Collections.Generic.LinkedListNode`1[System.Web.Caching.CacheItem]' was not present in the dictionary.
  at System.Collections.Generic.Dictionary`2[TKey,TValue].get_Item (TKey key) [0x0001e] in <7c6726ddba2042189622013443252667>:0 
  at System.Web.Caching.CacheItemLRU.EvictIfNecessary () [0x00028] in <979c6d3c1df54a4b9abbd8b5ecbe7cbb>:0 
  at System.Web.Caching.Cache.SetItemTimeout (System.Web.Caching.CacheItem ci, System.DateTime absoluteExpiration, System.TimeSpan slidingExpiration, System.Web.Caching.CacheItemRemovedCallback onRemoveCallback, System.Web.Caching.CacheItemUpdateCallback onUpdateCallback, System.String key, System.Boolean doLock) [0x00069] in <979c6d3c1df54a4b9abbd8b5ecbe7cbb>:0 
  at System.Web.Caching.Cache.Insert (System.String key, System.Object value, System.Web.Caching.CacheDependency dependencies, System.DateTime absoluteExpiration, System.TimeSpan slidingExpiration, System.Web.Caching.CacheItemPriority priority, System.Web.Caching.CacheItemRemovedCallback onRemoveCallback, System.Web.Caching.CacheItemUpdateCallback onUpdateCallback, System.Boolean doLock) [0x000b7] in <979c6d3c1df54a4b9abbd8b5ecbe7cbb>:0 
  at System.Web.Caching.Cache.Add (System.String key, System.Object value, System.Web.Caching.CacheDependency dependencies, System.DateTime absoluteExpiration, System.TimeSpan slidingExpiration, System.Web.Caching.CacheItemPriority priority, System.Web.Caching.CacheItemRemovedCallback onRemoveCallback) [0x00033] in <979c6d3c1df54a4b9abbd8b5ecbe7cbb>:0 
  at ConsoleApplication10.Program.Adder (System.Web.Caching.Cache cache) [0x00012] in <74a7b3b5d7d44f67a275f3c44d703ba9>:0 

System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.InvalidOperationException: The LinkedList node does not belong to current LinkedList.
  at System.Collections.Generic.LinkedList`1[T].ValidateNode (System.Collections.Generic.LinkedListNode`1[T] node) [0x00017] in <102ddc49e9884872921866bb2bd1bab6>:0 
  at System.Collections.Generic.LinkedList`1[T].Remove (System.Collections.Generic.LinkedListNode`1[T] node) [0x00000] in <102ddc49e9884872921866bb2bd1bab6>:0 
  at System.Web.Caching.CacheItemLRU.get_Item (System.String key) [0x00028] in <979c6d3c1df54a4b9abbd8b5ecbe7cbb>:0 
  at System.Web.Caching.Cache.GetKeyLastChange (System.String key) [0x0000b] in <979c6d3c1df54a4b9abbd8b5ecbe7cbb>:0 
  at (wrapper managed-to-native) System.Reflection.MonoMethod.InternalInvoke(System.Reflection.MonoMethod,object,object[],System.Exception&)
  at System.Reflection.MonoMethod.Invoke (System.Object obj, System.Reflection.BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) [0x0003b] in <7c6726ddba2042189622013443252667>:0 
   --- End of inner exception stack trace ---
  at System.Reflection.MonoMethod.Invoke (System.Object obj, System.Reflection.BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) [0x00051] in <7c6726ddba2042189622013443252667>:0 
  at System.Reflection.MethodBase.Invoke (System.Object obj, System.Object[] parameters) [0x00000] in <7c6726ddba2042189622013443252667>:0 
  at ConsoleApplication10.Program.CheckGetKeyLastChange (System.Web.Caching.Cache cache) [0x0001f] in <74a7b3b5d7d44f67a275f3c44d703ba9>:0 

System.ArgumentNullException: Value cannot be null.
Parameter name: key
  at System.Collections.Generic.Dictionary`2[TKey,TValue].FindEntry (TKey key) [0x00008] in <7c6726ddba2042189622013443252667>:0 
  at System.Collections.Generic.Dictionary`2[TKey,TValue].get_Item (TKey key) [0x00000] in <7c6726ddba2042189622013443252667>:0 
  at System.Web.Caching.CacheItemLRU.EvictIfNecessary () [0x00028] in <979c6d3c1df54a4b9abbd8b5ecbe7cbb>:0 
  at System.Web.Caching.Cache.SetItemTimeout (System.Web.Caching.CacheItem ci, System.DateTime absoluteExpiration, System.TimeSpan slidingExpiration, System.Web.Caching.CacheItemRemovedCallback onRemoveCallback, System.Web.Caching.CacheItemUpdateCallback onUpdateCallback, System.String key, System.Boolean doLock) [0x00069] in <979c6d3c1df54a4b9abbd8b5ecbe7cbb>:0 
  at System.Web.Caching.Cache.Insert (System.String key, System.Object value, System.Web.Caching.CacheDependency dependencies, System.DateTime absoluteExpiration, System.TimeSpan slidingExpiration, System.Web.Caching.CacheItemPriority priority, System.Web.Caching.CacheItemRemovedCallback onRemoveCallback, System.Web.Caching.CacheItemUpdateCallback onUpdateCallback, System.Boolean doLock) [0x000b7] in <979c6d3c1df54a4b9abbd8b5ecbe7cbb>:0 
  at System.Web.Caching.Cache.Add (System.String key, System.Object value, System.Web.Caching.CacheDependency dependencies, System.DateTime absoluteExpiration, System.TimeSpan slidingExpiration, System.Web.Caching.CacheItemPriority priority, System.Web.Caching.CacheItemRemovedCallback onRemoveCallback) [0x00033] in <979c6d3c1df54a4b9abbd8b5ecbe7cbb>:0 
  at ConsoleApplication10.Program.Adder (System.Web.Caching.Cache cache) [0x00012] in <74a7b3b5d7d44f67a275f3c44d703ba9>:0