Deadlock detection?

[oconnor0]

I ran into a deadlock in the process of converting some older caching code to use FusionCache. The below program reproduces it. Basically the idea was that GetInt should set the GetDict cache value when it is called. We can assume that in the real application the entire process was complex enough to (at least at one time) to have been worth the goofiness.

This obviously (in retrospect) deadlocks. If I uncomment LockTimeout, the interior GetOrSet of "d" fails silently but the program runs correctly. If I replace the interior GetOrSet of "d" with a Set of "d", the program runs correctly.

This all makes sense tho I clearly missed it and made some mistakes in my conversion. My question is: what support does FusionCache have for deadlock detection? This is obviously an easy case: a single call chain GetOrSets "d", "i", and then "d" so the cache stampede protection kicks in and prevents the innermost "d" from making progress. I know that, in general, deadlock detection is hard, and I recognize that in production it may not be worth the performance penalty when it could be caught in development.

It's worth mentioning that, with a debugger attached, the real caching object construction methods involved are complex & slow enough that it's not always obvious if something has deadlocked or is still plugging along 5 minutes later.

#:package ZiggyCreatures.FusionCache@2.4.0

using ZiggyCreatures.Caching.Fusion;
using System.Collections.Generic;

var options = new FusionCacheOptions() {
    DefaultEntryOptions = new FusionCacheEntryOptions {
        // LockTimeout = TimeSpan.FromSeconds(1),
    },
};
var cache = new FusionCache(options);
var d = GetDict();
foreach (var (k, v) in d) {
    System.Console.WriteLine($"[{k}] = {v}");
}

IReadOnlyDictionary<string, int> BuildDict(int i) {
    return new Dictionary<string, int>() {
        ["i"] = i,
    };
}

int GetInt() {
    return cache.GetOrSet<int>("i", ct => {
        var i = 10;
        // cache.Set("d", () => BuildDict(i));
        cache.GetOrSet("d", () => BuildDict(i));
        return i;
    });
}

IReadOnlyDictionary<string, int> GetDict() {
    return cache.GetOrSet<IReadOnlyDictionary<string, int>>("d", ct => {
        var i = GetInt();
        return BuildDict(i);
    });
}

[jodydonetti]

Hi @oconnor0 , currently a lock cannot be acquired twice (see: re-entrancy) by the same subject.
So yes, trying to do that will inadvertently get into a deadlock.

Currently in FusionCache the component responsible for it is the StandardDistributedLocker (default implementation of the IFusionCacheMemoryLocker core abstraction), which internally uses a series of SemaphoreSlims.

I thought about thsi in the past, but I've never been sure if it was possible at all, at least considering all the constraints:

• must support both sync and async
• must support acquisition timeouts
• timeout can even be zero (e.g.: TimeSpan.Zero), because it's used for Eager Refresh and other stuff

Also, there's this.

Opinions? Let me know what you think.

[oconnor0]

I was trying to think through if, because this is deadlock detection, catching any cases where the same lock is attempted to be acquired in the same AsyncContext would be a viable option - since it might not have to be perfect. But... false positives & false negatives in a system like this are going to be annoying.

So, yes, all of this seems to indicate that maybe it's just not worth trying to implement. And might not even be possible.