Catching CacheManager exceptions globally

[jasenf]

Hi,

We are using Redis as our backplane and would like to prevent the CacheManager exceptions from throwing when there is a connection exception as it short circuits a lot of code that should continue on (i.e. keep working even if the cache is down). How do we do this globally? There's no feasible way to wrap this around every call.

Specifically I'd like to prevent this from collapsing on us when we get in within our production Azure environment:

Message: An unknown error occurred when writing the message
StackExchange.Redis.ConnectionMultiplexer.ExecuteSyncImpl[T](Message message, ResultProcessor`1 processor, ServerEndPoint server) in /_/src/StackExchange.Redis/ConnectionMultiplexer.cs:2622
StackExchange.Redis.RedisBase.ExecuteSync[T](Message message, ResultProcessor`1 processor, ServerEndPoint server) in /_/src/StackExchange.Redis/RedisBase.cs:54
StackExchange.Redis.RedisDatabase.ScriptEvaluate(Byte[] hash, RedisKey[] keys, RedisValue[] values, CommandFlags flags) in /_/src/StackExchange.Redis/RedisDatabase.cs:1182
CacheManager.Redis.RedisCacheHandle`1.Eval(ScriptType scriptType, RedisKey redisKey, RedisValue[] values, CommandFlags flags):138
CacheManager.Redis.RedisCacheHandle`1+<>c__DisplayClass48_0.<GetCacheItemAndVersion>b__0()
CacheManager.Redis.RetryHelper.Retry[T](Func`1 retryme, Int32 timeOut, Int32 retries, ILogger logger):50
CacheManager.Redis.RedisCacheHandle`1.Retry[T](Func`1 retryme)
CacheManager.Redis.RedisCacheHandle`1.GetCacheItemAndVersion(String key, String region, Int32& version)
CacheManager.Redis.RedisCacheHandle`1.GetCacheItemInternal(String key, String region)
CacheManager.Redis.RedisCacheHandle`1.GetCacheItemInternal(String key)
CacheManager.Core.Internal.BaseCache`1.GetCacheItem(String key):12
CacheManager.Core.BaseCacheManager`1.GetCacheItemInternal(String key, String region):70
CacheManager.Core.BaseCacheManager`1.GetCacheItemInternal(String key)
CacheManager.Core.Internal.BaseCache`1.GetCacheItem(String key):12
CacheManager.Core.Internal.BaseCache`1.Get(String key):40
CacheManager.Core.Internal.BaseCache`1.Get[TOut](String key)
EFCoreSecondLevelCacheInterceptor.EFCacheManagerCoreProvider+<>c__DisplayClass6_0.<GetValue>b__0()
EFCoreSecondLevelCacheInterceptor.ReaderWriterLockProvider.TryReadLocked[T](Func`1 action, Int32 timeout):19
EFCoreSecondLevelCacheInterceptor.EFCacheManagerCoreProvider.GetValue(EFCacheKey cacheKey, EFCachePolicy cachePolicy):38
EFCoreSecondLevelCacheInterceptor.DbCommandInterceptorProcessor.ProcessExecutingCommands[T](DbCommand command, DbContext context, T result, String methodName):506
EFCoreSecondLevelCacheInterceptor.SecondLevelCacheInterceptor.ReaderExecutingAsync(DbCommand command, CommandEventData eventData, InterceptionResult`1 result, CancellationToken cancellationToken):8

thanks,
jasen

[MichaCo]

I'm not really sure what you mean by that exactly and what EFCacheManager is.
This library can only do so much to ensure the code is working, which is, trying to make sure the call succeeds but eventually will fail.
Consumers have to react to that as needed. In most cases, errors are fine and can happen.
If in your case errors have to be handled, that has to be done in a layer on top of CacheManager I think.

But if you can explain the concrete use case with a good example, maybe I can help you out ;)

[jasenf]

I'm pretty sure I understand what all these components are :-)

What I'm saying is that, in most scenarios, caching is not an choke point on failing out an entire transaction. The design decision to throw an exception on anything that happens within the cache, in my case Redis connection errors, means that entire transaction has to deal with those exceptions.

Since most caching exceptions can probably be ignored, it would be nice if CacheManager supported either a configuration property to not throw any exceptions, or provided a global handler that would let us create a single point of managing them.

The problem is compounded even further when other third parties implement your library. For example, our primary caching implementation is done using EFCoreSecondLevelCache, which perfectly integrates your library as a provider. The end result is we have thousands of entity framework calls ultimately implementing your library with simple EF queries. These thousands of calls look something as basic as:

var x = await _context.MyTable.Where(....).Cacheable().ToListAsync()

We don't want to, nor do I think it would be a proper implementation to, to wrap all of our calls to deal with caching exceptions on the off chance that a cache provider is not available. With database transactions that's obviously important, but with caching, it simply is not critical path and we want the "normal" operations to continue, even if cache storage fails. This is especially important with simple cache GETS. I understand we can cause stale caches if exceptions are ignored on PUTS, but if the consumer of the cache doesn't care about potentially stale data, we should have that option.

I was just wondering if there was a way to handle this scenario globally without having to account for it on every, in our case, query.

Thanks again for this great library btw, I think that often times doesn't get said enough.

[jasenf]

any chance of a "silent" option to not throw exceptions and interrupt the pipeline if caching fails?

[MichaCo]

It is very unlikely I'd add a single option to disable any exceptions. That would be extremely bad for most use cases.
I'd probably go for a more extendable error/retry handling.

Not sure yet what the best way would be or when that would be released though, sorry.

[jasenf]

That would be my preferred suggestion as well, thank you! We have a high volume web environment and we've found 99% of the exceptions that get thrown are Redis connections. Perhaps there's an easy way to manage that one globally?