Hybrid Cache API proposal

[mgravell]

This is the API proposal related to Epic: IDistributedCache updates in .NET 9

Hybrid Cache specification

Hybrid Cache is a new API designed to build on top of the existing Microsoft.Extensions.Caching.Distributed.IDistributedCache API, to fill multiple functional gaps in the usability of the IDistributedCache API,
including:

• stampede protection
• simple pass-thru API usage (i.e. a single method replaces multiple discrete steps required with the old API)
• multi-tier (in-process plus backend) caching
• configurable serialization
• tag-based eviction
• metrics

Overview

The primary API is a new abstract class, HybridCache, in a new Microsoft.Extensions.Caching.Distributed package:

namespace Microsoft.Extensions.Caching.Distributed;

public abstract class HybridCache
{ /* more detail below */ }

This type acts as the primary API that users will interact with for caching using this feature, replacing IDistributedCache (which now becomes a backend API); the purpose of HybridCache
is to encapsulate the state required to implement new functionality. This required additional state means that the feature cannot be implemented simply as extension methods
on top of IDistributedCache - for example for stampede protection we need to track a bucket of in-flight operations so that we can join existing backend operations. Every feature listed
(except perhaps for the pass-thru API usage) requires some state or additional service.

Microsoft will provide a concrete implementation of HybridCache via dependency injection, but it is explicitly intended that the API can be implemented independently if desired.

Why "Hybrid Cache"?

This name seems to capture the multiple roles being fulfilled by the cache implementation. A number of otions have been considered, including "read thru cache",
"advanced cache", "distributed cache 2"; this seems to work, though.

Why not IHybridCache?

1. the primary pass-thru API (discussed below) exists in a dual "stateful"/"stateless" mode, with it being possible to reliably and automatically implement one via the other;
   providing this at the definition level halves this aspect of the API surface for concrete implementations, providing a consistent experince
2. it is anticipated that additional future capabilities will be desired on this API; if we limit this as IHybridCache it is harder to extend than with an abstract base class that
   can implement features with default implementations that implementors can override as desired

It is noted that in both cases, "default interface methods", also serve this function; if provide a mechanism to achieve this same goal with an IHybridCache approach.
If we feel that "default interface methods" are now fully greenlit for this scenario, we could indeed use an IHybridCache approach.

---

Registering and configuring HybridCache

Registering hybrid cache is performed by HybridCacheServiceExtensions:

namespace Microsoft.Extensions.DependencyInjection;

public static class HybridCacheServiceExtensions
{
    // adds HybridCache using default options
    public static IHybridCacheBuilder AddHybridCache(this IServiceCollection services);
    // adds HybridCache using custom options
    public static IHybridCacheBuilder AddHybridCache(this IServiceCollection services, Action<HybridCacheOptions> configureOptions);

    // adds TImplementation via DI as the serializer for T
    public static IHybridCacheBuilder WithSerializer<T, [DynamicallyAccessedMembers(DynamicallyAccessedMemberTypes.PublicConstructors)] TImplementation>(
        this IHybridCacheBuilder builder)
        where TImplementation : class, IHybridCacheSerializer<T>;
    // adds a concrete custom serializer for a given type
    public static IHybridCacheBuilder WithSerializer<T>(this IHybridCacheBuilder builder, IHybridCacheSerializer<T> serializer);

    // adds T via DI as a serializer factory
    public static IHybridCacheBuilder WithSerializerFactory<[DynamicallyAccessedMembers(DynamicallyAccessedMemberTypes.PublicConstructors)] TImplementation>(
        this IHybridCacheBuilder builder)
        where TImplementation : class, IHybridCacheSerializerFactory;
    // adds a concrete custom serializer factory
    public static IHybridCacheBuilder WithSerializerFactory(this IHybridCacheBuilder builder, IHybridCacheSerializerFactory factory);
}

namespace Microsoft.Extensions.Caching.Distributed;

public interface IHybridCacheBuilder
{
    IServiceCollection Services { get; }
}
public interface IHybridCacheSerializer<T>
{
    T Deserialize(ReadOnlySequence<byte> source);
    void Serialize(T value, IBufferWriter<byte> target);
}
public interface IHybridCacheSerializerFactory
{
    bool TryCreateSerializer<T>([NotNullWhen(true)] out IHybridCacheSerializer<T>? serializer);
}
public class HybridCacheOptions
{
    // default expiration etc configuration, if omitted
    public HybridCacheEntryOptions? DefaultOptions { get; set; }

    // quotas
    public long MaximumPayloadBytes { get; set; } = 1 << 20; // 1MiB
    public int MaximumKeyLength { get; set; } = 1024; // characters

    // whether compression is enabled
    public bool AllowCompression { get; set; } = true;

    // opt-in support for using "tags" as dimensions on metric reporting; this is opt-in
    // because "tags" could contain user data, which we do not want to expose by default
    public bool ReportTagMetrics { get; set; }
}
public class HybridCacheEntryOptions
{ /* more detail below */ }

where IHybridCacheBuilder here functions purely as a wrapper (via .Services) to provide contextual API services to configure related services such as serialization,
for API discoverability, for example making it trivial to configure serialization, rather than having to magically know about the existence of specific services that
can be added to influence behaviour. The return value is the same input services collection, for chaining purposes.

The HybridCacheOptions provides additional global options for the cache, including payload max quota and a default cache configuration (primarily: lifetime).

The user will often also wish to register an out-of-process IDistributedCache backend (Redis, SQL Server, etc) in the usual manner, as
discussed here. Note that this is not required; it is anticipated that simply having
the L1 cache with stampede protection against the backend provides compelling value. Options specific to the chosen IDistributedCache backend will
be configured as part of that IDistributedCache registration, and are not considered here.

---

Using HybridCache

The HybridCache instance will be dependency-injected into code that requires them; from there, the primary API is GetOrCreateAsync which provides
a stateless and stateful overload pair:

public abstract class HybridCache
{
    protected HybridCache() { }

    public abstract ValueTask<T> GetOrCreateAsync<TState, T>(string key, TState state, Func<TState, CancellationToken, ValueTask<T>> callback,
        HybridCacheEntryOptions? options = null, ReadOnlyMemory<string> tags = default, CancellationToken cancellationToken = default);

    public virtual ValueTask<T> GetOrCreateAsync<T>(string key, Func<CancellationToken, ValueTask<T>> callback,
        HybridCacheEntryOptions? options = null, ReadOnlyMemory<string> tags = default, CancellationToken cancellationToken = default)
        => // default implemention provided automatically via GetOrCreateAsync<TState, T>

    // ...

It should be noted that all APIs are designed as async, with ValueTask<T> used to respect that values may be available
synchronously (in the cache-hit case); however, the fact that we're caching means we can reasonably assume this operation
will be non-trivial, and possibly one or both of an an out-of-process backend store call (with non-trivial payload) and an underlying data fetch (with non-trivial total time); async is strongly desirable.

The simplest use-case is the stateless option, typically used with a lambda callback using "captured" state, for example:

public MyConsumerCode(HybridCache cache)
{
    public async Task<Customer> GetCustomer(Region region, int id)
        => cache.GetOrCreateAsync($"/customer/{region}/{id}", async _ => await SomeExternalBackend.GetAsync(region, id));
}

The GetOrCreateAsync name is chosen for parity with IMemoryCache; it
takes a string key, and a callback that is used to fetch the underlying data if it is not available in any other cache. In some high throughput scenarios, it may be
preferable to avoid this capture overhead using a static callback and the stateful overload:

public MyConsumerCode(HybridCache cache)
{
    public async Task<Customer> GetCustomer(Region region, int id)
        => cache.GetOrCreateAsync($"/customer/{region}/{id}", static (region, id) async (_, state) => await SomeExternalBackend.GetAsync(state.region, state.id));
}

Optionally, this API allows:

• HybridCacheEntryOptions, controlling the duration of the cache entry (see below)
• zero, one or more "tags", which work similarly to the "tags" feature of "Output Cache"
• cancellation

For the options, timeout is only described in relative terms:

public sealed class HybridCacheEntryOptions
{
    public HybridCacheEntryOptions(TimeSpan expiry, TimeSpan? localCacheExpiry = null, HybridCacheEntryFlags flags = 0);

    public TimeSpan Expiry { get; } // overall cache duration

    /// <summary>
    /// Cache duration in local cache; when retrieving a cached value
    /// from an external cache store, this value will be used to calculate the local
    /// cache expiration, not exceeding the remaining overall cache lifetime
    /// </summary>
    public TimeSpan LocalCacheExpiry { get; } // TTL in L1

    public HybridCacheEntryFlags Flags { get; }
}
[Flags]
public enum HybridCacheEntryFlags
{
    None = 0,
    DisableLocalCache = 1 << 0,
    DisableDistributedCache = 1 << 1,
    DisableCompression = 1 << 2,
}

The Flags also allow features such as specific caching tiers or compression to be electively disabled on a per-scenario basis. It will directed that
entry options should usually be shared (static readonly) and reused on a per-scenario basis. To this end, the type is immutable. If no options is supplied,
the default from HybridCacheOptions is used; this has an implied "reasonable" default timeout (low minutes, probably) in the eventuality that none is specified.

In many cases, GetOrCreateAsync is the only API needed, but additionally, HybridCache has auxiliary APIs:

public abstract ValueTask<HybridCacheEntry<T>?> GetAsync<T>(string key, HybridCacheEntryOptions? options = null, CancellationToken cancellationToken = default);
public abstract ValueTask SetAsync<T>(string key, T value, HybridCacheEntryOptions? options = null, ReadOnlyMemory<string> tags = default, CancellationToken cancellationToken = default);
public abstract ValueTask RemoveKeyAsync(string key, CancellationToken cancellationToken = default);
public virtual ValueTask RemoveKeysAsync(ReadOnlyMemory<string> keys, CancellationToken cancellationToken = default) // implemented via RemoveKeyAsync
public virtual ValueTask RemoveTagAsync(string tag, CancellationToken cancellationToken = default) // implemented via RemoveTags
public abstract ValueTask RemoveTagsAsync(ReadOnlyMemory<string> tags, CancellationToken cancellationToken = default)

// ...
public sealed class HybridCacheEntry<T>
{
    public T Value { get; set; } = default!;
    public ReadOnlyMemory<string> Tags { get; set; }
    public DateTime Expiry { get; set; } // absolute time of expiry
    public DateTime LocalExpiry { get; set; } // absolute time of L1 expiry
}

These APIs provide for explicit manual fetch/assignment, and for explicit invalidation at the key or tag level.
The HybridCacheEntry type is used only to encapsulate return state for GetAsync; a null response indicates
a cache-miss.

---

Backend services

CLARIFICATION: IBufferDistributedCache actually needs to go into the "runtime" repo (to sit alongside IDistributedCache, and avoid a framework reference); it is included here as a placeholder, but this part is actually over here

HOWEVER, timescales mean that "runtime" is likely to lag behind "asp.net", in which case in order to allow
further development etc ASAP, I propose "ship it in asp.net for now; when it gets into runtime, remove it from asp.net"

To provide the enhanced capabilities, some new additional services are required; IDistributedCache has both performance and feature limitations that make it incomplete for this purpose. For
out-of-process caches, the byte[] nature of IDistributedCache makes for allocation concerns, so a new API is optionally supported, based on similar work for Output Cache; however, the system functions without demanding it and all
pre-existing IDistributedCache implementations will continue to work. The system will type-test for the new capability:

namespace Microsoft.Extensions.Caching.Distributed;

public interface IBufferDistributedCache : IDistributedCache
{
    bool TryGet(string key, IBufferWriter<byte> destination);
    ValueTask<bool> TryGetAsync(string key, IBufferWriter<byte> destination, CancellationToken token = default);

    void Set(string key, ReadOnlySequence<byte> value, DistributedCacheEntryOptions options);
    ValueTask SetAsync(string key, ReadOnlySequence<byte> value, DistributedCacheEntryOptions options, CancellationToken token = default);
}

If the IDistributedCache service injected also implements this optional API, these buffer-based overloads will be used in preference to the byte[] API. We will absorb the work
to implement this API efficiently in the Redis implementation, and advice on others.

This feature has been prototyped using a FASTER backend cache implementation; it works very well:

| Method                | KeyLength | PayloadLength | Mean        | Error       | StdDev      | Gen0   | Gen1   | Allocated |
|---------------------- |---------- |-------------- |------------:|------------:|------------:|-------:|-------:|----------:|
| FASTER_Get            | 128       | 10240         |    576.0 ns |     9.79 ns |     5.83 ns | 0.6123 |      - |   10264 B |
| FASTER_Set            | 128       | 10240         |    882.0 ns |    23.99 ns |    22.44 ns | 0.6123 |      - |   10264 B |
| FASTER_GetAsync       | 128       | 10240         |    657.6 ns |    16.96 ns |    14.16 ns | 0.6189 |      - |   10360 B |
| FASTER_SetAsync       | 128       | 10240         |  1,094.7 ns |    55.15 ns |    51.58 ns | 0.6123 |      - |   10264 B |
|                       |           |               |             |             |             |        |        |           |
| FASTER_GetBuffer      | 128       | 10240         |    366.1 ns |     6.22 ns |     5.20 ns |      - |      - |         - |
| FASTER_SetBuffer      | 128       | 10240         |    495.4 ns |     7.11 ns |     2.54 ns |      - |      - |         - |
| FASTER_GetAsyncBuffer | 128       | 10240         |    387.9 ns |     7.60 ns |     1.97 ns | 0.0014 |      - |      24 B |
| FASTER_SetAsyncBuffer | 128       | 10240         |    649.9 ns |    12.70 ns |    11.88 ns |      - |      - |         - |

(the top half of the table uses IDistributedCache; the bottom half uses IBufferDistributedCache, and assumes the caller will utilize pooling etc, which hybrid cache: will)

---

Similarly, invalidation (at the key and tag level) will be implemented via an optional auxiliary service; however this API is still in design and is not discussed here.

It is anticipated that cache hit/miss/etc usage metrics will be reported via normal profiling APIs. By default this
will be global, but by enabling HybridCacheOptions.ReportTagMetrics, per-tag reporting will be enabled.

Serializer configuration

By default, the system will "just work", with defaults:

• string will be treated as UTF-8 bytes
• byte[] will be treated as raw bytes
• any other type will be serialized with System.Text.Json, as a reasonable in-box experience

However, it is important to be able to configure other serializers. Towards this, two serialization APIs are proposed:

namespace Microsoft.Extensions.Caching.Distributed;

public interface IHybridCacheSerializerFactory
{
    bool TryCreateSerializer<T>([NotNullWhen(true)] out IHybridCacheSerializer<T>? serializer);
}

public interface IHybridCacheSerializer<T>
{
    T Deserialize(ReadOnlySequence<byte> source);
    void Serialize(T value, IBufferWriter<byte> target);
}

With this API, serializers can be configured at both granular and coarse levels using the WithSerializer and WithSerializerFactory APIs at registration;
for any T, if a IHybridCacheSerializer<T> is known, it will be used as the serializer. Otherwise, the set of IHybridCacheSerializerFactory entries
will be enumerated; the last (i.e. most recently added/overridden) factory that returns true and provides a serializer: wins (this value may be cached),
with that serializer being used. This allows, for example, a protobuf-net serializer implementation to detect types marked [ProtoContract], or
the use of Newtonsoft.Json to replace System.Text.Json.

Binary payload implementation

The payload sent to IDistributedCache is not simply the raw buffer data; it also contains header metadata, to include:

• a version signifier (for safety with future data changes); in the case of 1:
• the key (used to guard against confusion attacks, i.e. non-equal keys that are equated by the cache implementation)
• the time (in absolute terms) that the entry was created
• the time (in absolute terms) that the entry expires
• the tags (if any) associated with the entry
• whether the payload is compressed
• payload length (for validation purposes)
• (followed by the payload)

All times are managed via TimeProvider. Upon fetching an entry from the cache, the expiration is compared using the current time;
expired entries are discarded as though they had not been received (this avoids a problem with time skew between in-process
and out-of-process stores, although out-of-process stores are still free to actively expire items).
Separately, the system maintains a cache of known tags and their last-invalidation-time (in absolute terms); if a cache entry has any tag that has a
last-invalidation-time after the creation time of the cache entry, then it is discarded as though it had not been received. This
effectively implements "tag" expiration without requiring that a backend is itself capable of categorized/"tagged" deletes (this feature is
not efficient or effective to implement in Redis, for example).

Additional implementation notes and assumptions

• Valid keys and tags are always non-null``, non-empty string` values
• Cache entries have maximum key and payload lengths that are enforced with reasonable (but configurable) defaults
• The header and payload are treated as an opaque BLOB for the purposes of IDistributedCache
  • Due the the header and possible compression, it should not be assumed that the value is inspectable in storage
  • The key/value will be inserted/updated/deleted as an atomic operation ("torn" values are not considered, although the payload length in the header will be verified
    with mismatches logged and the entry discarded)
  • There is no "type" metadata associated with a cache entry; the caller must know and specify (via the <T>) what they are requesting; if this is incorrect for
    the received data, an error may occur
• The backend store is treated as trusted, and it is assumed that any/all required authentication, encryption, etc requirements are controlled by the IDistributedCache
  registration, and the backend store is secure from tampering and exfiltration. Specifically: the data will not be additionally encrypted
• The backend store must be capable of servicing queries, inserts, updates and deletes
• Multi-node concurrency against the same backend store is assumed as a key scenario
• External systems might insert/update/delete against the same backend store; if data outside the expected form is encountered, it will be logged and discarded
• It is assumed that keys and tags cannot be aliased in the backend store; foo and FOO are separate; a-b and a%2Db are separate, etc; if the data retrieved
  has a non-matching key, it will be logged and discarded
• Keys and tags will be well-formed Unicode
• In the L1 in-process cache, the system will assume control of the string comparer and will apply safe logic; it will not be possible to specify a custom comparer
• It is assumed that keys and tags may contain untrusted user-provided tokens; backend implementations will use appropriate mechanisms to handle these values
• It is assumed that the backend storage will be lossless vs the stored data; payload length will be validated with mismatches logged and the entry discarded
• It is assumed that inserting / modifying / retrieving / deleting an entry in the backing store takes, at worst, amortized O((n ln n) + m)​ time,
  where n := number of chars in the key and m := number of bytes in the value
• the specific contents of the BLOB that gets stored via IDistributedCache is not explicitly documented (it is an implementation detail), and should be treated as an opaque BLOB
• the keys, tags and payloads are not encrypted and may be inspectable by an actor with unrestricted access to the backend store
• keys are not isolated by type; if a caller attempts to use <Foo> and <Bar> (different types) with the same cache key: the behaviour is undefined (it may or may not error, depending on the serializer and type compatibility); likewise, if a type is heavily refactored (i.e. in a way that impacts serializer compatibility) without changing the cache key: the behaviour is undefined

[jodydonetti]

Here we go: as usual I'll go back and forth between this and FusionCache to share my experience with it.

Overview

The primary API is a new abstract class, HybridCache, in a new Microsoft.Extensions.Caching.Distributed package:
[...]
This type acts as the primary API that users will interact with for caching using this feature...
[...]
Microsoft will provide a concrete implementation of HybridCache via dependency injection, but it is explicitly intended that the API can be implemented independently if desired.

LGTM, one question though: can we expect all the "extra state" needed will be in the concrete impl (eg: StandardHybridCache or whatever name will be used) and not in the abstract class (HybridCache)?
I'm asking because I'd like different 3rd party implementations, like the one I'll do for FusionCache, not to waste resources for things like cache stampede protection which is already in FusionCache itself.
Basically the abstract class would be an evolvable form of interface, makes sense?

Why "Hybrid Cache"?

This name seems to capture the multiple roles being fulfilled by the cache implementation. A number of otions have been considered, including "read thru cache", "advanced cache", "distributed cache 2"; this seems to work, though.

Perfect, no notes 👍

Why not IHybridCache?

1. the primary pass-thru API (discussed below) exists in a dual "stateful"/"stateless" mode, with it being possible to reliably and automatically implement one via the other; providing this at the definition level halves this aspect of the API surface for concrete implementations, providing a consistent experince

Sorry but I did not understand this part, can you elaborate more?

2. it is anticipated that additional future capabilities will be desired on this API; if we limit this as IHybridCache it is harder to extend than with an abstract base class that can implement features with default implementations that implementors can override as desired

Makes sense, interfaces (at least as of today) are less evolvable, and the only other approach would be interface-per-featture, where new features will be defined in new interfaces that will be added over time and that consumers may check for support (like the opt-in buffered distributed cache), but that is not always possible and in the long run may lead to a lot of different interfaces.

Watch out for people asking for interfaces nonetheless though, because testing etc (been there).

It is noted that in both cases, "default interface methods", also serve this function; if provide a mechanism to achieve this same goal with an IHybridCache approach. If we feel that "default interface methods" are now fully greenlit for this scenario, we could indeed use an IHybridCache approach.

Imho it's worth exploring the idea, not so sure about how good it would work in practice over time: anybody knows of an example of a project that successfully used default interface members to evolve it over time without breakings?
Maybe too soon, but asking just in case.

Registering and configuring HybridCache

[...]

In general looks good, but I don't see an overload of AddHybridCache(...) with a string name nor a string Name in the HybridCacheOptions: are you planning to not support multiple named caches in the first version?

Also I don't see methods to specify the distributed cache to use: will it be picked automatically from DI (if one is registered)?
If that is the case, a couple of things to look out for and that I had to deal with:

• how can someone use a distributed cache when none is registered via DI?
• if you'll force one to be registered via DI to be used automatically, that will in turn force other components to use it if they follow the same discovery approach
• if an IDistributedCache is registered via DI but someone doesn't want to use it as an L2, how can they do that?
• etc (ifthe automatic way is the only way, there will be problems I think)

In FusionCache I solved this by having different methods on the builder, like WithRegisteredDistributedCache() (automatic discovery from DI), WithDistributedCache(cache) (via direct instance), WithDistributedCache(factory) (via factory), etc.

If you are interested in some ideas, read here for more.

where IHybridCacheBuilder here functions purely as a wrapper (via .Services) to provide contextual API services to configure related services such as serialization

LGTM, but out of curiosity: why an interface here and not a class? Not that I necessarily would prefer one, but wouldn't the same rationale for IHybridCache work here, too (eg: concerns about future evolutions breaking existing impl) ? Or is it because you don't see other people implementing it? Just curious.

Also, any plan in supporting the fluent builder approach without DI?

I mean something like this;

// OPTION 1
var builder = new HybridCacheBuilder()
  .WithFoo()
  .WithBar();

// OPTION 2 (like WebApplication.CreateBuilder)
var builder = HybridCache.CreateBuilder()
  .WithFoo()
  .WithBar();

var cache = builder.Build();

Currently FusionCache does not support it for... reasons... but I'm thinking about adding it and other libs have already done it.
Again, just wondering.

The HybridCacheOptions provides additional global options for the cache, including ...

One note about naming: since there are both HybridCacheOptions (in short normally referred to as "options") and HybridCacheEntryOptions (in short normally referred to as "entry options"), I would suggest the default entry options prop to be named DefaultEntryOptions instead of DefaultOptions to avoid any potential confusion: in FusionCache I did it this way and it seemed to have worked intuitively well for users.
A minor thing, I know, just sharing my experience here.

... it is anticipated that simply having the L1 cache with stampede protection against the backend provides compelling value.

100% agree, there's a lot of value even just for that. After all that's the whole reason a library like LazyCache exists for example.

Also, it seems the new memory cache thing (discussed here) will not have cache stampede protection, so it would be even more useful.

Btw, about the L1: what will you use? The new memory cache mentioned above, the old MemoryCache, a custom thing built on a raw ConcurrentDictionary?
It seems it will be an hidden impl detail (I haven't see a method in the builder, so I guess that's the case).

Using HybridCache

[..]
It should be noted that all APIs are designed as async [...] async is strongly desirable.

Agree, even though some people will end up asking for the sync version because sometimes that's the only thing you can do (luckily these places are fewer and fewer every day), just warning you.

For the options, timeout is only described in relative terms

Yes, good call, no absolute terms 👍

[Flags]
public enum HybridCacheEntryFlags
{
    None = 0,
    DisableLocalCache = 1 << 0,
    DisableDistributedCache = 1 << 1,
    DisableCompression = 1 << 2,
}

Does this mean that compression will be a cross-cutting concern implemented by the HybridCache impl itself instead of each serializer? I mean a generic binary compression applied to the binary payload resulting from the byte[]/IBufferWriter<byte> processing, instead of a specific one implemented by each serializer with the knowledge of each serialization algorithm.
I would not have expected that, will see how it goes.

In many cases, GetOrCreateAsync is the only API needed, but additionally, HybridCache has auxiliary APIs:

public abstract ValueTask<HybridCacheEntry<T>?> GetAsync<T>(string key, HybridCacheEntryOptions? options = null, CancellationToken cancellationToken = default);

Naming is hard. I would suggest a more specific GetEntryAsync instead of GetAsync, since normally the value is what is being returned: for example the GetOrCreateAsync method returnes a value, so for GetAsync will expect the same.
If in the future you'll have new methods that will work explicitly on an entry, like for example a set method that accepts an entry directly, it would be called SetEntryAsync so that all will be uniform.
Just my 2 cents.

public abstract ValueTask RemoveKeyAsync(string key, CancellationToken cancellationToken = default);
public virtual ValueTask RemoveKeysAsync(ReadOnlyMemory<string> keys, CancellationToken cancellationToken = default) // implemented via RemoveKeyAsync
public virtual ValueTask RemoveTagAsync(string tag, CancellationToken cancellationToken = default) // implemented via RemoveTags
public abstract ValueTask RemoveTagsAsync(ReadOnlyMemory<string> tags, CancellationToken cancellationToken = default)

Naming is hard. I would suggest the use of "By" here: RemoveByKeyAsync, RemoveByKeysAsync, RemoveByTagsAsync, etc.

public sealed class HybridCacheEntry<T>
{
    public T Value { get; set; } = default!;
    public ReadOnlyMemory<string> Tags { get; set; }
    public DateTime Expiry { get; set; } // absolute time of expiry
    public DateTime LocalExpiry { get; set; } // absolute time of L1 expiry
}

Shouldn't LocalExpiry be DateTime? (nullable) so that if not set it will take the normal Expiry? Otherwise users will have to specify both each time, and/or have confusion about what would happen if not set.

Backend services

To provide the enhanced capabilities, some new additional services are required
[...]
the system functions without demanding it and all pre-existing IDistributedCache implementations will continue to work. The system will type-test for the new capability
If the IDistributedCache service injected also implements this optional API, these buffer-based overloads will be used in preference to the byte[] API.

Good old Progressive Enhancement, this is the way.

It is anticipated that cache hit/miss/etc usage metrics will be reported via normal profiling APIs. By default this will be global, but by enabling HybridCacheOptions.ReportTagMetrics, per-tag reporting will be enabled.

One thing to note is that since the tags values will be a lot, that means high cardinality, which in the observability world (metrics in particular) can make systems easily explode.
If I got it correctly this option will be opt-in (meaning false by default) so the default dev experience should be fine, but the reason highlighted above was about "sensitive data" so I wanted to add the high cardinality thing to the pile.

Serializer configuration

By default, the system will "just work", with defaults:

• string will be treated as UTF-8 bytes
• byte[] will be treated as raw bytes
• any other type will be serialized with System.Text.Json, as a reasonable in-box experience

However, it is important to be able to configure other serializers. Towards this, two serialization APIs are proposed:

Nice catch about being able to directly use byte[] values without having to serialize them!

One note though: does this design mean that if I provide my own serializer it will be used only for non-string/non-byte[] types or for any type?

With this API, serializers can be configured at both granular and coarse levels using the WithSerializer and WithSerializerFactory APIs at registration; for any T, if a IHybridCacheSerializer<T> is known, it will be used as the serializer. Otherwise, the set of IHybridCacheSerializerFactory entries will be enumerated; the last (i.e. most recently added/overridden) factory that returns true and provides a serializer wins

Intuitive design 👍

Binary payload implementation

The payload sent to IDistributedCache is not simply the raw buffer data; it also contains header metadata, to include:

• a version signifier (for safety with future data changes); in the case of 1:

Eh, this is delicate.

I don't know if you have already thought about this, so I'll share my own exp on this.

With this approach, since any reader can check the version signifier before proceeding, there will be no problems of corrupted entries: this is true.

But the problem is that when upgrading a live system in the future (say from v1 to v2) composed of multiple apps/services, users will have different v1 clients writing v1 payloads and v2 clients writing v2 payloads untile the entire system will be updated, and that realistically cannot be done at the same time.
Now, in theory v2 clients will be able to read both v1 and v2 payloads, although new data from v2 will be missing from the v1 payloads, forcing the new v2 code to be able to handle all the missing data cases, and so becoming more complex (ask me how I know).
Also the same would not be true the other way around, meaning of course v1 clients will not be able to read v2 payloads.
So the situation will end up being that updated v2 clients will keep writing new payloads that will not be readable by v1 clients, and v1 clients will keep writing v1 payloads that will (potentially?) be readable by all, but without some important data missing from the v1 metadata.

To share what has been my exp with FusionCache, what I did was use the version signifier as an additional prefix/suffix in the cache key used in the distributed cache: in this way v1 clients and v2 clients will write different entries without disturbing each others, and as all the clients will be updated the cache entries in v1 format will be less and less, and then disappear after all the system (read: all the clients) will be updated to v2.

More space consumed in the distributed cache? Yes, but only temporarily, and only IF the entire system is not updated at the same time (this will depend on each user's scenario).

Of course I'm not necessarily saying this is a better design, but just exposing a different one for you to reason about (if you haven't already!).

• the key (used to guard against confusion attacks, i.e. non-equal keys that are equated by the cache implementation)

Interesting! Haven't thought of this before.

Separately, the system maintains a cache of known tags and their last-invalidation-time (in absolute terms); if a cache entry has any tag that has a last-invalidation-time after the creation time of the cache entry, then it is discarded as though it had not been received. This effectively implements "tag" expiration without requiring that a backend is itself capable of categorized/"tagged" deletes (this feature is not efficient or effective to implement in Redis, for example).

So in the end you decided to go on with the invalidation by tag? How did you solve the problems highlighted previously.

I'll re-quote here for brevity:

Consider this chain of events:

1. set "foo" in the cache with 1 min duration (to L1+L2)
2. after 1 sec get -> value is there (from L1)
3. after 1 sec clear -> "clear timestamp" added (locally)
4. after 1 sec get -> nothing is there (it's in L1, but logical check with clear timestamp)
5. app restarts -> L1 is now empty, L2 is still there
6. get "foo" -> L1 is not there, L2 is there, copy to L1, no clear timestamp -> value magically reappears

The example was for an hypothetical Clear() method, but the rationale is the same for any "multi remove/invalidate by tags".

Basically, the gist of it was that it is basically impossible to do invalidation by tag(s) consistently by not really doing it but instead trying to simulate it by relying on a sort of "in-memory barrier" that will do additional checks before returning a value. That was because if the "invalidation dates" or similar would be stored in memory, and they would be wiped at the next restart of the app.

Above you said "separately, the system maintains a cache of known tags and their last-invalidation-time" but still in memory? If so, the problems highlighted above still stands.

Unless of course you came up with a different technique, in which case I'm really interested to discover what will be 😬

Additional implementation notes and assumptions

• Cache entries have maximum key and payload lengths that are enforced with reasonable (but configurable) defaults

Interesting! It's something I thought about for some time but haven't ened up doing yet, so I was wodnering: what are you planning to do when the limits are crossed? Log it? Throw an exception? Skip/ignore?

• It is assumed that keys and tags cannot be aliased in the backend store; foo and FOO are separate; a-b and a%2Db are separate, etc; if the data retrieved has a non-matching key, it will be logged and discarded

Interesting, again. Never thought about this.

[andreaskromann]

6. get "foo" -> L1 is not there, L2 is there, copy to L1, no clear timestamp -> value magically reappears

Since the timestamp is stored in the cache itself, it will also be fetched from L2 and inserted to L1 on the first get after the app has been restarted.

[jodydonetti]

Since the timestamp is stored in the cache itself, it will also be fetched from L2 and inserted to L1 on the first get after the app has been restarted.

@andreaskromann I'm not following here, are you talking about each invalidation's timestamp? Where will it be stored? In a single cache entry for all the invalidations by tag(s)? One cache entry per tag?

[mgravell]

on tag expiration; I was deliberately deferring on that, but:

• when available, that will use a new as-yet-undefined auxiliary API on the backend that is tag expiration specific
• in the absence of that, we'll use an arbitrary key or tag-specific keys in the backend to spoof storage (but not active expiration) of tag expiry metadata - this is an implementation detail, though - for example __MSFT__DC_Tag:{tagname} = 1711384791 with a much longer duration than regular tag entries

So yes, tag expiration will outlive process restart

[andreaskromann]

@jodydonetti I guess Marc explained it above, but yes one entry per tag.

Regarding the API proposal, I was positively surprised so much of what we discussed made it into the proposal. It looks very promising. The auxiliary API for invalidations is still undefined, so it will be interesting to see. Another thing I noticed was that the concept of serving stale values with a background refresh (stale-while-revalidate) didn't make the cut.

[mgravell]

Stale with background refresh is highly desirable, and I'm confident it will get added later. One huge problem is the safety of the callback (vs escaping the execution path of the calling code), meaning we'll need this to be opt-in and contextual per usage. That's another reason for the [Flags] options on the per-call options. If rather not commit to all of that for the first release, though!

[jodydonetti]

on tag expiration; I was deliberately deferring on that, but:

• when available, that will use a new as-yet-undefined auxiliary API on the backend that is tag expiration specific

I'll eagerly await to see which design will make it work reasonably, can't wait 😬

• in the absence of that, we'll use an arbitrary key or tag-specific keys in the backend to spoof storage (but not active expiration) of tag expiry metadata - this is an implementation detail, though - for example __MSFT__DC_Tag:{tagname} = 1711384791 with a much longer duration than regular tag entries

Mmmh... that's why I asked: it's not the first time I played with such approach (and others) but they never really worked, at least not in a reasonable way.

Tags are stored inside each entry, sure, but when a user asks for an entry it does it by cache key, and that is the only thing we know upfront.

So what happens is, with a concrete example for a get:

1. get "top_products" (let's say for the top 5 products)
2. load from L1 -> nothing there
3. load from L2 -> found, copy locally
4. check the entry tags: "product/1", "product/2", "product/3", product/4" "product/5"
5. for each tag (watch out: SELECT N + 1):
6. -> get "__MSFT_DC_Tag:{tag}"
7. -> load from L1
8. -> (maybe) load from L2
9. -> (maybe) copy to L1
10. check all tags' invalidation entries, if any, against the main entry's creation timestamp and act accordingly
11. etc

On top of this, should each "invalidation tag entry" go through the same cycle as any normal key? Eh, that's not necessarily so immediate to answer, and a fun one.

Anyway, at this point I think it's better if I just stop here and wait for the design/api surface to get out, so I can reason on something concrete and don't waste your time in speculations or spoilers.

[dotnet-policy-service]

Thank you for submitting this for API review. This will be reviewed by @dotnet/aspnet-api-review at the next meeting of the ASP.NET Core API Review group. Please ensure you take a look at the API review process documentation and ensure that:

• The PR contains changes to the reference-assembly that describe the API change. Or, you have included a snippet of reference-assembly-style code that illustrates the API change.
• The PR describes the impact to users, both positive (useful new APIs) and negative (breaking changes).
• Someone is assigned to "champion" this change in the meeting, and they understand the impact and design of the change.

[amcasey]

• Goal: cache arbitrary binary blobs
  • App data and server stuff
• IDistributedCache is the backend of HybridCache
  • This is important because there are already 3P impls of IDistributedCache
• Why is HybridCache extensible?
  • We want to leave room for community projects like FusionCache
  • It would be nice to have a proof-of-concept 3P impl of HybridCache before we finalize the API (i.e. ship)
• We want to avoid confusion arising from the package name, which seems potentially overlapping with the distributed caching package(s)
  • Should contain "hybrid"
  • Microsoft.Extensions.Caching.Hybrid
    • The other Microsoft.Extensions.Caching.* packages don't have a "Cache" suffix
• Package probably goes in shared framework, rather than on nuget, but not finalized
• We want it to target net standard, so it probably shouldn't depend on DIM for extensibility
• What re-entrancy does GetOrCreateAsync have?
  • We don't want re-entrancy - can we guard against it?
  • If there's a race, only one lambda will be invoked
  • In this context "re-entrancy" is referring to how many times the lambda is invoked
• Make the non-stateful overload of GetOrCreateAsync non-virtual so we don't have to reason about them getting out of sync
• Intentionally not providing sync API because why are you caching if it's fast enough to do inline?
  • You can use IMemoryCache for sync caching
• IHybridCacheSerializer is a general interface - it would be nice if we could rely on a framework interface, but there doesn't seem to be one
• You can have multiple serializers and type-specific ones will be preferred
  • Order is a tie-breaker: last wins
• Nit: Move serializer extension methods into their own type (separate from AddHybridCache)
• If you want one serializer for everything, you pass object
• WithSerializer - is T exact or does it allow subtypes?
  • We expect exact
• Should IHybridCacheSerializer be ValueTask-returning if it can wrap arbitrary serializers?
  • Philosophically, we don't like async serialization
    • This may rule out some existing serializers
• Rename WithSerializer to AddSerializer

[amcasey]

• Why not put the whole thing in dotnet/runtime?
  • Nothing in here depends on aspnetcore
  • It would be nice to have a shared impl of stampede protection
  • We can probably ship it sooner in aspnetcore
    • Let's put it in aspnetcore for now and continue the discussion after it ships in a preview
• Nuget package name Microsoft.Extensions.Caching.Hybrid
  • Might make a good namespace name too
• Some packaging questions remain to be decided
  • Combine with existing?
  • Include in shared framework?
  • aspnetcore or runtime?
  • Maybe put the base time in the existing abstractions package?
    • Seems important to separate base type from impl to avoid pulling in impl
• Is "Expiry" our standard terminology? Check for consistency
  • Does it need a prefix to make the difference from LocalCacheExpiry clearer?
  • Pattern might be "AbsoluteExpirationRelativeToNow"
• These options get passed to all impls (e.g. FusionCache) and seem generic enough to be meaningful in all of them
• These options are immutable but we frequently have settable properties - it's worth checking for consistency and making sure nothing breaks without being able to set these properties
  • Check ConfigurationBinder, in particular
  • Probably want options properties to be nullable
• Expected pattern: users will have a static readonly Options object - not make a new one each time
  • Would still work with short-lived objects
• HybridCacheEntryFlags seem best-effort for impls - ignoring them hurts perf but not behavior
• Disabling compression beats enabling compression
  • Probably change AllowCompression to DisableCompression to make that clearer
• HybridCacheEntryOptions
  • Make properties init
    • netstandard example
  • Change "Expiry" to "Expiration" globally
• Rename HybridCacheOptions.DefaultOptions to DefaultEntryOptions
• TryGet could be called TryRead, TryWrite, or TryCopy
  • Get is consistent with output caching
  • Get is consistent with the underlying IDistributedCache method it parallels
• Note that a zero-length payload is potentially valid, so we need a boolean return from TryGet
• The non-async IBufferDistributedCache members are blocking
• Why are the tags passed to GetOrCreateAsync a ReadOnlyMemory (rather than an IEnumerable, e.g.)?
  • We don't want any laziness for perf reasons (so no IEnumerable)
  • Note that ROM is a collection of strings
  • What about IReadOnlyCollection/List?
  • What about StringValues?
  • What about a more intelligible/familiar overload?
  • We want to make sure we don't lead callers down the path of allocating unnecessarily
  • The tag collection is unordered
  • Do we need defensive copying of tags?
    • Yes (and another argument against ROM)
  • Fixed via ICollection<string>?
• HybridCacheEntry
  • Can we drop the type entirely?
    • Can bring it back in a later preview if we miss it
  • Expiry => Expiration
  • Entries are not stored by the cache
• Might want usage of global IDistributedCache or IMemoryCache to be configurable
  • Might come up again in the context of keyed DI

[amcasey]

Next step: address the feedback and put a clean copy of the API in a comment and/or PR. We can finalize over email/GH.

[mgravell]

merged: #55084

[mgravell]

actually, I need to check the normal "who closes, when" - reopening

[mgravell]

@jodydonetti @joegoldman2 you both raised the By naming thing; I've logged that for separate API review - we're fine to making breaking changes between preview4 and release - that's the point of previews ;p #55332

[amcasey]

API Approved (offline)!

[jodydonetti]

@mgravell and I know that you almost answered my notes some time ago but then the mobile app discarded your answer, and I see that this has been already closed, but you think you'll be able to find some time to type them again?
I'd be really curious to know your thoughts about them to move the conversation forward to avoid issues down the road.
Thanks!

ps: of course the parts related to tag invalidation are already answered in the other issue, which I'll answer to in a moment.

[VenkateshSrini]

I need this library to be working with .NET 8 also. Also Will I be able to specify the compression algorithm also ?