HybridCache: tags parsing issue breaks remove by tags behavior

[Oleg52]

Description

I see Cache backend data rejected: ParseFault. warning when calling GetOrCreateAsync without providing tags. I'm passing a key that i had previously added with 2 or more tags. I'm using Redis as a backend cache.

Reproduction Steps

using Microsoft.Extensions.Caching.Hybrid;

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddStackExchangeRedisCache(options => { options.Configuration = "127.0.0.1:6379"; });
builder.Services.AddHybridCache();

var app = builder.Build();

app.MapGet("/tags", async (HybridCache hybridCache) =>
{
    string[] tags = ["tag1", "tag2"];
    return await hybridCache.GetOrCreateAsync("key", _ => ValueTask.FromResult("data"), tags: tags);
});

app.MapGet("/no-tags", async (HybridCache hybridCache) =>
{
    return await hybridCache.GetOrCreateAsync("key", _ => ValueTask.FromResult("data"));
});

app.Run();

1. Run app and send GET request on /tags.
2. Re-run app and send GET request on /no-tags.

Expected behavior

No warning is shown

Actual behavior

I get this warning:

      Cache backend data rejected: ParseFault.
      System.ArgumentException: Destination array was not long enough. Check the destination index, length, and the array's lower bounds. (Parameter 'destinationArray')
         at System.Array.CopyImpl(Array sourceArray, Int32 sourceIndex, Array destinationArray, Int32 destinationIndex, Int32 length, Boolean reliable)
         at System.Array.Copy(Array sourceArray, Int32 sourceIndex, Array destinationArray, Int32 destinationIndex, Int32 length)
         at System.Array.CopyTo(Array array, Int32 index)
         at Microsoft.Extensions.Caching.Hybrid.Internal.HybridCachePayload.TryParse(ArraySegment`1 source, String key, TagSet knownTags, DefaultHybridCache cache, ArraySegment`1& payload, TimeSpan& remainingTime, PayloadFlags& flags, UInt16& entropy, TagSet& pendingTags, Exception& fault)

Regression?

No response

Known Workarounds

No response

Configuration

.NET SDK:
 Version:           9.0.110
 Commit:            cecf5898ba
 Workload version:  9.0.100-manifests.d96a6bf8
 MSBuild version:   17.12.44+db5c766a7

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.26100
 OS Platform: Windows
 RID:         win-x64
 Base Path:   C:\Program Files\dotnet\sdk\9.0.110\

.NET workloads installed:
There are no installed workloads to display.
Configured to use loose manifests when installing new manifests.

Host:
  Version:      9.0.9
  Architecture: x64
  Commit:       893c2ebbd4

.NET SDKs installed:
  9.0.101 [C:\Program Files\dotnet\sdk]
  9.0.110 [C:\Program Files\dotnet\sdk]

.NET runtimes installed:
  Microsoft.AspNetCore.App 9.0.0 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 9.0.9 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 9.0.0 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 9.0.9 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.WindowsDesktop.App 9.0.0 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 9.0.9 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]

Packages:

Microsoft.Extensions.Caching.Hybrid                        9.9.0
Microsoft.Extensions.Caching.StackExchangeRedis            9.0.9

Other information

The problem seems to occur here

extensions/src/Libraries/Microsoft.Extensions.Caching.Hybrid/Internal/HybridCachePayload.cs

Lines 278 to 281 in c4e57fb

	string[] newBuffer = ArrayPool<string>.Shared.Rent(Math.Max(4, pendingTagsCount * 2));
	pendingTagBuffer.CopyTo(newBuffer, 0);
	ArrayPool<string>.Shared.Return(pendingTagBuffer);
	pendingTagBuffer = newBuffer;

pendingTagBuffer size is increased, and later in default case

extensions/src/Libraries/Microsoft.Extensions.Caching.Hybrid/Internal/HybridCachePayload.cs

Lines 310 to 314 in c4e57fb

	default:
	string[] final = new string[pendingTagsCount];
	pendingTagBuffer.CopyTo(final, 0);
	pendingTags = new(final);
	break;

CopyTo fails because final array length is less than pendingTagBuffer

[Oleg52]

I also noticed that after removing by tags GetOrCreateAsync still returns old data.

Slightly modified reproduction code(reproduction steps are not changed):

using Microsoft.Extensions.Caching.Hybrid;

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddStackExchangeRedisCache(options => { options.Configuration = "127.0.0.1:6379"; });
builder.Services.AddHybridCache();

var app = builder.Build();

app.MapGet("/tags", async (HybridCache hybridCache) =>
{
    string[] tags = ["tag1", "tag2"];
    return await hybridCache.GetOrCreateAsync("key", _ => ValueTask.FromResult("data"), tags: tags);
});

app.MapGet("/no-tags", async (HybridCache hybridCache) =>
{
    _ = await hybridCache.GetOrCreateAsync("key", _ => ValueTask.FromResult("data"));
    await hybridCache.RemoveByTagAsync("tag1");
    return await hybridCache.GetOrCreateAsync("key", _ => ValueTask.FromResult("new data"));
});

app.Run();

If i send GET request on /tags and then on /no-tags without re-running app, /no-tags returns new data.
But if i re-run app after sending GET request on /tags, /no-tags returns data, so RemoveByTagAsync does not work as expected.

[torarnegustad]

@Oleg52 Hi, we have the same issue and would very much like this to be fixed. In our use case it actually gives us the wrong data on the first request, which in many ways eliminates the benefits of hybrid cache since each replica in our application ends up having to retrieve the same information..

Could you recreate the issue in https://github.com/dotnet/runtime since this is the new code base since July 2024 (NET 9)?

[Oleg52]

@torarnegustad Hi, thanks for notifying me, i recreated it here: #122143

[jodydonetti]

Why recreate it in the dotnet/runtime repo?

Isn't HybricCache part of the dotnet/extensions repo?

[jodydonetti]

Hi @torarnegustad

@Oleg52 Hi, we have the same issue and would very much like this to be fixed. In our use case it actually gives us the wrong data on the first request, which in many ways eliminates the benefits of hybrid cache since each replica in our application ends up having to retrieve the same information..

As a (temporary?) fix you may look into using the FusionCache based implementation of HybridCache, see here.

It also has a backplane to keep the cache coherent even in a multi-node environment, and that also works with the HybridCache impl.

It's a 100% compatible drop-in impl replacement, no code changes required except for 1 line during setup.

Hope this helps.

NB: FusionCache creator here, just to be clear.