.Net: Redis Connector - Unable to successfully query results

[aventus13]

I'm trying to get the .Net Redis Connector to work to no avail so far.

First, using the same approach that I've been using for in-memory as well as Azure AI Search connectors, I've been saving items to the collection in the following way:

var collectionId = // some uuid
// IVectorStore
var collection = _vectorStore.GetCollection<string, DocumentEmbeddingRecord>(collectionId);

// Bulk store chunks as vectors
for (var i = 0; i < chunks.Count; i++)
{
    var record = new DocumentEmbeddingRecord
    {
        Key = Guid.NewGuid().ToString(),
        Text = chunks[i],
        Description = "PDF content chunk",
        ExternalSourceName = document.Filename,
        Embedding = vectors[i]
    };

    vectorRecords.Add(record);
}

await collection.UpsertAsync(vectorRecords);

Here's my DocumentEmbeddingRecord:

public class DocumentEmbeddingRecord
{
    [VectorStoreKey]
    public required string Key { get; init; }
    
    [VectorStoreData]
    public required string Text { get; init; }
    
    [VectorStoreData]
    public required string Description { get; init; }
    
    [VectorStoreData]
    public required string ExternalSourceName { get; init; }
    
    [VectorStoreVector(Dimensions: 1536, DistanceFunction = DistanceFunction.CosineSimilarity)]
    public required ReadOnlyMemory<float> Embedding { get; init; }
}

However, when doing so, I was getting the following error from Redis:
StackExchange.Redis.RedisServerException: CROSSSLOT Keys in request don't hash to the same slot (context='', command='json.mset', original-slot='14999', wrong-slot='10973'...

After some digging, I updated the Key to force the item to the same slot:

Key = $"{{{collectionId}}}:{Guid.NewGuid().ToString()}"

This allows to successfully save data to Redis. However, when querying, no results can be found:

var collection = _vectorStore.GetCollection<string, DocumentEmbeddingRecord>(collectionId);
var searchResults = collection.SearchAsync(queryEmbedding.Vector, top: 3);
var results = new List<VectorSearchResult<DocumentEmbeddingRecord>>();

await foreach (var result in searchResults)
{
    results.Add(result);
}

// results.Count == 0

What am I doing wrong? To reiterate, the above approach works for in-memory and AI Search connectors. The generated collection in Redis is structured like this:

Example of a stored record:

I also tried generating data as hash set. While such generated collections can be found, the query still fails, and the embedding vector doesn't look right: This was caused by invalid dimensions used as Redis connector seems to be strict in that regard. However, the issue with collection names still remains using JSON data.

[westey-m]

Hi @aventus13,

While I am not entirely sure, I think the issue may be related to index prefixes. Redis doesn't have the concept of a collection similar to most other vector dbs, but it does have indexes. Documents are only indexed by an index if the start of the document key matches a prefix setting on the index.

The SK Redis implementation automatically uses a prefix of $"{this.Name}:" where name is the collection name. In your example therefore the value of the collection name would be collectionId.
E.g. 01e0afed-8453-468e-abeb-6d75ca8e09f8:
Note the colon at the end of the Guid.

It's therefore important to double check that the prefix configured for the index that you created, is matching the records in the database.

E.g. if the record key is prefixed with {01e0afed-8453-468e-abeb-6d75ca8e09f8}:, this will mean that the record is not indexed by the index with the prefix of 01e0afed-8453-468e-abeb-6d75ca8e09f8: since the one is surrounded by curly braces and the other is not.

The SK Redis implementation will prefix all record keys on Upsert or Get with the index prefix, unless this is switched off via the PrefixCollectionNameToKeyNames option passed to the collection constructor.

var collection = new RedisJsonCollection<string, DocumentEmbeddingRecord>(
    ConnectionMultiplexer.Connect("localhost:6379").GetDatabase(),
    collectionId,
    new() { PrefixCollectionNameToKeyNames = false });

FYI, it is possible to check the prefix configured for an existing index in the redis-stack browser.

[westey-m]

Closing due to lack of information to troubleshoot further.