.Net: Collection typing with custom schemas

[markwallace-microsoft]

In my scenarios, schemas are always built programmatically based on configuration or user input. This means collections cannot be strongly typed to a specific class.
Currently, I use:

→ var collection = vectorStore.GetCollection<ulong, object>("name", definition);

Setting TRecord to object seems to work but feels hacky. A class extension to abstract/hide this would improve usability, something like this:

→ var collection = vectorStore.GetCollection("name", definition);

[roji]

@dluc how does this work? What type is actually returned when you call the search API? At the end fo the day, MEVD needs to instantiate a .NET type when reading results back from the database, what type would it be?

As @westey-m wrote offline, we have VectorStoreGenericDataModel for this purpose; I believe it should be possible to simplify that and simply use Dictionary<string, object> as the dynamic CLR type (rather than requiring the special VectorStoreGenericDataModel), but some concrete type is required.

[westey-m]

Also see our documentation page here for the generic data model:
https://learn.microsoft.com/en-us/semantic-kernel/concepts/vector-store-connectors/generic-data-model

[dluc]

I'll check out VectorStoreGenericDataModel 👍

I'd still like having a solution where the consumer app code is the same regardless of the engine selected in the configuration. Currently, the need for different TKey, TRecord for each engine is leaking in the app, requiring switch statements (something I would not expect from an abstraction layer).

Btw, I'm writing an abstraction on top of the abstraction, so that I can switch from Qdrant to Azure AI Search to Postgres, etc, so it is definitely possible to do it. The question is whether we want users (like me) having to develop another abstraction layer or whether the library takes care of it.

[dluc]

VectorStoreGenericDataModel working fine, thanks for pointing there.

The remaining challenge is code complexity, e.g. ~N distinct cases to code in order to support N storage engines, with runtime type checking.

Example:

public async Task<IResult> CreateCollectionAsync(IVectorStore vectorStore, CreateCollectionRequest req, CancellationToken cancellationToken)
{
    // Record definition
    var properties = new List<VectorStoreRecordProperty>();
    foreach (var f in req.Fields)
    {
        switch (f.Type)
        {
            case FieldTypes.PrimaryKey:
                // More conditions needed here to support other storage engines (ulong keys, long, int, composite keys, etc)

                properties.Add(vectorStore is QdrantVectorStore     // <--------
                    ? new VectorStoreRecordKeyProperty(f.Name, typeof(Guid))
                    : new VectorStoreRecordKeyProperty(f.Name, typeof(string)));

                break;
            case FieldTypes.Vector:
                properties.Add(new VectorStoreRecordVectorProperty(f.Name, typeof(ReadOnlyMemory<float>))
                {
                    Dimensions = req.Dimensions,
                    DistanceFunction = req.DistanceFunction,
                    IndexKind = req.IndexKind
                });
                break;
            // ...
        }
    }
    var recordDefinition = new VectorStoreRecordDefinition { Properties = properties };

    // Create collection
    try
    {
        switch (vectorStore)                                // <--------
        {
            case QdrantVectorStore:                         // <--------
            {
                var collection = vectorStore.GetCollection<Guid, object>(req.CollectionName, recordDefinition);
                await collection.CreateCollectionAsync(cancellationToken).ConfigureAwait(false);
                break;
            }
            case AzureAISearchVectorStore:                  // <--------
            {
                var collection = vectorStore.GetCollection<string, object>(req.CollectionName, recordDefinition);
                await collection.CreateCollectionAsync(cancellationToken).ConfigureAwait(false);
                break;
            }
            case InMemoryVectorStore:                       // <--------
            {
                var collection = vectorStore.GetCollection<string, VectorStoreGenericDataModel<string>>(req.CollectionName, recordDefinition);
                await collection.CreateCollectionAsync(cancellationToken).ConfigureAwait(false);
                break;
            }
        }
    }
    catch (VectorStoreOperationException e)
    {
        // Qdrant + Azure AI Search                         // <--------
        if (e.InnerException?.Message.Contains("already exists", StringComparison.OrdinalIgnoreCase) == true)
        {
            this._log.LogError("Unable to create {StorageType} collection, it already exists", req.StorageType);
            return ...
        }

        // InMemory? (not throwing yet)
                                                            // <--------

        // More string matching for other storage engines
                                                            // <--------

        this._log.LogError(e, "Unable to create {StorageType} collection, unexpected error: {Message}", req.StorageType, e.Message);
        return ...
    }
}

[roji]

Currently, the need for different TKey, TRecord for each engine is leaking in the app, requiring switch statements (something I would not expect from an abstraction layer).

TRecord does not, in itself, leak anything from the database/connector the app; it's a user type to map to the database, and at least if you restrict yourself to a common denominator of property types, the same type will work across all connectors.

For TKey, I think the discussion is ongoing in #10872, let's continue it there.