.Net: minRelevancyScore is "backwards" for Weaviate

[matthewbolanos]

For Weaviate, this does not return any results...

var matches = await memoryStore.GetNearestMatchAsync(
    collectionName: "scene",
    embedding: await textEmbeddingGenerationService.GenerateEmbeddingAsync(completePrompt),
    minRelevanceScore: 0
);

But this one does

var matches = await memoryStore.GetNearestMatchAsync(
    collectionName: "scene",
    embedding: await textEmbeddingGenerationService.GenerateEmbeddingAsync(completePrompt),
    minRelevanceScore: =1
);

[dmytrostruk]

@matthewbolanos From Weaviate point of view, it is correct, because default distance metric is cosine, distance value has range 0 <= d <= 2, where 0 is identical vector and 2 is opposing vector:
https://weaviate.io/developers/weaviate/config-refs/distances#available-distance-metrics

On the other hand, when you use IMemoryStore abstraction, you don't know which connector is used, which sounds like on abstraction level, the score should be aligned across all connectors. For example, for Weaviate, we will need to convert 0 <= d <= 2 to 1 >= d >= 0 and vice versa. cc: @westey-m

And this is just for cosine metric, but I think users will want to support other metrics as well, for example l2-squared with range 0 <= d < ∞.

It's not a problem to fix that for Weaviate at this point, but I'm a little bit worried that it may change the logic in users' applications now, if they already found a good threshold for their scenarios.

I'm wondering if new memory abstraction and implementation for connectors will be a good timing for such changes?

[westey-m]

Agreed, we should definitely try to address this as part of the new memory connectors work. Normalization is a potential option, but it will be difficult to normalize across some of the disparate ranges, e.g. turning -∞ < d < ∞ into 0 to 1 or vice versa.
Alternatively we should consider making it clear what distance metric is in use and what the appropriate ranges are when specifying a required distance / similarity range. The interface will therefore be the same across different databases for the same distance metric, but distance metrics will be treated differently to each other.

CC @markwallace-microsoft