search(q, index="dense") returns values with high scores even if there should not be a match

[ddofborg]

I am using product titles as my corpus. I embed them using:

    embeddings = txtai.Embeddings(
        defaults=False,
        normalize=True,
        indexes={
            "keyword": {
                "keyword": True
            },
            "dense": {
                "path": "NetherlandsForensicInstitute/robbert-2022-dutch-sentence-transformers"
            }
        }
    )

Then index as normal.

When I do a search for words which are 100% unrelated to the corpus. The dense index almost always returns products with scores between 0.10 and 0.35. Sometimes 0.50. But a fully correct and matching products would get 0.60.

Are where more finetune methods to dive deeper and find if the match is making sense?

I also use the BM25 matcher and weight the results, but it's not ideal, as the dense index gives a very high score to bad results.

[davidmezzetti]

I'm not familiar with this model. But one easy way to test the similarity scores is below.

from txtai import Embeddings

embeddings = Embeddings(path="NetherlandsForensicInstitute/robbert-2022-dutch-sentence-transformers")
embeddings.similarity("text 1", ["text2"])

I would use that method to test the underlying vector model.

[ddofborg]

Thanks for the idea to look at this a little deeper. I tested a few models:

from txtai import Embeddings
models = [
    'sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2',
    'intfloat/multilingual-e5-large-instruct',
    'NetherlandsForensicInstitute/robbert-2022-dutch-sentence-transformers',
]
for c, model in enumerate(models):
    embeddings = Embeddings(path=model)
    # meaning in English: 'wheeled dumper', [ 'drive belt', 'windshield wiper', ''dumper', 'wheeled dumper']
    s = embeddings.similarity("wheeled dumper", ["aandrijfriem", "ruitenwisser", "dumper", "wheeled dumper"])
    s = sorted(s)
    print(f"{c}: {model}:")
    print(f'   {s}')

This does not look great to me. The first two first should have a lower similarity, I would say. And the third word should be much closer than the first two.

0: sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2:
   [(0, 0.4416919946670532), (1, 0.34208500385284424), (2, 0.53569495677948), (3, 0.9999999403953552)]
1: intfloat/multilingual-e5-large-instruct:
   [(0, 0.8510584831237793), (1, 0.846233069896698), (2, 0.9564969539642334), (3, 0.9999998807907104)]
2: NetherlandsForensicInstitute/robbert-2022-dutch-sentence-transformers:
   [(0, 0.27119553089141846), (1, 0.37502622604370117), (2, 0.4329324960708618), (3, 0.9999998211860657)]

What would be the best way to analyse this?

[davidmezzetti]

This is just the way these models are. Most are not designed for word similarity but sentence similarity. I would continue evaluating different models on real-word sentences instead of words to see the one that works the best.

[davidmezzetti]

I believe this is no longer an issue as mentioned in #910