Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Docs: Fix and supplement answer relevancy description #705

Conversation

jokokojote
Copy link
Contributor

The docs for the answer relevancy metric are wrong in that it is claimed that the score ranges between 0 and 1. As it is written in the calculation section an averaged cosine similarity is used to calculate the metric. The cosine similarity can range between -1 and 1. Of course, most of the time the original question and generated questions will be kind of similar in practice, leading to scores between 0 and 1, but this is not mathematically guaranteed and depends on the used embedder and how it stretches the embedding space.

The following example shows this:

from ragas.metrics import AnswerRelevancy
from langchain_core import embeddings

# demo embedder
class MyEmbeddings(embeddings.Embeddings):

    embedding_lookup = { # some example embeddings
        "Where is France and what is it’s capital?": [0.33, 0.33, 0.333],
         "Pig?": [-0.33, -0.33, -0.33],
        "Orange!": [0.1, 0.2, 0.3],
        "Dark 897897?": [-1000, 0, -1000]
    }

    def embed_documents(self, texts):
        return [self.embedding_lookup[text] for text in texts]

    def embed_query(self, text):
        return self.embedding_lookup[text]
    

answer_relevancy = AnswerRelevancy()
answer_relevancy.embeddings =  MyEmbeddings()

question = "Where is France and what is it’s capital?"

generated_questions = [
    "Pig?", 
    "Orange!", 
    "Dark 897897?"
]

r = answer_relevancy.calculate_similarity(question, generated_questions)
# Score can be negative, depending on embedder:
print(r)
[-0.99999087  0.92721016 -0.8177225 ]

I updated the description and added a formula to make it clearer.

@shahules786
Copy link
Member

Thanks buddy @jokokojote You are awesome!

@shahules786 shahules786 merged commit 0cd01df into explodinggradients:main Mar 5, 2024
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants