[Question] Context Recall different scores for the same context (questions and GT)

[checkcode555]

I am wondering why the RAGAS metric "context recall" is sometimes different for the same question, the same context and the same ground truth.

I reduced my dataset to 1 question. I see the same context for each test, because the retrieval model is the same, but different answers since I test different LLMs for summarization. The ground truth is also the same for all cases.

The values I get are 0.5 for one test case and 1.0 for another. If I change the question I can get 0 and 1, for example.

I expect "context recall" values be the same for both cases.

The same problem with "context precision".

[shahules786]

Hey, can you share the data point for which you observed this? @checkcode555 so that we can debug the issue

[checkcode555]

Hi @shahules786, unfortunately I cannot share my data. I was wondering whether someone else experiences this problem as well.

[roxanamirel]

I have a somewhat similar issue related to how context recall is computed. For an empty list context, the context recall is always 1. The ground truth is not empty, but given that no context was retrieved from the retriever component, shouldn't the context recall be 0?

[shahules786]

Hey @roxanamirel feel free to reinstall ragas from source and to resolve your issue.

[roxanamirel]

@shahules786 I reinstalled from source "Successfully installed ragas-0.1.4.dev5+g169c6e5" and I get the same result, context recall 1, where contexts list is empty. Thanks!

[checkcode555]

Just an update, that it turns to be that LLM is involved in calculating the context recall score. It means that the usage of different LLMs would lead to different scores. What is confusing is that the scores vary a lot and can be even 0 while using gpt4 for exampe and 0.8 while using gpt3.5 for the same input data. The question is then how to interpret these scores and whether it is trustable. I was wondering whether the calculation can be more transparent and intermediate results can be provided (not only the final score as it is at the moment).