-
Notifications
You must be signed in to change notification settings - Fork 1.1k
Open
Labels
enhancementNew feature or requestNew feature or requestmodule-metricsthis is part of metrics modulethis is part of metrics module
Description
Problem
The new collections-based AnswerCorrectness metric requires embeddings even when they're not used (when weights[1] == 0).
Current Behavior:
# Embeddings always required, even if unused
scorer = AnswerCorrectness(
llm=llm,
embeddings=embeddings, # Required even with weights=[1.0, 0.0]
weights=[1.0, 0.0] # 100% factuality, 0% similarity
)Legacy Behavior:
The legacy implementation only created embeddings when weights[1] != 0, making them truly optional for factuality-only use cases.
Proposed Solution
Make embeddings optional and validate based on weights:
def __init__(
self,
llm: "InstructorBaseRagasLLM",
embeddings: Optional["BaseRagasEmbedding"] = None,
weights: List[float] = [0.75, 0.25],
...
):
if weights[1] > 0 and embeddings is None:
raise ValueError(
"Embeddings required when similarity weight > 0. "
"Provide embeddings or set weights=[1.0, 0.0] for factuality only."
)
self.embeddings = embeddingsThis maintains flexibility while providing clear error messages when embeddings are actually needed.
File: src/ragas/metrics/collections/_answer_correctness.py
dosubot
Metadata
Metadata
Assignees
Labels
enhancementNew feature or requestNew feature or requestmodule-metricsthis is part of metrics modulethis is part of metrics module