Skip to content

Make embeddings optional in AnswerCorrectness metric when similarity weight is 0 #2408

@sanjeed5

Description

@sanjeed5

Problem

The new collections-based AnswerCorrectness metric requires embeddings even when they're not used (when weights[1] == 0).

Current Behavior:

# Embeddings always required, even if unused
scorer = AnswerCorrectness(
    llm=llm, 
    embeddings=embeddings,  # Required even with weights=[1.0, 0.0]
    weights=[1.0, 0.0]  # 100% factuality, 0% similarity
)

Legacy Behavior:
The legacy implementation only created embeddings when weights[1] != 0, making them truly optional for factuality-only use cases.

Proposed Solution

Make embeddings optional and validate based on weights:

def __init__(
    self,
    llm: "InstructorBaseRagasLLM",
    embeddings: Optional["BaseRagasEmbedding"] = None,
    weights: List[float] = [0.75, 0.25],
    ...
):
    if weights[1] > 0 and embeddings is None:
        raise ValueError(
            "Embeddings required when similarity weight > 0. "
            "Provide embeddings or set weights=[1.0, 0.0] for factuality only."
        )
    
    self.embeddings = embeddings

This maintains flexibility while providing clear error messages when embeddings are actually needed.

File: src/ragas/metrics/collections/_answer_correctness.py

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestmodule-metricsthis is part of metrics module

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions