feat: Add FaithfulnessEvaluator component

[julian-risch]

Related Issues

• fixes LLM Eval - Implement Faithfulness/Factual Accuracy metric #7024

Proposed Changes:

• Added a new component FaithfulnessEvaluator that returns one aggregated faithfulness score and individual faithfulness scores for inputs of queries, contexts and responses. Uses LLMEvaluator under the hood.

How did you test it?

New unit tests and the following local example:

from haystack import Pipeline
from haystack.components.evaluators import FaithfulnessEvaluator

QUESTIONS = ["Which is the most popular global sport?", "Who created the Python language?"]
CONTEXTS = [
    [
        "The popularity of sports can be measured in various ways, including TV viewership, social media presence, number of participants, and economic impact. Football is undoubtedly the world's most popular sport with major events like the FIFA World Cup and sports personalities like Ronaldo and Messi, drawing a followership of more than 4 billion people."
    ],
    [
        "Python, created by Guido van Rossum in the late 1980s, is a high-level general-purpose programming language. Its design philosophy emphasizes code readability, and its language constructs aim to help programmers write clear, logical code for both small and large-scale software projects."
    ],
]
RESPONSES = [
    "Football is the most popular sport with around 4 billion followers worldwide.",
    "Python is a high-level general-purpose programming language that was created by George Lucas.",
]

pipeline = Pipeline()
evaluator = FaithfulnessEvaluator()
pipeline.add_component("evaluator", evaluator)

results = pipeline.run({"evaluator": {"questions": QUESTIONS, "contexts": CONTEXTS, "responses": RESPONSES}})

print(results["evaluator"])
# {'results': [{'statements': ["Football is undoubtedly the world's most popular sport.", 'Football has around 4
# billion followers worldwide.'], 'statement_scores': [1, 1], 'name': 'llm', 'score': 1.0}, {'statements': ['Python
# is a high-level general-purpose programming language.', 'Python was created by George Lucas.'], 'statement_scores':
# [1, 0], 'score': 0.5}], 'score': 0.75, 'individual_scores': [1.0, 0.5]}

Notes for the reviewer

We can discuss a good name separately. FaithfulnessEvaluator, GroundednessEvaluator or HallucinationEvaluator are good candidates. deepset Cloud has a groundedness metric already: https://docs.cloud.deepset.ai/docs/use-groundedness-observability

In contrast to the original issue description this PR doesn't calculate a binary score per answer but per statement in answer. This calculation is more complex but it's also more meaningful and standard in other eval frameworks.

Other frameworks do the splitting of an answer into statements in separate prompts. Here are examples:

• https://github.com/explodinggradients/ragas/blob/e9418237d0dd06f640b0f6bbae7a5a3f9c1029ea/src/ragas/metrics/_faithfulness.py#L91
• https://github.com/confident-ai/deepeval/blob/d5aa87b65308a0d683f09dafdb020715838c4e27/deepeval/metrics/faithfulness/template.py#L4

Checklist

• I have read the contributors guidelines and the code of conduct
• I have updated the related issue with new insights and changes
• I added unit tests and updated the docstrings
• I've used one of the conventional commit types for my PR title: fix:, feat:, build:, chore:, ci:, docs:, style:, refactor:, perf:, test:.
• I documented my code
• I ran pre-commit hooks and fixed any issue

[shadeMe]

LGTM!

[coveralls]

Pull Request Test Coverage Report for Build 8557979991

Warning: This coverage report may be inaccurate.

This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.

• For more information on this, see Tracking coverage changes with pull request builds.
• To avoid this issue with future PRs, see these Recommended CI Configurations.
• For a quick fix, rebase this PR at GitHub. Your next report should be accurate.

Details

• 0 of 0 changed or added relevant lines in 0 files are covered.
• No unchanged relevant lines lost coverage.
• Overall coverage increased (+0.05%) to 89.407%

---

Totals
Change from base Build 8557433583:	0.05%
Covered Lines:	5697
Relevant Lines:	6372

---

💛 - Coveralls