feature/Factuality test

[Prikshit7766]

Description

Title: Factuality Test Implementation for Language Models

Overview

This pull request introduces the implementation of the Factuality Test for language models (LLMs). The Factuality Test is designed to evaluate the ability of LLMs to determine the factuality of statements within summaries, particularly focusing on the accuracy of LLM-generated summaries and potential biases in their judgments.

Test Objective

The primary goal of the Factuality Test is to assess how well LLMs can identify the factual accuracy of summary sentences. This ensures that LLMs generate summaries consistent with the information presented in the source article.

Data Source

For this test, we utilize the Factual-Summary-Pairs dataset, which is sourced from the following GitHub repository: Factual-Summary-Pairs Dataset.

Methodology

Our test methodology draws inspiration from a reference article titled "LLAMA-2 is about as factually accurate as GPT-4 for summaries and is 30x cheaper".

Bias Identification

We identify bias in the responses based on specific patterns:

• Bias Towards A: Occurs when both the "result" and "swapped_result" are "A." This bias is in favor of "A," but it's incorrect, so it's marked as False.
• Bias Towards B: Occurs when both the "result" and "swapped_result" are "B." This bias is in favor of "B," but it's incorrect, so it's marked as False.
• No Bias : When "result" is "B" and "swapped_result" is "A," there is no bias. However, this statement is incorrect, so it's marked as False.
• No Bias : When "result" is "A" and "swapped_result" is "B," there is no bias. This statement is correct, so it's marked as True.

Accuracy Assessment

Accuracy is assessed by examining the "pass" column. If "pass" is marked as True, it indicates a correct response. Conversely, if "pass" is marked as False, it indicates an incorrect response.

Notebook

• Fixes Add new Factuality test #748

Type of change

Please delete options that are not relevant.

• New feature (non-breaking change which adds functionality)
• This change requires a documentation update

Usage

from langtest import Harness

import os
import openai
os.environ["OPENAI_API_KEY"] = "<YOUR_API_KEY>"

model = {"model": "text-davinci-003", "hub":"openai"}
data = {"data_source": "Factual-Summary-Pairs"}
harness = Harness(task="factuality-test", model=model, data=data)

Checklist:

• I've added Google style docstrings to my code.
• I've used pydantic for typing when/where necessary.
• I have linted my code
• I have added tests to cover my changes.

Screenshots (if appropriate):

These results were obtained after running the model on a set of 50 records.

[chakravarthik27]

LGTM 😄