API KEY

[amin-kh96]

[ ] I checked the documentation and related resources and couldn't find an answer to my question.

Your Question
I wrote this code and I get the error:
The api_key client option must be set either by passing api_key to the client or by setting the OPENAI_API_KEY environment variable
is it mandatory to set an API key to evaluate?
if i have to use Api-key is there any free way to use it?

Code Examples

import json
import pandas as pd
from datasets import Dataset
from langchain.embeddings import HuggingFaceEmbeddings
from ragas.embeddings import LangchainEmbeddingsWrapper
from ragas.metrics import faithfulness,context_utilization
from ragas import evaluate

# Load the ground_truthpip install --upgrade langchain ragas
file_path = 'assets\\GT.json'
with open(file_path) as f:
    ground_truth_data = json.load(f)

#load the question and the answer and the chunks
file_path = 'assets\\user_llm_interaction_embeddings_c1521dd5_b819_4241_b3a4_3e5c1388037c.json'
with open(file_path) as f:
     llm=json.load(f)
#Initialize an empty list to hold the new dataset
data_set = []

#Iterate through the list and combine every two dictionaries
for i in range(0, len(llm), 2):
    combined_dict = {
        "text_vector_1": llm[i].get("text_vector", []),
        "text_vector_2": llm[i + 1].get("text_vector", []),
        'chunks': llm[i + 1].get('chunks', [])
    }
    data_set.append(combined_dict)



def map_chunks(data_set, ground_truth_data):
    for item in data_set:  # Iterate over each dictionary in data_set
        c = []  # Reset c for each item
        for chunk_id in item['chunks']:  # Loop through 'chunks' in the current dictionary
            for element in ground_truth_data:  # Loop through ground_truth_data
                if element['id'] == chunk_id:  # Match chunk_id with element's id
                    c.append(element['text_vector'])  # Append the matching text_vector to c
        item['chunks'] = c  # Replace the original 'chunks' (ids) with the mapped text_vector values

    return data_set  # Return the updated data_set

data_set = map_chunks(data_set, ground_truth_data)

# Assuming data_set is a list of dictionaries
ragas_data = [
    {
        "question": entry["text_vector_1"],
        "answer": entry["text_vector_2"],
        "contexts": entry["chunks"]
    }
    for entry in data_set
]
# Convert list of dictionaries to dictionary of lists
ragas_data_dict = {key: [d[key] for d in ragas_data] for key in ragas_data[0]}


# Initialize Hugging Face embeddings
emb = HuggingFaceEmbeddings(model_name="sentence-transformers/all-MiniLM-L6-v2")

# Wrap the embeddings
wrapped_embeddings = LangchainEmbeddingsWrapper(embeddings=emb)

# Assuming you have your dataset ready
dataset = ragas_data_dict  # Your dataset here


# Step 4: Define the list of evaluation metrics
metrics = [faithfulness,context_utilization]

# Step 5: Evaluate the data using the RAGAS `evaluate` function
evaluation_report = evaluate(dataset=dataset, metrics=metrics, embeddings=emb)
# Step 6: Display the evaluation results
print("RAGAS Evaluation Report:")
print(evaluation_report)

# Optional: Convert the evaluation report to a DataFrame for better readability
evaluation_df = pd.DataFrame(evaluation_report.items(), columns=['Metric', 'Score'])
print("\nEvaluation Report as DataFrame:")
print(evaluation_df)

Additional context
I have the embeddings my dataset is not str

[dosubot]

Yes, it is mandatory to set an API key to evaluate using the evaluate function in RAGAS. The document states:

"Many metrics use OpenAI models and require an environment variable OPENAI_API_KEY to be set."

However, there is an alternative mentioned in the document. You can use the Opik platform, which provides a hosted version and also allows for local setup. This means you can run the Opik platform locally without needing an OpenAI API key. For more information on setting up Opik locally, refer to the installation guide.

Additionally, you can use Langfuse for online evaluation of your RAG pipelines. Langfuse allows you to score your traces and spans without needing ground-truth data, making it suitable for production environments. You can set up Langfuse with your own keys and use it to perform model-based evaluations. For more details, refer to the Langfuse documentation ^[1][2][3].

To continue talking to Dosu, mention @dosu.

[amin-kh96]

but I am confused when I calculated the embeddings and also I did not set the LLM why do I have to use the API key? it is for embedding.

[dosubot]

You need to use an API key for embedding calculations because the MetricWithEmbeddings class requires valid embeddings to be provided. The init method of this class checks if the embeddings are valid and also verifies the API key for services like OpenAI and AzureOpenAI. If no valid embeddings are provided, it raises a ValueError.

@dataclass
class MetricWithEmbeddings(Metric):
    embeddings: t.Optional[BaseRagasEmbeddings] = None

    def init(self, run_config: RunConfig):
        """
        Init any models in the metric, this is invoked before evaluate()
        to load all the models
        Also check if the api key is valid for OpenAI and AzureOpenAI
        """
        if self.embeddings is None:
            raise ValueError(
                f"Metric '{self.name}' has no valid embeddings provided (self.embeddings is None). Please initantiate a the metric with an embeddings to run."  # noqa
            )
        self.embeddings.set_run_config(run_config)

This means that even if you did not set the LLM, you still need to provide valid embeddings and an API key for the embedding calculations to proceed. Ragas uses LLMs and Embeddings for both evaluation and test set generation, and by default, it uses OpenAI models. If you are using custom embeddings, they should be a subclass of BaseRagasEmbeddings, and you need to ensure that the API key is valid for the service you are using ^[1][2].

To continue talking to Dosu, mention @dosu.

[amin-kh96]

I have another question, I have a vector database where PDFs and other kinds of documents are stored there. but they are split into chunks and they are vectorized. these files are very specialized for example chunks are explaining a specific error that is related to a specific medical device, should I use an LLM for ground truth in this case? or I should create a ground truth dataset? but even if I create a ground truth dataset I have to use the PDFs or documents that I have in my vector database and literally they are the same, help me with this

[dosubot]

You should create a ground truth dataset when your vector database contains specialized documents split into chunks and vectorized, such as those explaining specific errors related to medical devices. This ensures accurate and reliable evaluations, especially in specialized domains like medical devices ^[1].

The ground truth dataset should include the following columns:

1. Question: A set of questions.
2. Contexts: Retrieved contexts corresponding to each question. This is a list[list] since each question can retrieve multiple text chunks.
3. Answer: Generated answer corresponding to each question.
4. Ground truth: Ground truth corresponding to each question. This is a list[str] which corresponds to the expected answer for each question.

Here is an example of how to structure your dataset:

from datasets import Dataset 

data_samples = {
    'question': ['When was the first super bowl?', 'Who won the most super bowls?'],
    'answer': ['The first superbowl was held on January 15, 1967', 'The most super bowls have been won by The New England Patriots'],
    'contexts' : [['The Super Bowl....season since 1966,','replacing the NFL...in February.'], 
    ['The Green Bay Packers...Green Bay, Wisconsin.','The Packers compete...Football Conference']],
    'ground_truth': [['The first superbowl was held on January 15, 1967'], ['The New England Patriots have won the Super Bowl a record six times']]
}
dataset = Dataset.from_dict(data_samples)

The dataset should have the following structure:

Dataset[question: list[str], contexts: list[list[str]], answer: list[str], ground_truth: list[list[str]]]

This format ensures that the metrics can properly evaluate the RAG pipeline ^[2][3][4].

To continue talking to Dosu, mention @dosu.

[amin-kh96]

no you are noticed my question. lets assume that I have dataset for rag evaluation, but I do not have the ground truth, how can I create or use GT

[dosubot]

If you do not have an existing ground truth (GT) dataset for RAG evaluation, you can use Ragas for reference-free evaluations. Ragas can perform evaluations on various aspects of your RAG pipeline without needing ground-truth data. This allows you to run evaluations on production traces collected with Langfuse.

Here is a brief overview of how you can set up and use Ragas for this purpose:

1. Install Required Packages:

   %pip install datasets ragas llama_index python-dotenv --upgrade

2. Load Your Dataset:

   from datasets import load_dataset
   
   amnesty_qa = load_dataset("explodinggradients/amnesty_qa", "english_v2")["eval"]
   amnesty_qa

3. Define Metrics:

   from ragas.metrics import faithfulness, answer_relevancy, context_precision
   from ragas.metrics.critique import SUPPORTED_ASPECTS, harmfulness
   
   metrics = [faithfulness, answer_relevancy, context_precision, harmfulness]

4. Initialize Metrics with LLMs and Embeddings:

   from ragas.run_config import RunConfig
   from ragas.metrics.base import MetricWithLLM, MetricWithEmbeddings
   from langchain_openai.chat_models import ChatOpenAI
   from langchain_openai.embeddings import OpenAIEmbeddings
   from ragas.llms import LangchainLLMWrapper
   from ragas.embeddings import LangchainEmbeddingsWrapper
   
   def init_ragas_metrics(metrics, llm, embedding):
       for metric in metrics:
           if isinstance(metric, MetricWithLLM):
               metric.llm = llm
           if isinstance(metric, MetricWithEmbeddings):
               metric.embeddings = embedding
           run_config = RunConfig()
           metric.init(run_config)
   
   llm = ChatOpenAI()
   emb = OpenAIEmbeddings()
   
   init_ragas_metrics(
       metrics,
       llm=LangchainLLMWrapper(llm),
       embedding=LangchainEmbeddingsWrapper(emb),
   )

5. Score Your Traces:

   async def score_with_ragas(query, chunks, answer):
       scores = {}
       for m in metrics:
           print(f"calculating {m.name}")
           scores[m.name] = await m.ascore(
               row={"question": query, "contexts": chunks, "answer": answer}
           )
       return scores
   
   row = amnesty_qa[0]
   question, contexts, answer = row["question"], row["contexts"], row["answer"]
   await score_with_ragas(question, contexts, answer)

By following these steps, you can evaluate your RAG pipeline without needing a ground truth dataset ^[1].

To continue talking to Dosu, mention @dosu.

[amin-kh96]

what metrics need GT

[amin-kh96]

how can I make sure that my embeddings are a subclass of BaseRagasEmbeddings?
is it better to let the ragas compute the embeddings from beginning?

[dosubot]

Metrics in Ragas that require ground truth (GT) include those that have the G - ground_truth: ground truth answer in their schema ^[1].

To ensure that your embeddings are a subclass of BaseRagasEmbeddings, you need to define your embeddings class to inherit from BaseRagasEmbeddings. Here is an example:

from ragas.embeddings import BaseRagasEmbeddings

class MyEmbeddings(BaseRagasEmbeddings):
    # Implement required methods and properties
    pass

It is generally better to let Ragas compute the embeddings from the beginning to ensure compatibility and proper initialization within the framework. This is because the MetricWithEmbeddings class checks if the embeddings are provided and initializes them accordingly ^[1].

To continue talking to Dosu, mention @dosu.

[amin-kh96]

if we let ragas to compute the embeddings is it more costly than we define an class embedding that inherit from baseragasembeddings ? or there is no such difference

[dosubot]

Letting Ragas compute the embeddings using its default settings can be more costly compared to defining a custom class that inherits from BaseRagasEmbeddings, depending on the models and APIs used.

Ragas' default behavior uses the "BAAI/bge-small-en-v1.5" model from Huggingface, which can be cost-effective if you have the necessary infrastructure to run it locally. However, if you use external services like OpenAI for embedding computation, the costs can add up due to API usage fees.

By defining a custom class that inherits from BaseRagasEmbeddings, you can control the embedding process and potentially use more cost-effective or specialized models that suit your needs. This approach allows you to optimize for both performance and cost.

Here is an example of how to define a custom embedding class:

from ragas.embeddings import BaseRagasEmbeddings
from typing import List

class MyCustomEmbeddings(BaseRagasEmbeddings):
    def embed_query(self, text: str) -> List[float]:
        # Implement your custom embedding logic here
        pass

    def embed_documents(self, texts: List[str]) -> List[List[float]]:
        # Implement your custom embedding logic here
        pass

    async def aembed_query(self, text: str) -> List[float]:
        # Implement your custom embedding logic here
        pass

    async def aembed_documents(self, texts: List[str]) -> List[List[float]]:
        # Implement your custom embedding logic here
        pass

    def set_run_config(self, run_config: RunConfig):
        self.run_config = run_config

This way, you can use your own models and infrastructure, potentially reducing costs compared to using external APIs like OpenAI ^[1].

To continue talking to Dosu, mention @dosu.

[amin-kh96]

give me the conclusion for this topic we are getting far from the real context.
I know it is better to let the ragas do the embeddings but if i want to consider the cost or price for this
is it better to use my own calculated embeddings and then in some ways or method make sure that my embeddings are suitable or compatible for using ragas,