Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: is_async not passed for llm generation in context recall #777

Conversation

jokokojote
Copy link
Contributor

In _context_recall.py the is_async flag is not passed to the llm.generate function, like it is done for the other metrics which leads to problems when utilising other APIs then OpenAI.

Example:

from ragas.metrics import context_recall
from datasets import Dataset
from ragas import evaluate

import os
os.environ["OPENAI_API_KEY"] = ""
os.environ["RAGAS_DO_NOT_TRACK"] = "true"

questions = [
    "Where is France and what is it’s capital?",
  ]

ground_truths = [
  "France is in Western Europe and its capital is Paris.",
  ]

contexts =[
    [  
        "The country is also renowned for its wines and sophisticated cuisine. Lascaux’s ancient cave drawings, Lyon’s Roman theater and", 
        "The country is also renowned for its wines and sophisticated cuisine. Lascaux’s ancient cave drawings, Lyon’s Roman theater and", 
    ]
]

data = {
    "question": questions,
    "contexts": contexts,
    "ground_truth": ground_truths
}

# Convert dict to dataset
dataset = Dataset.from_dict(data)
dataset


# OpenAI works
result = evaluate(
    dataset = dataset, 
    metrics=[
        context_recall,
    ],
) 

print(result.to_pandas())


# Bedrock does not work
os.environ['AWS_ACCESS_KEY_ID'] = ''
os.environ['AWS_SECRET_ACCESS_KEY'] = ''
from langchain_community.llms.bedrock import Bedrock
llm = Bedrock(model_id="anthropic.claude-v2:1")

result = evaluate(
    dataset = dataset, 
    metrics=[
        context_recall,
    ],
    llm=llm
) # not working, timeout, error: "Streaming must be set to True for async operations."

print(result.to_pandas())

Copy link
Member

@shahules786 shahules786 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks bro @jokokojote

@shahules786 shahules786 merged commit 66d236c into explodinggradients:main Mar 19, 2024
15 checks passed
mrtj pushed a commit to mrtj/ragas that referenced this pull request Mar 26, 2024
…inggradients#777)

In `_context_recall.py` the `is_async` flag is not passed to the
`llm.generate` function, like it is done for the other metrics which
leads to problems when utilising other APIs then OpenAI.

Example:
```
from ragas.metrics import context_recall
from datasets import Dataset
from ragas import evaluate

import os
os.environ["OPENAI_API_KEY"] = ""
os.environ["RAGAS_DO_NOT_TRACK"] = "true"

questions = [
    "Where is France and what is it’s capital?",
  ]

ground_truths = [
  "France is in Western Europe and its capital is Paris.",
  ]

contexts =[
    [  
        "The country is also renowned for its wines and sophisticated cuisine. Lascaux’s ancient cave drawings, Lyon’s Roman theater and", 
        "The country is also renowned for its wines and sophisticated cuisine. Lascaux’s ancient cave drawings, Lyon’s Roman theater and", 
    ]
]

data = {
    "question": questions,
    "contexts": contexts,
    "ground_truth": ground_truths
}

# Convert dict to dataset
dataset = Dataset.from_dict(data)
dataset


# OpenAI works
result = evaluate(
    dataset = dataset, 
    metrics=[
        context_recall,
    ],
) 

print(result.to_pandas())


# Bedrock does not work
os.environ['AWS_ACCESS_KEY_ID'] = ''
os.environ['AWS_SECRET_ACCESS_KEY'] = ''
from langchain_community.llms.bedrock import Bedrock
llm = Bedrock(model_id="anthropic.claude-v2:1")

result = evaluate(
    dataset = dataset, 
    metrics=[
        context_recall,
    ],
    llm=llm
) # not working, timeout, error: "Streaming must be set to True for async operations."

print(result.to_pandas())
```
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants