-
Notifications
You must be signed in to change notification settings - Fork 1.1k
Closed
Description
Describe the bug
ValueError: diag requires an array of at least two dimensions
Ragas version: git+https://github.com/explodinggradients/ragas.git
5f105c0
Python version:
3.8.18
Code to Reproduce
from langchain.chat_models import ChatOpenAI
from ragas.llms.base import LangchainLLMWrapper
from datasets import Dataset
data_samples = {
'question': ['When was the first super bowl?', 'Who won the most super bowls?'],
'answer': ['The first superbowl was held on January 15, 1967', 'The most super bowls have been won by The New England Patriots'],
'contexts' : [['The Super Bowl....season since 1966,','replacing the NFL...in February.'],
['The Green Bay Packers...Green Bay, Wisconsin.','The Packers compete...Football Conference']],
'ground_truths': [['The first superbowl was held on January 15, 1967'], ['The New England Patriots have won the Super Bowl a record six times']]
}
dataset = Dataset.from_dict(data_samples)
gpt4 = ChatOpenAI(
model="gpt-3.5-turbo",
openai_api_key="sk-xxxx",
openai_api_base="https://xxxx",
)
vllm = LangchainLLMWrapper(langchain_llm=gpt4)
from ragas.embeddings import HuggingfaceEmbeddings
hf_embeddings = HuggingfaceEmbeddings(model_name="BAAI/bge-small-en")
from ragas.metrics import AnswerSimilarity
answer_similarity = AnswerSimilarity(llm=vllm, embeddings=hf_embeddings)
from ragas import evaluate
result = evaluate(llm=vllm,embeddings=hf_embeddings,dataset=dataset,metrics=[answer_similarity])
print(result)
Error trace
`from langchain_community.chat_models import ChatOpenAI`.
To install langchain-community run `pip install -U langchain-community`.
warnings.warn(
/Users/test/PycharmProjects/gptTest/venv/lib/python3.8/site-packages/langchain/chat_models/__init__.py:31: LangChainDeprecationWarning: Importing chat models from langchain is deprecated. Importing from langchain will no longer be supported as of langchain==0.2.0. Please import from langchain-community instead:
`from langchain_community.chat_models import AzureChatOpenAI`.
To install langchain-community run `pip install -U langchain-community`.
warnings.warn(
/Users/test/PycharmProjects/gptTest/venv/lib/python3.8/site-packages/langchain/chat_models/__init__.py:31: LangChainDeprecationWarning: Importing chat models from langchain is deprecated. Importing from langchain will no longer be supported as of langchain==0.2.0. Please import from langchain-community instead:
`from langchain_community.chat_models import ChatVertexAI`.
To install langchain-community run `pip install -U langchain-community`.
warnings.warn(
/Users/test/PycharmProjects/gptTest/venv/lib/python3.8/site-packages/langchain/llms/__init__.py:548: LangChainDeprecationWarning: Importing LLMs from langchain is deprecated. Importing from langchain will no longer be supported as of langchain==0.2.0. Please import from langchain-community instead:
`from langchain_community.llms import AzureOpenAI`.
To install langchain-community run `pip install -U langchain-community`.
warnings.warn(
/Users/test/PycharmProjects/gptTest/venv/lib/python3.8/site-packages/langchain/llms/__init__.py:548: LangChainDeprecationWarning: Importing LLMs from langchain is deprecated. Importing from langchain will no longer be supported as of langchain==0.2.0. Please import from langchain-community instead:
`from langchain_community.llms import OpenAI`.
To install langchain-community run `pip install -U langchain-community`.
warnings.warn(
/Users/test/PycharmProjects/gptTest/venv/lib/python3.8/site-packages/langchain/llms/__init__.py:548: LangChainDeprecationWarning: Importing LLMs from langchain is deprecated. Importing from langchain will no longer be supported as of langchain==0.2.0. Please import from langchain-community instead:
`from langchain_community.llms import VertexAI`.
To install langchain-community run `pip install -U langchain-community`.
warnings.warn(
/Users/test/PycharmProjects/gptTest/venv/lib/python3.8/site-packages/langchain/embeddings/__init__.py:29: LangChainDeprecationWarning: Importing embeddings from langchain is deprecated. Importing from langchain will no longer be supported as of langchain==0.2.0. Please import from langchain-community instead:
`from langchain_community.embeddings import AzureOpenAIEmbeddings`.
To install langchain-community run `pip install -U langchain-community`.
warnings.warn(
/Users/test/PycharmProjects/gptTest/venv/lib/python3.8/site-packages/langchain/embeddings/__init__.py:29: LangChainDeprecationWarning: Importing embeddings from langchain is deprecated. Importing from langchain will no longer be supported as of langchain==0.2.0. Please import from langchain-community instead:
`from langchain_community.embeddings import FastEmbedEmbeddings`.
To install langchain-community run `pip install -U langchain-community`.
warnings.warn(
/Users/test/PycharmProjects/gptTest/venv/lib/python3.8/site-packages/langchain/embeddings/__init__.py:29: LangChainDeprecationWarning: Importing embeddings from langchain is deprecated. Importing from langchain will no longer be supported as of langchain==0.2.0. Please import from langchain-community instead:
`from langchain_community.embeddings import OpenAIEmbeddings`.
To install langchain-community run `pip install -U langchain-community`.
warnings.warn(
/Users/test/PycharmProjects/gptTest/venv/lib/python3.8/site-packages/langchain_core/_api/deprecation.py:117: LangChainDeprecationWarning: The class `langchain_community.chat_models.openai.ChatOpenAI` was deprecated in langchain-community 0.0.10 and will be removed in 0.2.0. An updated version of the class exists in the langchain-openai package and should be used instead. To use it run `pip install -U langchain-openai` and import as `from langchain_openai import ChatOpenAI`.
warn_deprecated(
Evaluating: 0%| | 0/2 [00:00<?, ?it/s]
Traceback (most recent call last):
File "/Users/test/PycharmProjects/gptTest/venv/lib/python3.8/site-packages/ragas/evaluation.py", line 176, in evaluate
raise e
File "/Users/test/PycharmProjects/gptTest/venv/lib/python3.8/site-packages/ragas/evaluation.py", line 159, in evaluate
results = executor.results()
File "/Users/test/PycharmProjects/gptTest/venv/lib/python3.8/site-packages/ragas/executor.py", line 118, in results
raise e
File "/Users/test/PycharmProjects/gptTest/venv/lib/python3.8/site-packages/ragas/executor.py", line 114, in results
r = future.result()
File "/opt/homebrew/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/concurrent/futures/_base.py", line 437, in result
return self.__get_result()
File "/opt/homebrew/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
raise self._exception
File "/opt/homebrew/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/concurrent/futures/thread.py", line 57, in run
result = self.fn(*self.args, **self.kwargs)
File "/Users/test/PycharmProjects/gptTest/venv/lib/python3.8/site-packages/ragas/executor.py", line 36, in wrapped_callable
return counter, callable(*args, **kwargs)
File "/Users/test/PycharmProjects/gptTest/venv/lib/python3.8/site-packages/ragas/metrics/base.py", line 75, in score
raise e
File "/Users/test/PycharmProjects/gptTest/venv/lib/python3.8/site-packages/ragas/metrics/base.py", line 71, in score
score = self._score(row=row, callbacks=group_cm)
File "/Users/test/PycharmProjects/gptTest/venv/lib/python3.8/site-packages/ragas/metrics/_answer_similarity.py", line 73, in _score
scores = np.diagonal(similarity)
File "<__array_function__ internals>", line 200, in diagonal
File "/Users/test/PycharmProjects/gptTest/venv/lib/python3.8/site-packages/numpy/core/fromnumeric.py", line 1705, in diagonal
return asanyarray(a).diagonal(offset=offset, axis1=axis1, axis2=axis2)
ValueError: diag requires an array of at least two dimensions
Expected behavior
I would like to create a pull request to solve this issue:
In the AnswerSimilarity class, the function def _score(self, row: t.Dict, callbacks: Callbacks) -> float on line 73 has scores = np.diagonal(similarity).
If the variable similarity contains only one element, for example, [0.784738], then there is no need to extract the diagonal. The similarity score is already the only element in the similarity array.
The code can be modified to:
if similarity.size == 1:
# If similarity has only one value, directly use this value as scores
scores = similarity.flatten()
else:
# If similarity contains multiple values, extract the diagonal as scores
scores = np.diagonal(similarity)
Metadata
Metadata
Assignees
Labels
No labels