Skip to content

ValueError: diag requires an array of at least two dimensions #474

@herman159

Description

@herman159

Describe the bug
ValueError: diag requires an array of at least two dimensions

Ragas version: git+https://github.com/explodinggradients/ragas.git
5f105c0
Python version:
3.8.18

Code to Reproduce

from langchain.chat_models import ChatOpenAI
from ragas.llms.base import LangchainLLMWrapper
from datasets import Dataset

data_samples = {
    'question': ['When was the first super bowl?', 'Who won the most super bowls?'],
    'answer': ['The first superbowl was held on January 15, 1967', 'The most super bowls have been won by The New England Patriots'],
    'contexts' : [['The Super Bowl....season since 1966,','replacing the NFL...in February.'],
    ['The Green Bay Packers...Green Bay, Wisconsin.','The Packers compete...Football Conference']],
    'ground_truths': [['The first superbowl was held on January 15, 1967'], ['The New England Patriots have won the Super Bowl a record six times']]
}
dataset = Dataset.from_dict(data_samples)

gpt4 = ChatOpenAI(
    model="gpt-3.5-turbo",
    openai_api_key="sk-xxxx",
    openai_api_base="https://xxxx",
)
vllm = LangchainLLMWrapper(langchain_llm=gpt4)

from ragas.embeddings import HuggingfaceEmbeddings
hf_embeddings = HuggingfaceEmbeddings(model_name="BAAI/bge-small-en")

from ragas.metrics import AnswerSimilarity

answer_similarity = AnswerSimilarity(llm=vllm, embeddings=hf_embeddings)

from ragas import evaluate
result = evaluate(llm=vllm,embeddings=hf_embeddings,dataset=dataset,metrics=[answer_similarity])
print(result)

Error trace

`from langchain_community.chat_models import ChatOpenAI`.
To install langchain-community run `pip install -U langchain-community`.
  warnings.warn(
/Users/test/PycharmProjects/gptTest/venv/lib/python3.8/site-packages/langchain/chat_models/__init__.py:31: LangChainDeprecationWarning: Importing chat models from langchain is deprecated. Importing from langchain will no longer be supported as of langchain==0.2.0. Please import from langchain-community instead:
`from langchain_community.chat_models import AzureChatOpenAI`.
To install langchain-community run `pip install -U langchain-community`.
  warnings.warn(
/Users/test/PycharmProjects/gptTest/venv/lib/python3.8/site-packages/langchain/chat_models/__init__.py:31: LangChainDeprecationWarning: Importing chat models from langchain is deprecated. Importing from langchain will no longer be supported as of langchain==0.2.0. Please import from langchain-community instead:
`from langchain_community.chat_models import ChatVertexAI`.
To install langchain-community run `pip install -U langchain-community`.
  warnings.warn(
/Users/test/PycharmProjects/gptTest/venv/lib/python3.8/site-packages/langchain/llms/__init__.py:548: LangChainDeprecationWarning: Importing LLMs from langchain is deprecated. Importing from langchain will no longer be supported as of langchain==0.2.0. Please import from langchain-community instead:
`from langchain_community.llms import AzureOpenAI`.
To install langchain-community run `pip install -U langchain-community`.
  warnings.warn(
/Users/test/PycharmProjects/gptTest/venv/lib/python3.8/site-packages/langchain/llms/__init__.py:548: LangChainDeprecationWarning: Importing LLMs from langchain is deprecated. Importing from langchain will no longer be supported as of langchain==0.2.0. Please import from langchain-community instead:
`from langchain_community.llms import OpenAI`.
To install langchain-community run `pip install -U langchain-community`.
  warnings.warn(
/Users/test/PycharmProjects/gptTest/venv/lib/python3.8/site-packages/langchain/llms/__init__.py:548: LangChainDeprecationWarning: Importing LLMs from langchain is deprecated. Importing from langchain will no longer be supported as of langchain==0.2.0. Please import from langchain-community instead:
`from langchain_community.llms import VertexAI`.
To install langchain-community run `pip install -U langchain-community`.
  warnings.warn(
/Users/test/PycharmProjects/gptTest/venv/lib/python3.8/site-packages/langchain/embeddings/__init__.py:29: LangChainDeprecationWarning: Importing embeddings from langchain is deprecated. Importing from langchain will no longer be supported as of langchain==0.2.0. Please import from langchain-community instead:
`from langchain_community.embeddings import AzureOpenAIEmbeddings`.
To install langchain-community run `pip install -U langchain-community`.
  warnings.warn(
/Users/test/PycharmProjects/gptTest/venv/lib/python3.8/site-packages/langchain/embeddings/__init__.py:29: LangChainDeprecationWarning: Importing embeddings from langchain is deprecated. Importing from langchain will no longer be supported as of langchain==0.2.0. Please import from langchain-community instead:
`from langchain_community.embeddings import FastEmbedEmbeddings`.
To install langchain-community run `pip install -U langchain-community`.
  warnings.warn(
/Users/test/PycharmProjects/gptTest/venv/lib/python3.8/site-packages/langchain/embeddings/__init__.py:29: LangChainDeprecationWarning: Importing embeddings from langchain is deprecated. Importing from langchain will no longer be supported as of langchain==0.2.0. Please import from langchain-community instead:
`from langchain_community.embeddings import OpenAIEmbeddings`.
To install langchain-community run `pip install -U langchain-community`.
  warnings.warn(
/Users/test/PycharmProjects/gptTest/venv/lib/python3.8/site-packages/langchain_core/_api/deprecation.py:117: LangChainDeprecationWarning: The class `langchain_community.chat_models.openai.ChatOpenAI` was deprecated in langchain-community 0.0.10 and will be removed in 0.2.0. An updated version of the class exists in the langchain-openai package and should be used instead. To use it run `pip install -U langchain-openai` and import as `from langchain_openai import ChatOpenAI`.
  warn_deprecated(
Evaluating:   0%|          | 0/2 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "/Users/test/PycharmProjects/gptTest/venv/lib/python3.8/site-packages/ragas/evaluation.py", line 176, in evaluate
    raise e
  File "/Users/test/PycharmProjects/gptTest/venv/lib/python3.8/site-packages/ragas/evaluation.py", line 159, in evaluate
    results = executor.results()
  File "/Users/test/PycharmProjects/gptTest/venv/lib/python3.8/site-packages/ragas/executor.py", line 118, in results
    raise e
  File "/Users/test/PycharmProjects/gptTest/venv/lib/python3.8/site-packages/ragas/executor.py", line 114, in results
    r = future.result()
  File "/opt/homebrew/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/concurrent/futures/_base.py", line 437, in result
    return self.__get_result()
  File "/opt/homebrew/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
    raise self._exception
  File "/opt/homebrew/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/Users/test/PycharmProjects/gptTest/venv/lib/python3.8/site-packages/ragas/executor.py", line 36, in wrapped_callable
    return counter, callable(*args, **kwargs)
  File "/Users/test/PycharmProjects/gptTest/venv/lib/python3.8/site-packages/ragas/metrics/base.py", line 75, in score
    raise e
  File "/Users/test/PycharmProjects/gptTest/venv/lib/python3.8/site-packages/ragas/metrics/base.py", line 71, in score
    score = self._score(row=row, callbacks=group_cm)
  File "/Users/test/PycharmProjects/gptTest/venv/lib/python3.8/site-packages/ragas/metrics/_answer_similarity.py", line 73, in _score
    scores = np.diagonal(similarity)
  File "<__array_function__ internals>", line 200, in diagonal
  File "/Users/test/PycharmProjects/gptTest/venv/lib/python3.8/site-packages/numpy/core/fromnumeric.py", line 1705, in diagonal
    return asanyarray(a).diagonal(offset=offset, axis1=axis1, axis2=axis2)
ValueError: diag requires an array of at least two dimensions

Expected behavior
I would like to create a pull request to solve this issue:
In the AnswerSimilarity class, the function def _score(self, row: t.Dict, callbacks: Callbacks) -> float on line 73 has scores = np.diagonal(similarity).
If the variable similarity contains only one element, for example, [0.784738], then there is no need to extract the diagonal. The similarity score is already the only element in the similarity array.
The code can be modified to:

if similarity.size == 1:
    # If similarity has only one value, directly use this value as scores
    scores = similarity.flatten()
else:
    # If similarity contains multiple values, extract the diagonal as scores
    scores = np.diagonal(similarity)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions