IndexError: list index out of range

[0ENZO]

Describe the bug
I have added some print statement to the source code. results[len(metrics) * i + j] is causing the error because results is empty.
The error is not triggered at the beginning of the process, but later, 61% (408/672).

Ragas version: up-to date with the github repo
Python version: 3.10.13

Code to Reproduce

embedding_model_id = "BAAI/bge-large-en-v1.5"
embeddings = HuggingFaceEmbeddings(model_name=embedding_model_id)

#eval_df ['question', 'question_id', 'reference_answer', 'reference_source', 'reference_source_id', 'answer', 'source', 'source_id']
def ragas_format(example):
    example["contexts"] = [example["reference_source"]]
    example["ground_truth"] = example["reference_answer"]
    return example

project_id = ""
config = {"project_id": project_id}
creds, _ = google.auth.default(quota_project_id=project_id)

chat = ChatVertexAI(credentials=creds)
ragas_vertexai_llm = LangchainLLMWrapper(chat)

eval_ds = datasets.Dataset.from_pandas(eval_df)

metrics = [
    ragas.metrics.faithfulness,
    ragas.metrics.context_relevancy,
    ragas.metrics.context_recall
]

for m in metrics:
    m.__setattr__("llm", ragas_vertexai_llm)

    if hasattr(m, "embeddings"):
        m.__setattr__("embeddings", embeddings)

results = ragas.evaluate(
    eval_ds.map(ragas_format), 
    metrics=metrics,
    llm=ragas_vertexai_llm,
    embeddings=embeddings
)

Error trace

Invalid JSON response. Expected dictionary with key 'Attributed'
Invalid JSON response. Expected dictionary with key 'Attributed'
Invalid JSON response. Expected dictionary with key 'Attributed'
Invalid JSON response. Expected dictionary with key 'Attributed'
Invalid JSON response. Expected dictionary with key 'Attributed'
Invalid JSON response. Expected dictionary with key 'Attributed'
Invalid JSON response. Expected dictionary with key 'Attributed'
Invalid JSON response. Expected dictionary with key 'Attributed'
Invalid JSON response. Expected dictionary with key 'Attributed'
Invalid JSON response. Expected dictionary with key 'Attributed'
Invalid JSON response. Expected dictionary with key 'Attributed'
Invalid JSON response. Expected dictionary with key 'Attributed'
Invalid JSON response. Expected dictionary with key 'Attributed'
Invalid JSON response. Expected dictionary with key 'Attributed'
Invalid JSON response. Expected dictionary with key 'Attributed'
Invalid JSON response. Expected dictionary with key 'Attributed'
Invalid JSON response. Expected dictionary with key 'Attributed'
Invalid JSON response. Expected dictionary with key 'Attributed'
Invalid JSON response. Expected dictionary with key 'Attributed'
Invalid JSON response. Expected dictionary with key 'Attributed'
Invalid JSON response. Expected dictionary with key 'Attributed'
Invalid JSON response. Expected dictionary with key 'Attributed'
Invalid JSON response. Expected dictionary with key 'Attributed'
Invalid JSON response. Expected dictionary with key 'Attributed'
Invalid JSON response. Expected dictionary with key 'Attributed'
Exception in thread Thread-10:
Traceback (most recent call last):
  File "/opt/conda/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "/home/jupyter/ragas/src/ragas/executor.py", line 66, in run
    results = self.loop.run_until_complete(self._aresults())
  File "/opt/conda/lib/python3.10/site-packages/nest_asyncio.py", line 99, in run_until_complete
    return f.result()
  File "/opt/conda/lib/python3.10/asyncio/futures.py", line 201, in result
    raise self._exception.with_traceback(self._exception_tb)
  File "/opt/conda/lib/python3.10/asyncio/tasks.py", line 232, in __step
    result = coro.send(None)
  File "/home/jupyter/ragas/src/ragas/executor.py", line 54, in _aresults
    raise e
  File "/home/jupyter/ragas/src/ragas/executor.py", line 49, in _aresults
    r = await future
  File "/opt/conda/lib/python3.10/asyncio/tasks.py", line 571, in _wait_for_one
    return f.result()  # May raise f.exception().
  File "/opt/conda/lib/python3.10/asyncio/futures.py", line 201, in result
    raise self._exception.with_traceback(self._exception_tb)
  File "/opt/conda/lib/python3.10/asyncio/tasks.py", line 232, in __step
    result = coro.send(None)
  File "/home/jupyter/ragas/src/ragas/executor.py", line 82, in wrapped_callable_async
    return counter, await callable(*args, **kwargs)
  File "/home/jupyter/ragas/src/ragas/metrics/base.py", line 91, in ascore
    raise e
  File "/home/jupyter/ragas/src/ragas/metrics/base.py", line 87, in ascore
    score = await self._ascore(row=row, callbacks=group_cm, is_async=is_async)
  File "/home/jupyter/ragas/src/ragas/metrics/_context_recall.py", line 119, in _ascore
    return self._compute_score(response)
  File "/home/jupyter/ragas/src/ragas/metrics/_context_recall.py", line 94, in _compute_score
    response = [
  File "/home/jupyter/ragas/src/ragas/metrics/_context_recall.py", line 96, in <listcomp>
    if item.get("Attributed")
AttributeError: 'str' object has no attribute 'get'

len results : 0
len dataset : 224
len(metrics) * i + j : 0

---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
Cell In[18], line 3
      1 nest_asyncio.apply()
----> 3 results = ragas.evaluate(
      4     eval_ds.map(ragas_format), 
      5     metrics=metrics,
      6     llm=ragas_vertexai_llm,
      7     embeddings=embeddings
      8 )

File ~/ragas/src/ragas/evaluation.py:222, in evaluate(dataset, metrics, llm, embeddings, callbacks, is_async, max_workers, run_config, raise_exceptions, column_map)
    219     if not evaluation_group_cm.ended:
    220         evaluation_rm.on_chain_error(e)
--> 222     raise e
    223 else:
    224     result = Result(
    225         scores=Dataset.from_list(scores),
    226         dataset=dataset,
    227         binary_columns=binary_metrics,
    228     )

File ~/ragas/src/ragas/evaluation.py:210, in evaluate(dataset, metrics, llm, embeddings, callbacks, is_async, max_workers, run_config, raise_exceptions, column_map)
    208 for j, m in enumerate(metrics):
    209     print(f"len(metrics) * i + j : {len(metrics) *i +j}")
--> 210     s[m.name] = results[len(metrics) * i + j]
    211 scores.append(s)
    212 # close the row chain

IndexError: list index out of range

[jjmachan]

thanks for putting the effort to write a really good issue. much appreciated @0ENZO 🙂

there are 2 issues here as far as I can see

1. Vertex AI is not working as it should be. the JSON format is not being followed. tagging @shahules786 for his expertise here
2. the misleading error message - that I will fix right now :)

thanks for bringing this up 0ENZO!

[jjmachan]

hey @0ENZO fixed the incorrect error messsage here #561. Let me know if that one makes more sense when trying to debug

will try to look into the vertex AI issue shortly

[shahules786]

Hey @0ENZO , I'm on it. Most likely it is due to that vertex AI is unable to generate valid json output even with the retry mechanism.