generate_with_llamaindex_docs does not run to completion on v0.1.1 #612
Description
Describe the bug
generate_with_llamaindex_docs appears to have a regression in v0.1.1 and does not run to completion. The issue does not appear on v0.1.0
Ragas version: 0.1.1
Python version: 3.10
Code to Reproduce
Run the notebook in this PR.
Error trace
Exception in thread Thread-8:
Traceback (most recent call last):
File "/Users/xandersong/miniforge3/envs/ragas-test/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
self.run()
File "/Users/xandersong/miniforge3/envs/ragas-test/lib/python3.10/site-packages/ragas/executor.py", line 75, in run
results = self.loop.run_until_complete(self._aresults())
File "/Users/xandersong/miniforge3/envs/ragas-test/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
return future.result()
File "/Users/xandersong/miniforge3/envs/ragas-test/lib/python3.10/site-packages/ragas/executor.py", line 63, in _aresults
raise e
File "/Users/xandersong/miniforge3/envs/ragas-test/lib/python3.10/site-packages/ragas/executor.py", line 58, in _aresults
r = await future
File "/Users/xandersong/miniforge3/envs/ragas-test/lib/python3.10/asyncio/tasks.py", line 571, in _wait_for_one
return f.result() # May raise f.exception().
File "/Users/xandersong/miniforge3/envs/ragas-test/lib/python3.10/site-packages/ragas/executor.py", line 91, in wrapped_callable_async
return counter, await callable(*args, **kwargs)
File "/Users/xandersong/miniforge3/envs/ragas-test/lib/python3.10/site-packages/ragas/testset/evolutions.py", line 161, in evolve
) = await self._aevolve(current_tries, current_nodes)
File "/Users/xandersong/miniforge3/envs/ragas-test/lib/python3.10/site-packages/ragas/testset/evolutions.py", line 507, in _aevolve
result = await self._acomplex_evolution(
File "/Users/xandersong/miniforge3/envs/ragas-test/lib/python3.10/site-packages/ragas/testset/evolutions.py", line 380, in _acomplex_evolution
current_nodes = self.se._get_more_adjacent_nodes(current_nodes)
File "/Users/xandersong/miniforge3/envs/ragas-test/lib/python3.10/site-packages/ragas/testset/evolutions.py", line 128, in _get_more_adjacent_nodes
prev_adjacent_node = self.docstore.get_adjacent(
File "/Users/xandersong/miniforge3/envs/ragas-test/lib/python3.10/site-packages/ragas/testset/docstore.py", line 300, in get_adjacent
index = self.nodes.index(node)
ValueError: Node(page_content='complex reasoning by decomposing input-output\nmappings into many intermediate steps. There are\nmany pieces of research on CoT prompting strate-\ngies (Qiao et al., 2022) including prompt designing\nand process optimization. In this paper, we mainly\nfocus on CoT designing strategies.\nSimilar to demonstration selection, CoT design-\ning also considers CoT selection. Different from\nWei et al. (2022c) manually writing CoTs, Auto-\nCoT (Zhang et al., 2022b) used LLMs with Let's\nthink step by step to generate CoTs. In addi-\ntion, Fu et al. (2022) proposed a complexity-based\ndemonstration selection method. They selected\ndemonstrations with more reasoning steps for CoT\nprompting.\nAs input-output mappings are decomposed into\nstep-by-step reasoning, some researchers apply\nmulti-stage ICL for CoT prompting and design\nCoT demonstrations for each step. Multi-stage\nICL queries LLMs with different demonstrations in\neach reasoning step. Self-Ask (Press et al., 2022) al-\nlows LLMs to generate follow-up questions for the\ninput and ask themselves these questions. Then the\nquestions and intermediate answers will be added\nto CoTs. iCAP (Wang et al., 2022a) proposes a\ncontext-aware prompter that can dynamically ad-\njust contexts for each reasoning step. Least-to-\nMost Prompting (Zhou et al., 2022a) is a two-stage\nICL including question reduction and subquestion\nsolution. The first stage decomposes a complex\nquestion into subquestions; in the second stage,\nLLMs answer subquestions sequentially, and previ-\nously answered questions and generated answers\nwill be added into the context.\nXu et al. (2023b) fine-tuned small LMs on spe-\ncific task as plug-ins to generate pseudo reasoning\nsteps. Given an input-output pair (xi, yi), Super-\nICL regarded the prediction y′\niand confidence ci\nof small LMs for the input xias reasoning steps by\nconcatenating (xi, y′\ni, ci, yi).\n3Takeaway :(1) Demonstration selection\nstrategies improve the I', metadata={'page_label': '6', 'file_name': '2301.00234v3.A_Survey_on_In_context_Learning.pdf', 'file_path': 'prompt-engineering-papers/2301.00234v3.A_Survey_on_In_context_Learning.pdf', 'file_type': 'application/pdf', 'file_size': 4898135, 'creation_date': '2024-02-13', 'last_modified_date': '2024-02-08', 'last_accessed_date': '2024-02-13'}, doc_id='963d6900-8fd2-4b7e-bfd2-00d8f29f635e', filename='963d6900-8fd2-4b7e-bfd2-00d8f29f635e') is not in list
---------------------------------------------------------------------------
ExceptionInRunner Traceback (most recent call last)
Cell In[12], line 13
10 distribution = {simple: 0.5, reasoning: 0.25, multi_context: 0.25}
12 # generate testset
---> 13 testset = generator.generate_with_llamaindex_docs(
14 documents, test_size=TEST_SIZE, distributions=distribution
15 )
16 test_df = testset.to_pandas()
17 test_df.head()
File ~/miniforge3/envs/ragas-test/lib/python3.10/site-packages/ragas/testset/generator.py:128, in TestsetGenerator.generate_with_llamaindex_docs(self, documents, test_size, distributions, with_debugging_logs, is_async, raise_exceptions, run_config)
113 def generate_with_llamaindex_docs(
114 self,
115 documents: t.Sequence[LlamaindexDocument],
(...)
122 ):
123 # chunk documents and add to docstore
124 self.docstore.add_documents(
125 [Document.from_llamaindex_document(doc) for doc in documents]
126 )
--> 128 return self.generate(
129 test_size=test_size,
130 distributions=distributions,
131 with_debugging_logs=with_debugging_logs,
132 is_async=is_async,
133 run_config=run_config,
134 raise_exceptions=raise_exceptions,
135 )
File ~/miniforge3/envs/ragas-test/lib/python3.10/site-packages/ragas/testset/generator.py:246, in TestsetGenerator.generate(self, test_size, distributions, with_debugging_logs, is_async, raise_exceptions, run_config)
244 test_data_rows = exec.results()
245 if test_data_rows == []:
--> 246 raise ExceptionInRunner()
248 except ValueError as e:
249 raise e
ExceptionInRunner: The runner thread which was running the jobs raised an exeception. Read the traceback above to debug it. You can also pass `raise_exception=False` incase you want to show only a warning message instead.
Expected behavior
Method should run without error.