Progress is stuck (or maybe too long) for generate_with_langchain_docs !!!

[noaft]

🚨 Bug Report: progress is stuck (or maybe too long) for generate_with_langchain_docs

• I have checked the documentation and related resources and couldn't resolve my bug.

---

📄 Describe the bug

The execution of generate_with_langchain_docs() is stuck at 50% with no progress for over an hour. No error is thrown, and CPU usage is minimal. It seems like the generation process either hangs or is too long for practical usage.

---

🐍 Python version

Python 3.132.2

💻 Ragas version

Ragas 0.3.0

---

📌 Code to Reproduce

from ragas.testset import TestsetGenerator
from pandas import DataFrame, concat
from PDF_load.pdf_loader import get_content
from LLM.settings import generator_llm, generator_embeddings
import pathlib
import asyncio

# Load PDF content
docs = get_content()

# Create generator
generator = TestsetGenerator(
    llm=generator_llm, embedding_model=generator_embeddings
)

output_dir = pathlib.Path("Output")
output_dir.mkdir(exist_ok=True)

async def run():
    docs = get_content()
    for item in docs:
        file_name = item.get("file_name")
        doc = item.get("doc")

        dataset = generator.generate_with_langchain_docs(doc, testset_size=10)
        df = dataset.to_pandas()

        output_path = output_dir / f"{file_name.split('.')[0]}.csv"
        df.to_csv(output_path, index=False)

asyncio.run(run())

---

Generating Scenarios: 50%|████████████████ | 1/2 [27:14<27:14, 1634.71s/it]

[noaft]

Same code but different runtime:
Generating Scenarios: 50%|██▌ | 1/2 [08:14<08:14, 494.01s/it]
I think have some problem with google api (maybe).

[noaft]

But now:
Generating Scenarios: 50%|██ | 1/2 [21:19<21:19, 1279.25s/it]
Can someone tell me what happend? 🥲 🥲 🥲 🥲 🥲

[ManasTrip]

Facing the same issue, don't know how long generating scenarios is going to take for 200 docs
Generating Scenarios: 0%| | 0/3 [00:00<?, ?it/s]

Also is there a intermediate step to resolve this, i am already passing a generator llm, embedding model, as well using default transforms but this still keeps happening . How do i prevent it from using CustomNode Filter
Node 7fa7a57a-f7c9-47d5-9e16-4679c8750eb2 does not have a summary. Skipping filtering.

[devsosin]

MultiHopAbstractQuerySynthesizer this makes the problem

from ragas.testset.synthesizers import MultiHopAbstractQuerySynthesizer

It sends to llm with exactly same prompt infinitely.
the one is concept_combination_prompt
and the other is themes_personas_matching_prompt

the default_query_distributor function is here (which is used by generator function as query_distributor parameter)

def default_query_distribution(
    llm: BaseRagasLLM, kg: t.Optional[KnowledgeGraph] = None
) -> QueryDistribution:
    """ """
    default_queries = [
        SingleHopSpecificQuerySynthesizer(llm=llm),
        MultiHopAbstractQuerySynthesizer(llm=llm),
        MultiHopSpecificQuerySynthesizer(llm=llm),
    ]
    if kg is not None:
        available_queries = []
        for query in default_queries:
            if query.get_node_clusters(kg):
                available_queries.append(query)
    else:
        available_queries = default_queries

    return [(query, 1 / len(available_queries)) for query in available_queries]

it makes [(single_query, 0.5), (multi_hop_abstrct_query, 0.5)]
and single_query takes not that much time (about 64s)

for i, (scenario, _) in enumerate(query_distribution):
    # print(scenario)
    exec.submit(
        scenario.generate_scenarios,
        n=splits[i],
        knowledge_graph=self.knowledge_graph,
        persona_list=self.persona_list[:num_personas],
        callbacks=scenario_generation_grp,
    )

try:
    scenario_sample_list: t.List[t.List[BaseScenario]] = exec.results()
except Exception as e:
    scenario_generation_rm.on_chain_error(e)
    raise e
else:
    scenario_generation_rm.on_chain_end(
        outputs={"scenario_sample_list": scenario_sample_list}
    )

we have to trace back of the exec.results() function i think
generate_scenarios makes the problem i think (which send the prompt to llm service)

[noaft]

I encountered a problem when trying to evaluate multi-hop questions using Ragas.

Single-hop questions → work as expected.

Multi-hop questions → failing (so slow) :<.

This makes me think the issue might come from the generation or evaluation process for multi-hop rather than the base setup itself.

[anistark]

This is fixed on recent versions. Please update to latest version and try again.

Feel free to reopen if still happening.

[noaft]

tks for support, i will test.