[Bug]: LLMSynonymRetriever always giving Empty Response...

[Johnbathappully]

Bug Description

#not working template wise LLMSynonymRetriever
from llama_index.core.indices.property_graph import LLMSynonymRetriever

import nest_asyncio
nest_asyncio.apply()

Define the prompt for generating synonyms or related keywords

prompt = (
"Given some initial query, generate synonyms or related keywords up to {max_keywords} in total, "
"considering possible cases of capitalization, pluralization, common expressions, etc.\n"
"Provide all synonyms/keywords separated by '^' symbols: 'keyword1^keyword2^...'\n"
"Note, result should be in one-line, separated by '^' symbols."
"----\n"
"QUERY: {query_str}\n"
"----\n"
"KEYWORDS: "
)

Define the output parsing function

def parse_fn(output: str) -> list[str]:
matches = output.strip().split("^")
return [x.strip().capitalize() for x in matches if x.strip()]

Initialize the synonym retriever

synonym_retriever = LLMSynonymRetriever(
graph_store,
llm=llm,
# Include source chunk text with retrieved paths
include_text=False,
synonym_prompt=prompt,
output_parsing_fn=parse_fn,
max_keywords=10,
# The depth of relations to follow after node retrieval
path_depth=1,
)

from llama_index.core import PropertyGraphIndex

Load the existing property graph index

index = PropertyGraphIndex.from_existing(
property_graph_store=graph_store
)

Create the retriever using the synonym retriever

retriever = index.as_retriever(sub_retrievers=[synonym_retriever])

Create the query engine using the retriever

query_engine = index.as_query_engine(
sub_retrievers=[retriever]
)

Perform a query using the query engine

response = query_engine.query("Insurbridge Connects to")

Print the response

print(response)

Version

llama-cloud 0.0.9 llama-index 0.10.55 llama-index-agent-openai 0.2.8 llama-index-cli 0.1.12 llama-index-core 0.10.55 llama-index-embeddings-openai 0.1.10 llama-index-graph-stores-neo4j 0.2.7 llama-index-indices-managed-llama-cloud 0.2.5 llama-index-legacy 0.9.48 llama-index-llms-openai 0.1.25 llama-index-multi-modal-llms-openai 0.1.7 llama-index-program-openai 0.1.6 llama-index-question-gen-openai 0.1.3 llama-index-readers-file 0.1.30 llama-index-readers-llama-parse 0.1.6 llama-parse 0.4.7

Steps to Reproduce

pls note below code without prompt /output prsing function etc..works
import nest_asyncio
nest_asyncio.apply()

from llama_index.core.indices.property_graph import (
PGRetriever,
VectorContextRetriever,
LLMSynonymRetriever,
)

sub_retrievers = [
LLMSynonymRetriever(graph_store),
]

retriever = PGRetriever(sub_retrievers=sub_retrievers)

nodes = retriever.retrieve("Insurbridge Connects to")

for node in nodes:
print(node.text)

from llama_index.core import PropertyGraphIndex
index = PropertyGraphIndex.from_existing(
property_graph_store=graph_store
)

query_engine = index.as_query_engine(sub_retrievers=[retriever], llm=llm)

response = query_engine.query("Insurbridge Connects to")
print(str(response))

the same graphstore which was ingested earlier is not able to retrive when moving towards this below method prompt base llmsynomretriever
prompt = (
"Given some initial query, generate synonyms or related keywords up to {max_keywords} in total, "
"considering possible cases of capitalization, pluralization, common expressions, etc.\n"
"Provide all synonyms/keywords separated by '^' symbols: 'keyword1^keyword2^...'\n"
"Note, result should be in one-line, separated by '^' symbols."
"----\n"
"QUERY: {query_str}\n"
"----\n"
"KEYWORDS: "
)

def parse_fn(output: str) -> list[str]:
matches = output.strip().split("^")
return [x.strip().capitalize() for x in matches if x.strip()]

synonym_retriever = LLMSynonymRetriever(
graph_store,
llm=llm,
include_text=False,
synonym_prompt=prompt,
output_parsing_fn=parse_fn,
max_keywords=10,
path_depth=1,
)

Relevant Logs/Tracbacks

No response

[dosubot]

The LLMSynonymRetriever is likely giving an empty response because the output_parsing_fn provided is returning an empty list or the response from the language model does not match the expected format for parsing.

In the _parse_llm_output method, if self._output_parsing_fn is provided, it is used to parse the output. If this function does not correctly parse the output or returns an empty list, the result will be empty.

Here is the relevant code snippet:

def _parse_llm_output(self, output: str) -> List[str]:
    if self._output_parsing_fn:
        matches = self._output_parsing_fn(output)
    else:
        matches = output.strip().split("^")

    # capitalize to normalize with ingestion
    return [x.strip().capitalize() for x in matches if x.strip()]

Ensure that the output_parsing_fn correctly parses the output from the language model and returns a non-empty list of synonyms. If the function is not provided, ensure that the output from the language model is in the expected format (i.e., synonyms separated by ^ symbols).

For example, the default prompt and parsing function for LLMSynonymRetriever are as follows:

prompt = (
    "Given some initial query, generate synonyms or related keywords up to {max_keywords} in total, "
    "considering possible cases of capitalization, pluralization, common expressions, etc.\n"
    "Provide all synonyms/keywords separated by '^' symbols: 'keyword1^keyword2^...'\n"
    "Note, result should be in one-line, separated by '^' symbols."
    "----\n"
    "QUERY: {query_str}\n"
    "----\n"
    "KEYWORDS: "
)

def parse_fn(output: str) -> list[str]:
    matches = output.strip().split("^")

    # capitalize to normalize with ingestion
    return [x.strip().capitalize() for x in matches if x.strip()]

Make sure that the language model's response adheres to this format and that the parse_fn is correctly implemented to handle the output ^[1][2].

To continue talking to Dosu, mention @dosu.

[logan-markewich]

The synonym retriever generates keywords and synonyms based on your query. If none of those match entities in your graph, then nothing is retrieved.

Usually this retriever alone is not enough, I always combine it with a vector retriever

Closing as a duplicate to your other issue