BooleanOutputParser expected output value error

[ankur287]

System Info

Hi, I am using LLMChainFilter.from_llm(llm) but while running, I am getting this error:

ValueError: BooleanOutputParser expected output value to either be YES or NO. Received Yes, the context is relevant to the question as it provides information about the problem in the.

How do I resolve this error?

Langchain version: 0.0.308

Who can help?

@agola11

Information

• The official example notebooks/scripts
• My own modified scripts

Related Components

• LLMs/Chat Models
• Embedding Models
• Prompts / Prompt Templates / Prompt Selectors
• Output Parsers
• Document Loaders
• Vector Stores / Retrievers
• Memory
• Agents / Agent Executors
• Tools / Toolkits
• Chains
• Callbacks/Tracing
• Async

Reproduction

from langchain.retrievers import ContextualCompressionRetriever
from langchain.retrievers.document_compressors import LLMChainExtractor, LLMChainFilter

llm = SageMakerEndpointModel
_filter = LLMChainFilter.from_llm(llm)
compressor = LLMChainExtractor.from_llm(llm)
compression_retriever = ContextualCompressionRetriever(base_compressor=_filter, base_retriever=faiss_retriever)

compressed_docs = compression_retriever.get_relevant_documents("What did the president say about Ketanji Jackson Brown?")

Expected behavior

Get filtered docs

[dosubot]

🤖

Hello,

The error you're encountering is due to the output of your language model (LLM) not matching the expected values "YES" or "NO" that the BooleanOutputParser class is designed to handle.

The BooleanOutputParser class in the LangChain codebase is used to parse the output of a language model call to a boolean value. It takes the output of a language model, represented as a string, and converts it into a boolean value. The class has two string attributes, true_val and false_val, which represent the string values that should be parsed as True and False respectively. By default, these are set to "YES" and "NO".

Here's the relevant code that's causing the error:

cleaned_text = text.strip()
if cleaned_text.upper() not in (self.true_val.upper(), self.false_val.upper()):
    raise ValueError(
        f"BooleanOutputParser expected output value to either be "
        f"{self.true_val} or {self.false_val}. Received {cleaned_text}."
    )

This code is checking if the cleaned, uppercased output of your LLM (cleaned_text.upper()) is either "YES" or "NO". If it's not, it raises a ValueError. So, you need to ensure that your LLM's output matches these expectations.

To resolve this issue, you need to ensure that the output from your LLM is either "YES" or "NO". If your LLM is producing different outputs, you might need to adjust your LLM or create a custom output parser that can handle the specific outputs of your LLM.

In the context shared, the LLMChainFilter class uses an LLM to filter documents based on their relevance to a query. The from_llm class method creates an LLMChainFilter from a language model. If no prompt is provided, it uses a default prompt that expects a boolean output from the LLM. This is where the BooleanOutputParser is used, and it expects the output to be either "YES" or "NO".

I hope this helps! If you have any other questions, feel free to ask.

Best,
Dosu

Sources

• Issue: value error in BooleanOutputParser
• libs/langchain/langchain/output_parsers/boolean.py
• libs/langchain/langchain/retrievers/document_compressors/chain_filter.py
• libs/langchain/tests/unit_tests/output_parsers/test_boolean_parser.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[sudranga]

@ankur287 Looks like the output parser doesn't retry when the LLM output isn't as expected. We should make it more robust by retrying. I can try fixing this issue if that's ok with you.

[sudranga]

@agola11 Can you please assign this issue to me? I'm unable to do it myself.

[ankur287]

I fixed the issue temporarily by creating a rule that if "YES" in output just return "YES". However, we need to make this change as we cannot expect LLM to always say just yes or no. In my case model said, 'Yes answer is available in context'

[davidgxue]

@ankur287 Do you mind posting how you were able to get around this issue in detail since LangChain itself hasn't really fixed this?

If not is there an issue tracking this problem? I have found a quick workaround by implementing my own boolean output parser to default to YES along with checking if YES/NO is in the output instead of strict matching. I am happy to make a PR to address this problem.

[ankur287]

I posted above how I fixed it. See my last comment