-
Notifications
You must be signed in to change notification settings - Fork 302
Description
When using with_structured_output()
, occasionally the return value becomes None
unexpectedly. This happens even with function calling that normally succeeds, where the Gemini API sometimes returns MALFORMED_FUNCTION_CALL
responses. This silent failure occurs without any error indication, making it extremely difficult to diagnose the root cause during development.
Through extensive debugging of this intermittent issue, I confirmed that the Gemini API sometimes probabilistically returns MALFORMED_FUNCTION_CALL
responses. The investigation revealed that these responses have the following structure:
{
"content": "",
"response_metadata": {
"finish_reason": "MALFORMED_FUNCTION_CALL",
"finish_message": "Malformed function call: print(default_api.Output(...",
"tool_calls": [],
"invalid_tool_calls": [],
...
}
...
}
When with_structured_output()
is used, PydanticToolsParser
and JsonOutputKeyToolsParser
are connected to the chain. These parsers return None
or empty lists when tool_calls
is empty, without detecting the underlying MALFORMED_FUNCTION_CALL
error.
- langchain-ai/langchain@langcha - libs/core/langchain_core/output_parsers/openai_tools.py#L328
- langchain-ai/langchain@langcha - libs/core/langchain_core/output_parsers/openai_tools.py#L256-L258
When using with_structured_output(..., method="json_mode")
, PydanticOutputParser
and JsonOutputParser
are used instead. These parsers properly throw exceptions when content parsing fails.
However, json_mode
doesn't support some JSON Schema features, so developers often must use the default method despite its error handling issues.
Suggestion: Throw an error when MALFORMED_FUNCTION_CALL
occurs instead of returning None
The implementation could be as simple as using a custom parser wrapper, for example:
class MalformedFunctionCallError(ChatGoogleGenerativeAIError):
pass
class StructuredOutputParserWithErrorHandling:
def __init__(self, parser):
self.parser = parser
def parse(self, ai_message):
if (hasattr(ai_message, 'response_metadata') and
ai_message.response_metadata.get('finish_reason') == 'MALFORMED_FUNCTION_CALL'):
finish_message = ai_message.response_metadata.get('finish_message', 'Unknown error')
raise MalformedFunctionCallError(finish_message)
return self.parser.parse(ai_message)
While the implementation seems not difficult, it would affect structured output behavior overall, so I haven't prepared a Pull Request and would like to wait for discussion from maintainers.