[Feature]: ChatLiteLLMRouter to support json_schema

[eilam-stream]

The Feature

Currently ChatLiteLLMRouter does not support json_schema method which is the gold standard for Structured Output using OpenAI.

Langchain's ChatOpenAI supports chat.with_structured_output(pydantic_model, method="json_schema") but if chat is ChatLiteLLMRouter the following error is received:

langchain_core/language_models/chat_models.py", line 1303, in with_structured_output
    raise ValueError(msg)
ValueError: Received unsupported arguments {'method': 'json_schema'}

Motivation, pitch

Langchain is a standard in the industry and supporting this functionality will not just improve user's reliability since Structured Output using json_mode, as opposed to json_schema, is frequently faulty and causes issues in production.

Are you a ML Ops Team?

No

Twitter / LinkedIn details

No response

[krrishdholakia]

Hi @eilam-stream litellm supports json_schema - https://docs.litellm.ai/docs/completion/json_mode#2-check-if-model-supports-json_schema.

I believe chatlitellmrouter is a community langfuse integration - we would welcome any PR to https://github.com/langchain-ai/langchain

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

[SlonSky]

@eilam-stream,

We at sembly.ai encountered a similar issue.

The reason is that ChatLiteLLM provides only default implementation for with_structured_output that is a tool-calling output.

You can either implement a custom wrapper that mimics the native Langchain OpenAI logic for with_structured_output,
or use .bind() to leverage LiteLLM's structured output support via completions.

from langchain_litellm import ChatLiteLLM
from langchain_core.messages import HumanMessage, SystemMessage
from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder
from langchain_core.output_parsers import PydanticOutputParser
from pydantic import BaseModel, Field


class LLMModel(BaseModel):
    model: str = Field(description="Specific model name")

messages = [HumanMessage(content="What model are you?")]


def create_json_object_chain(model):
    parser = PydanticOutputParser(pydantic_object=model)

    return (
        ChatPromptTemplate.from_messages(
            [
                SystemMessage(content="Respond in JSON format {format_instructions}"),
                MessagesPlaceholder(variable_name="messages"),
            ]
        ).partial(format_instructions=parser.get_format_instructions())
        | ChatLiteLLM(model="gpt-4.1-nano").bind(
            response_format={"type": "json_object"}
        )
        | parser
    )


def create_json_schema_chain(model):
    parser = PydanticOutputParser(pydantic_object=model)

    return (
        ChatLiteLLM(model="gpt-4.1-nano").bind(
            # Option 1: Explicit specification
            response_format={
                "type": "json_schema",
                "json_schema": {
                    "name": model.__name__,
                    "schema": model.model_json_schema(),
                },
            }
            # Option 2: Auto spec with Pydantic model
            # response_format=Model
        )
        | parser
    )


result_json_object = create_json_object_chain(LLMModel).invoke(messages)
print(f"Method: json_object; Result: {result_json_object}")
# >> Method: json_object; Result: model='GPT-4'


result_json_schema = create_json_schema_chain(LLMModel).invoke(messages)
print(f"Method: json_schema; Result: {result_json_schema}")
# >> Method: json_schema; Result: model='GPT-4'