Support service_tier (Flex/Priority tiers) in ChatGoogleGenerativeAI

[matbdev]

Package (Required)

• langchain-google-genai
• langchain-google-vertexai
• langchain-google-community
• Other / not sure / general

Checked other resources

• I added a descriptive title to this issue
• I searched the LangChain documentation and API reference (linked above)
• I used the GitHub search to find a similar issue and didn't find it
• I am sure this is a feature request and not a bug report or question

Feature Description

Google recently introduced the Flex inference tier for the Gemini API, which offers a 50% cost reduction in exchange for variable latency. This is highly desirable for latency-tolerant workloads like asynchronous data cataloging or batch processing.

In the official google-genai SDK, this is implemented by passing config={'service_tier': 'flex'} to the generate_content request.

Currently, in langchain-google-genai, there seems to be no native way to propagate the service_tier parameter down to the underlying GenerateContentConfig.

Use Case

I am building a batch data cataloging project using gemini-3-flash-preview with structured outputs. Since this runs in the background and doesn't require real-time latency, the 50% cost savings from the Flex tier is vital for optimizing the budget.

Proposed Solution

Allow service_tier to be defined during ChatGoogleGenerativeAI initialization or captured via .bind(), and ensure it gets properly mapped to the GenerateContentConfig object inside the _build_request_config method.

Alternatives Considered

The only current workaround is to bypass LangChain entirely and use the official google-genai SDK directly. However, this is not ideal as it breaks existing LCEL chains, callback handlers, and the highly convenient .with_structured_output() abstraction, requiring a significant rewrite of the application logic.

Code Example of Attempted Usage

Here is how I intuitively tried to implement it using LangChain, but the configuration isn't passed down to the Google API:

from langchain_google_genai import ChatGoogleGenerativeAI
from langchain.callbacks.base import AsyncCallbackHandler

class VerboseCallbackHandler(AsyncCallbackHandler):
    async def on_llm_error(self, error, **kwargs):
        print(f"  [x] [LangChain] Gemini Request Failed: {type(error).__name__}: {error}", flush=True)

# Initialize the model
llm = ChatGoogleGenerativeAI(
    model="gemini-3-flash-preview",
    temperature=0,
    callbacks=[VerboseCallbackHandler()],
    timeout=1800,
    max_retries=3
)

# Attempt to bind the Flex service tier
llm_flex = llm.bind(service_tier="flex") # or "FLEX"

# Use with structured output
structured_llm = llm_flex.with_structured_output(pydantic_schema, include_raw=True)

Additional Context

A similar issue was recently opened for the JS ecosystem: langchain-ai/langchainjs#10655
Link for the Google page that talks about Flex: https://ai.google.dev/gemini-api/docs/flex-inference

[saalihou]

For anyone looking for a patch solution while this is getting implemented, just extend the ChatGoogle class like below and use this class instead:

/**
 * Extends ChatGoogle to inject the `service_tier` parameter into the request body.
 * LangChain doesn't support this yet, but it's a top-level field in the Gemini API
 * alongside generationConfig/contents. Since _generate spreads invocationParams()
 * into the body, we can sneak it in here.
 * Values: 'standard' (default), 'priority' (low-latency, ~75-100% premium), 'flex' (50% discount, sheddable)
 */
class ChatGoogleWithServiceTier extends ChatGoogle {
  private serviceTier?: 'standard' | 'priority' | 'flex';

  constructor(fields: ConstructorParameters<typeof ChatGoogle>[0] & { serviceTier?: 'standard' | 'priority' | 'flex' }) {
    const { serviceTier, ...rest } = fields;
    super(rest);
    this.serviceTier = serviceTier;
  }

  invocationParams(options: Parameters<ChatGoogle['invocationParams']>[0]): ReturnType<ChatGoogle['invocationParams']> {
    const params = super.invocationParams(options);
    if (this.serviceTier && this.serviceTier !== 'standard') {
      return { ...params, serviceTier: this.serviceTier } as ReturnType<ChatGoogle['invocationParams']>;
    }
    return params;
  }
}

[josephbartlett-Cydea]

It looks likesupport for service_tier and custom headers was recently merged in the JS repository in langchainjs#10704.

Are there plans for this to be implemented in Python for ChatGoogleGenerativeAI when vertexai == True?

[ycrouin]

hi 👋 I use this workaround in the meantime:

model = ChatGoogleGenerativeAI(
    model="gemini-3.1-flash-lite",
    project=PROJECT_ID,
    labels=LABELS,
    additional_headers={
        "X-Vertex-AI-LLM-Request-Type": "shared",
        "X-Vertex-AI-LLM-Shared-Request-Type": "flex",
    },
)

you can also pass them at invoke time like so:

model.invoke(
    messages,
    http_options={
        "headers": {
            "X-Vertex-AI-LLM-Request-Type": "shared",
            "X-Vertex-AI-LLM-Shared-Request-Type": "flex",
        }
    },
)

Vertex seems to comply as it returns "trafficType": "ON_DEMAND_FLEX" in the response metadata.

[josephbartlett-Cydea]

Thanks Yoann (@ycrouin) can confirm this also worked for me!

My priority requests are being downgraded to standard but this appears to be a Vertex issue.

[axiom-of-choice]

Measured this today with langchain-google-genai 4.3.3 and google-genai 2.17.0, because the two backends do not take the tier over the same channel.

Vertex AI, gemini-3.5-flash, location global, same project and model in every row:

Channel	Value	usageMetadata.trafficType
none (baseline)	—	ON_DEMAND
body serviceTier	flex	400 Invalid value at 'service_tier'
body serviceTier	SERVICE_TIER_FLEX	ON_DEMAND
header X-Vertex-AI-LLM-Shared-Request-Type	flex	ON_DEMAND_FLEX
header, same	priority	ON_DEMAND_PRIORITY

Row three is the one to watch. Vertex accepts SERVICE_TIER_FLEX in the body, returns 200, and bills standard. Yoann (@ycrouin)'s header workaround is the one that actually applies the tier.

Gemini Developer API, same model, the opposite arrangement: body serviceTier: 'flex' comes back as usageMetadata.serviceTier: flex. No header needed there. priority came back standard on my key, which looks like an eligibility limit rather than a channel problem.

Flex availability is per model, not just per project. Served on gemini-3.1-pro-preview, gemini-3.5-flash and gemini-3.1-flash-lite; the 2.5 family returns 400 Flex API is not supported for model: .... Useful signal, since an unavailable tier errors instead of silently billing standard.

Separately, service_tier on the constructor was not just unsupported, it was dropped: not being a field, it landed in model_kwargs, which ChatGoogleGenerativeAI never reads when building a request. Per-request invoke kwargs did reach GenerateContentConfig, which is why the two paths behaved differently.

#1937 adds the field and sends the body field on the Developer API, the header on Vertex, matching the surface merged in langchain-ai/langchainjs#10704.

For context on the SDK side: googleapis/python-genai#2433 describes the Vertex body-vs-header mismatch and was closed by the stale bot rather than fixed, and googleapis/python-genai#2445, which moves the translation into the SDK, has been open since May. If that lands, this change stays correct: it clears the body field on Vertex, so there is nothing left for the SDK to translate.