Gemini 2.5 503 "High Demand" errors - recommended handling strategy?

[LiveNathan]

I've been hitting frequent 503 errors from Google Gemini 2.5 models (gemini-2.5-flash-lite in my case, but reports online indicate gemini-2.5-flash and gemini-2.5-pro are also affected) over the past two days. I'm looking for advice on the best way to handle this within Embabel, and whether any framework-level improvements could help.

Environment

• Embabel version: 0.5.0
• JDK 26
• Model: gemini-2.5-flash-lite

Error

The root cause coming from com.google.genai.errors.ServerException:

Caused by: com.google.genai.errors.ServerException: 503 . This model is currently experiencing high demand. Spikes in demand are usually temporary. Please try again later.
	at com.google.genai.errors.ApiException.throwFromResponse(ApiException.java:96)
	at com.google.genai.HttpApiResponse.getBody(HttpApiResponse.java:37)
	...

Full stack trace visible in logs via:

java.lang.RuntimeException: Failed to generate content
	at org.springframework.ai.google.genai.GoogleGenAiChatModel.getContentResponse(GoogleGenAiChatModel.java:965)
	...
	at com.embabel.agent.spi.support.springai.SpringAiLlmMessageSender.call(SpringAiLlmMessageSender.kt:71)
	at com.embabel.agent.spi.loop.support.DefaultToolLoop.execute(DefaultToolLoop.kt:117)

Observed behavior

Embabel's RetryProperties does retry on failure (Retry count: 1 logged from c.e.agent.spi.common.RetryProperties), and eventually succeeds - but the tool loop is taking ~47-55 seconds per call as a result:

[task-1] tool loop completed in 55268ms iterations=1 replan=false
[task-2] tool loop completed in 47626ms iterations=1 replan=false

What I've done

I wrote a custom ResilientLlmExecutor fallback wrapper that catches these errors and retries with exponential backoff (or falls back to an alternate model). It's working, but I'm not sure if this is the right layer to be solving this, or if there's a better hook Embabel provides.

application.properties

I have also attempted to tune the retry behavior:

embabel.models.default-llm=gemini-2.5-flash
embabel.agent.platform.models.googlegenai.backoff-multiplier=2
embabel.agent.platform.models.googlegenai.max-attempts=2
embabel.agent.platform.models.googlegenai.backoff-millis=500
embabel.agent.platform.models.googlegenai.backoff-max-interval=1000
embabel.agent.platform.llm-operations.prompts.default-timeout=300s
embabel.agent.platform.llm-operations.data-binding.max-attempts=3

Context: This is a widespread Google-side issue

For what it's worth, this appears to be a persistent infrastructure capacity problem on Google's end - there are many reports across [Google's developer forums](https://discuss.ai.google.dev/t/503-this-model-is-currently-experiencing-high-demand-spikes-in-demand-are-usually-temporary-please-try-again-later/138664) and the [googleapis Python SDK repo](googleapis/python-genai#1373). One particularly concerning finding: Google appears to be [counting failed 503 retries toward the daily rate limit quota](https://www.reddit.com/r/GeminiAI/comments/1ritany/google_is_counting_failed_requests_because_of/), meaning aggressive retry loops can accelerate quota exhaustion.

Questions

1. Is RetryProperties configurable per-provider? I'd like to tune the backoff specifically for the Google GenAI provider (e.g., longer initial delay, exponential backoff, fewer total attempts to avoid quota burn).
2. Does Embabel have - or plan to have - a model fallback/circuit-breaker pattern? For example, if gemini-2.5-flash-lite returns 503, automatically retry with gemini-2.5-flash before giving up.
3. Is my ResilientLlmExecutor approach (wrapping at the LlmOperations or ChatModel level) the right place for this, or is there a more idiomatic Embabel extension point?
4. Should 503 from the provider be surfaced differently in Embabel's event/observability model vs. a hard model error?

Happy to share the ResilientLlmExecutor implementation if useful as a reference or potential contribution.