Skip to content

feat: retry + timeout + uniform errors for LLM calls (closes #28)#34

Merged
PRATHAMESH75 merged 2 commits into
mainfrom
feat/28-llm-retry-errors
Jun 21, 2026
Merged

feat: retry + timeout + uniform errors for LLM calls (closes #28)#34
PRATHAMESH75 merged 2 commits into
mainfrom
feat/28-llm-retry-errors

Conversation

@PRATHAMESH75

Copy link
Copy Markdown
Owner

Summary

Closes #28. Adds resilient outbound handling for LLM provider calls — retry with backoff, per-request timeout, and a single actionable error surface — for every LLMClient-based agent (report, ontology, tools).

Stacked on #32 (feat/20) → #31#30. Merge order: #30#31#32#28.
Note: this is the outbound provider direction (429/timeout to the LLM). Incoming HTTP rate limiting is already wired via flask-limiter on the routes.

Changes

  • _request_with_retries — retries transient provider errors (RateLimitError, APITimeoutError, APIConnectionError, InternalServerError) with exponential backoff; non-transient errors (bad request, auth) raise immediately (no pointless retries).
  • Per-request timeout passed to the OpenAI client (primary + fallback).
  • Order of resilience: primary model (with retries) → backup model (with retries, from Per-task model/provider config + fallback #15) → give up.
  • LLMError — one user-facing, actionable failure surface raised when everything is exhausted. Subclasses RuntimeError and embeds the original detail + model name, so existing except Exception catchers and the Per-task model/provider config + fallback #15 fallback tests keep working unchanged.
  • Config: LLM_MAX_RETRIES, LLM_RETRY_INITIAL_DELAY, LLM_RETRY_MAX_DELAY, LLM_TIMEOUT (documented in .env.example).

Acceptance criteria

  • Provider 429/timeout handled gracefully across all (LLMClient-based) agents
  • User-facing errors are clear and actionable (LLMError with retry/provider guidance)

Testing

  • New tests/test_llm_retry.py (5 tests): retry-then-succeed, exhausted→LLMError with actionable message, non-transient not retried, transient-on-primary→fallback succeeds, timeout class membership.
  • Full backend suite: 54 passed.

Scope note

The two raw-OpenAI generators (oasis_profile_generator, simulation_config_generator) keep their own existing retry loops; this PR centralizes resilience for the shared LLMClient path. Folding those onto LLMClient is a reasonable follow-up.

🤖 Generated with Claude Code

LLMClient issued a single attempt per call, so a provider 429 / timeout /
connection blip propagated as a raw SDK exception. Centralize resilient
outbound handling for every LLMClient-based agent (report, ontology, tools).

- _request_with_retries: retry transient provider errors (RateLimitError,
  APITimeoutError, APIConnectionError, InternalServerError) with exponential
  backoff; non-transient errors (bad request, auth) raise immediately.
- Per-request timeout passed to the OpenAI client (primary + fallback).
- Order: primary (with retries) -> backup model (with retries) -> give up.
- LLMError: a single, actionable, user-facing failure surface raised when all
  attempts are exhausted. Subclasses RuntimeError and embeds the original
  detail + model, so existing catchers and the fallback tests still work.
- Config: LLM_MAX_RETRIES, LLM_RETRY_INITIAL_DELAY, LLM_RETRY_MAX_DELAY,
  LLM_TIMEOUT; documented in .env.example.
- tests/test_llm_retry.py: retry-then-succeed, exhausted->LLMError with
  actionable message, non-transient not retried, transient-then-fallback,
  timeout class membership.

Note: HTTP-side rate limiting (incoming) is already wired via flask-limiter on
the routes; this covers the outbound provider direction. Stacked on #20/#15.

Co-Authored-By: PRATHAMESH75 <prathamesh290504@gmail.com>
@PRATHAMESH75 PRATHAMESH75 changed the base branch from feat/20-llm-response-cache to main June 21, 2026 03:43
@PRATHAMESH75 PRATHAMESH75 merged commit 055cc2a into main Jun 21, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Rate limiting & error handling on LLM calls

1 participant