Skip to content

fix: enable LLM retry without fallbacks and surface SDK retries - #7

Merged
ngoclam9415 merged 1 commit into
developfrom
fix/llm-retry-connection-error-and-star-resilience
Apr 19, 2026
Merged

fix: enable LLM retry without fallbacks and surface SDK retries#7
ngoclam9415 merged 1 commit into
developfrom
fix/llm-retry-connection-error-and-star-resilience

Conversation

@ngoclam9415

Copy link
Copy Markdown
Contributor

Summary

Fixes a bug where an Azure APIConnectionError exited the STAR loop after ~7.5 minutes with no visible retry attempts, despite the LLMCaller having retry logic. Three converging bugs were responsible:

  1. LLMCaller retry was opt-in via fallback_providers — primary provider had no retry path. Now retry/backoff always runs; failover stays gated on fallback_providers.
  2. _is_transient_error did not classify "Connection error." as transient (no matching keyword) and could not see openai.APIConnectionError buried in __cause__. Added "connection" to _TRANSIENT_KEYWORDS and a __cause__-chain walk that detects OpenAI SDK exception class names.
  3. STAR loop exited on any exception. Added one bounded retry per iteration for transient LLM errors (sync + async paths), with full tracebacks via exc_info=True on all error sites.

Observability improvements

  • New make_logging_http_client wired into OpenAI/Azure/Moonshot providers — emits LLM HTTP request per attempt, surfacing SDK-internal retries that were previously silent (those took ~7.5 min in the original incident).
  • New structured events: llm_retries_exhausted, STAR transient retry warnings, OpenAI-compatible API connection error with traceback.
  • All exception logs gained error_type + exc_info=True.

Files changed

  • dana/core/llm/llm_caller.py — always-retry path, expanded transient detection, public is_transient_llm_error
  • dana/common/llm/providers/openai_compatible_base.py — APIConnectionError handling, logging http client helper
  • dana/common/llm/providers/{azure,openai,moonshot}.py — wire logging http client
  • dana/core/agent/base_star_agent.py — STAR transient retry (sync+async), tracebacks
  • dana/core/agent/star_agent_streaming.py — traceback + transient flag in error log
  • tests/unit/core/test_llm_caller_failover.py — regression tests for connection-error classification, __cause__ walk, no-fallback retry

Test plan

  • pytest tests/unit/core/test_llm_caller_failover.py — 24 passed
  • pytest tests/unit — 1055 passed, 20 skipped
  • pytest tests/regression — 44 passed, 18 skipped
  • tests/live/llm/test_multi_modal.py against real API keys — all providers (anthropic, openai, azure, gemini, gemini_openai, anthropic_like, moonshot) returned OK on every supported modality
  • Verify in a real Azure flake scenario that retry logs appear and the STAR loop survives

Bug: An Azure APIConnectionError exited the STAR loop after ~7.5min
without any visible retry, despite the LLMCaller having retry logic.

Three converging bugs were fixed:

1. LLMCaller retry was opt-in via fallback_providers, so the primary
   provider had no retry path. Now retry/backoff always runs; failover
   stays gated on fallback_providers.

2. _is_transient_error did not classify "Connection error." as
   transient (no matching keyword) and could not see openai.APIConnectionError
   buried in __cause__. Added "connection" to the keyword list and an
   exception-class-name walk over the __cause__ chain.

3. STAR loop exited on any exception. Added one bounded retry per
   iteration for transient LLM errors (sync + async paths), with full
   tracebacks via exc_info=True on all error sites.

Observability: OpenAI/Azure/Moonshot providers now use a logging
httpx.AsyncClient that emits "LLM HTTP request" per attempt, surfacing
SDK-internal retries that were previously silent. New structured events:
llm_retries_exhausted, STAR transient retry warnings, and an
APIConnectionError-specific log line with traceback.

Tests: updated test_no_fallbacks_exception_propagates (split into
transient/permanent variants), added regression tests for the
connection-error classification and __cause__ chain detection. Full
unit + regression suites pass.
@ngoclam9415
ngoclam9415 merged commit cd00589 into develop Apr 19, 2026
1 check passed
@TheVinhLuong102
TheVinhLuong102 deleted the fix/llm-retry-connection-error-and-star-resilience branch May 9, 2026 03:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant