fix: migrate inference client to httpx2 - #6826
Conversation
1e5741e to
07f9b59
Compare
chenghao-mou
changed the title
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
| except httpx2.TimeoutException as e: | ||
| # Only the request call runs inside the openai client's error mapping, so a | ||
| # timeout waiting on the stream body arrives as the raw httpx exception. | ||
| # timeout waiting on the stream body arrives as the raw httpx2 exception. | ||
| raise APITimeoutError(retryable=retryable) from e |
There was a problem hiding this comment.
🟡 Stalled responses from OpenAI-compatible providers are now reported as connection failures instead of timeouts
A stalled response body is only recognised as a timeout when it comes from the new HTTP library (except httpx2.TimeoutException at livekit-agents/livekit/agents/inference/llm.py:489), so providers still using the old HTTP library report a generic connection failure instead of a timeout.
Impact: Users and metrics of OpenAI-plugin-backed models (OpenAI, Groq, Cerebras, xAI, DeepSeek, Google AI Platform, …) see mislabelled connection errors when a provider goes quiet mid-stream.
Mechanism: shared LLMStream._run is reused by plugins that build httpx (v1) clients
livekit.agents.inference.llm.LLMStream is subclassed by livekit-plugins-openai (livekit-plugins/livekit-plugins-openai/livekit/plugins/openai/llm.py:1041) and used by livekit-plugins-google (livekit-plugins/livekit-plugins-google/livekit/plugins/google/aiplatform_llm.py:41); neither overrides _run. Those plugins construct their openai.AsyncClient with a plain httpx.AsyncClient (livekit-plugins/livekit-plugins-openai/livekit/plugins/openai/llm.py:165), so a read timeout while iterating the SSE body raises httpx.ReadTimeout, not httpx2.ReadTimeout. Since the request itself is the only part covered by the openai client's error mapping, that raw exception previously matched except httpx.TimeoutException and produced APITimeoutError; now it falls through to the trailing except Exception and produces APIConnectionError. Fix by catching both exception families (import httpx defensively, e.g. except (httpx2.TimeoutException, httpx.TimeoutException)).
Prompt for agents
In livekit-agents/livekit/agents/inference/llm.py, LLMStream._run now only catches httpx2.TimeoutException for stream-body stalls. This class is inherited by livekit-plugins-openai's LLMStream and used by livekit-plugins-google's aiplatform_llm, both of which pass openai clients built on the original httpx (v1) library. For those callers a mid-stream read timeout raises httpx.TimeoutException, which now falls into the generic `except Exception` branch and is surfaced as APIConnectionError instead of APITimeoutError. Make the handler tolerant of both HTTP client libraries, e.g. by importing httpx lazily/optionally and catching a tuple of both timeout exception types (falling back gracefully if httpx is not installed, since it is no longer a declared dependency of livekit-agents).
Was this helpful? React with 👍 or 👎 to provide feedback.
|
the plugins still depend on the httpx, should we add httpx to the agent instead? |
2 participants
livekit-agentsconfigures its inference HTTP transport directly. OpenAI 3uses HTTPX2, so standalone installs should not depend on HTTPX arriving
transitively.
Use OpenAI's supported HTTPX2 factory and preserve the existing timeout,
retry, and connection limits. Require OpenAI 2.47, the first 2.x release that
accepts HTTPX2 clients.
Addresses AGT-3286
Initial prompt and agent context
Model: GPT-5.6