[Bug]: APIConnectionError with custom endpoint ~80% failure rate while direct Python SDK works 100%

[shimuhua]

Bug Description

Hermes Agent version: v0.10.0

install : curl -fsSL https://raw.githubusercontent.com/NousResearch/hermes-agent/main/scripts/install.sh | bash

=========== Issue ==================

1. When executing: hermes chat -q "hello" --toolsets browser, it occasionally succeeds, but the success rate is extremely low.

⚠️ API call failed (attempt 1/3): APIConnectionError
🔌 Provider: custom Model: minimax-m25
🌐 Endpoint: http://maas.gd.chinamobile.com:36007/ai/uifm/open/v1
📝 Error: Connection error.
⏱️ Elapsed: 2.49s Context: 2 msgs, ~311 tokens
⏳ Retrying in 2.7s (attempt 1/3)...
⚠️ API call failed (attempt 2/3): APIConnectionError
🔌 Provider: custom Model: minimax-m25
🌐 Endpoint: my base_url
📝 Error: Connection error.
⏱️ Elapsed: 7.11s Context: 2 msgs, ~311 tokens
⏳ Retrying in 4.4s (attempt 2/3)...
⚠️ API call failed (attempt 3/3): APIConnectionError
🔌 Provider: custom Model: minimax-m25
🌐 Endpoint: my base_url
📝 Error: Connection error.
⏱️ Elapsed: 13.41s Context: 2 msgs, ~311 tokens
🔁 Transient APIConnectionError on custom — rebuilt client, waiting 6s before one last primary attempt.
⚠️ API call failed (attempt 1/3): APIConnectionError
🔌 Provider: custom Model: minimax-m25
🌐 Endpoint: my base_url
📝 Error: Connection error.
⏱️ Elapsed: 21.25s Context: 2 msgs, ~311 tokens
⏳ Retrying in 2.5s (attempt 1/3)...
⚠️ API call failed (attempt 2/3): APIConnectionError
🔌 Provider: custom Model: minimax-m25
🌐 Endpoint: my base_url
📝 Error: Connection error.
⏱️ Elapsed: 25.63s Context: 2 msgs, ~311 tokens
⏳ Retrying in 4.0s (attempt 2/3)...
─ ⚕ Hermes ─────────────────────────────────────────────────────────────────────────────────────────────────────────

 Hello! I'm here to help you navigate the web and get things done. I can browse websites, fill forms, click
 buttons, scroll pages, and more.

 What can I help you with today?

2. When executing my LLM request via a Python script, the success rate is 100%.

from openai import OpenAI

client = OpenAI(
base_url='my base_url',
api_key='my key', # ModelScope Token
)

response = client.chat.completions.create(
model='minimax-m25', # ModelScope Model-Id
messages=[
{
'role': 'system',
'content': 'You are a helpful assistant.'
},
{
'role': 'user',
'content': 'hello'
}
],
stream=True
)

for chunk in response:
if chunk.choices:
print(chunk.choices[0].delta.content, end='', flush=True)

=========== LLM configuration content in config.yaml file ==================

model:
default: minimax-m25
provider: custom
base_url: custom llm url
api_key: custom llm key content
stream: false
providers: {}
fallback_providers: []
credential_pool_strategies: {}
toolsets:

• hermes-cli

custom_providers:

• name: provider.costom.llm
  base_url: custom llm url
  api_key: custom llm key content
  model: minimax-m25
  stream: false
  models:
  minimax-m25:
  context_length: 204800

Globally disable streaming

streaming:
enabled: false

=========== LLM api_key configuration content in .env file ==================

OPENAI_API_KEY=custom llm key content

Steps to Reproduce

When repeatedly executing the test command hermes chat -q "hello" --toolsets browser, the success rate is extremely low.

Expected Behavior

Please see the description above.

Actual Behavior

Please see the description above.

Affected Component

CLI (interactive chat)

Messaging Platform (if gateway-related)

No response

Debug Report

Please see the description above.

Operating System

Ubuntu 24.04.4 LTS

Python Version

No response

Hermes Version

v0.10.0

Additional Logs / Traceback (optional)

Please see the description above.

Root Cause Analysis (optional)

Please see the description above.

Proposed Fix (optional)

Please see the description above.

Are you willing to submit a PR for this?

• I'd like to fix this myself and submit a PR

[ygd58]

Submitted a fix in #12110.

Root cause: the shared OpenAI client reuses pooled TCP connections that the custom endpoint server has already closed. Direct Python SDK calls always succeed because they create a fresh client (and pool) on every invocation.

Fix injects an explicit httpx.Client with keepalive_expiry=30s and max_keepalive_connections=2 for non-aggregator (custom/direct) endpoints. This prevents stale connection reuse without affecting OpenRouter/OpenAI/Anthropic which manage their own retry infrastructure.

[teknium1]

Thanks for the detailed report and for linking the custom-endpoint diagnosis in #12110. Automated hermes-sweeper review found that current main already provides the requested connection-lifecycle protection.

• run_agent.py:3918 builds the OpenAI HTTP client with httpx.Limits(keepalive_expiry=20.0), proactively expiring idle pooled connections before typical upstream proxy timeouts.
• agent/agent_runtime_helpers.py:1643 creates a fresh keepalive-enabled httpx.Client per OpenAI-client construction and copies input kwargs so a closed transport cannot be retained for later requests.
• agent/agent_runtime_helpers.py:960 also rebuilds the primary client after exhausted transient transport retries, specifically including direct/custom endpoints.
• Regression coverage at tests/run_agent/test_create_openai_client_reuse.py:65 protects against reusing a closed HTTP transport.
• The current pool-level implementation was shipped in 8324dd19ca283f1ea4ba34939489903c5d7450a6 (fix(agent): replace custom socket_options transport with httpx pool-level keepalive expiry), which is present on main.

Closing as implemented on main.