Auto-refreshing/health-checking stale auto/* candidate pools

[manchairwang]

Issue: Auto/* candidate pool has many connections. If some connections in front of the queue failed, the auto/* would keep trying them, which blocks auto/* service.

Suggestion: auto-refresh, or health-checking background, by a set time interval.

[diegosouzapw]

Hey @manchairwang! Good observation — when connections at the front of an auto/* queue keep failing, the pool spends cycles retrying bad candidates before reaching healthy ones.

OmniRoute already has a 3-layer resilience system that partially addresses this:

1. Provider circuit breaker (src/shared/utils/circuitBreaker.ts) — after N consecutive failures (configurable per provider type), the whole provider is temporarily excluded and combo routing skips it.
2. Connection cooldown (open-sse/services/accountFallback.ts) — individual connections get a cooldown with exponential backoff on repeated failures, so bad keys/accounts are skipped.
3. Model lockout — per-provider+connection+model failures don't disable the whole connection.

What you are describing is a proactive health-check (background pinging to evict stale candidates before they block requests) vs the current reactive approach (evict on failure). That is a meaningful improvement.

I have opened #8668 to track this. The existing reset-window and headroom combo strategies (docs/routing/AUTO-COMBO.md) are the closest to what you want, but they don't do background probing. A periodic health-check that pre-marks unhealthy connections would reduce latency for auto/* users significantly.

[manchairwang]

Thanks! I run into auto/* error again today. I'll try the existing reset-window and headroom combo strategies first. I'm also creating small local database to analyze available providers, models, and quotas.

[diegosouzapw]

Sounds like a solid plan, @manchairwang -- the reset-window/headroom combo strategies should get you most of the way there without a local DB, but building your own health/quota tracker on top is a reasonable belt-and-suspenders approach too. Let us know if you hit anything specific building it out.

[diegosouzapw]

Follow-up @manchairwang - I opened #9950 to track refresh and health-check behavior for stale auto/* candidate pools, including safe invalidation and observability.

[diegosouzapw]

Follow-up -- this is already tracked and shipped in #9950 (refresh and health-check for auto/* candidate pools).

[diegosouzapw]

@manchairwang A correction: my last reply said a refresh and health-check for auto/* candidate pools had shipped in #9950. Only part of it exists.

• A background credential health check runs every 5 minutes (src/lib/credentialHealth/scheduler.ts).
• auto/* pools drop candidates that are locked or in cooldown before routing (open-sse/services/autoCombo/resilienceCandidateFilter.ts). So a connection that just failed stops blocking the front of the queue once it's cooled down.

There's no periodic refresh or health probe built specifically for auto/* pools, which is what you asked for. #9950 was closed as backlog, with no PR. This discussion stays open for that part.