fix(llm-gateway): refresh provider sets when model_cost reloads

[richardsolomou]

Problem

LiteLLM's get_llm_provider resolves bare model names against module-level sets (anthropic_models, open_ai_chat_completion_models, etc.) populated only at import time. Refreshing litellm.model_cost doesn't touch them, so models added upstream after process start (e.g. claude-opus-4-8 added 7h ago) raise LLM Provider NOT provided on the gateway until a redeploy.

Changes

Call litellm.add_known_models(model_cost) after every litellm.model_cost = … reassignment in CostRefreshService.refresh and ModelCostService._refresh_cache. This is what LiteLLM itself does at import.

How did you test this code?

Agent-authored. Ran uv run pytest tests/test_cost_refresh.py and the broader test_model_registry.py/test_models_api.py/callbacks/test_rate_limiting.py suite (80 tests) — all green. New regression test asserts a freshly fetched model lands in both litellm.anthropic_models and litellm.open_ai_chat_completion_models.

Automatic notifications

• Publish to changelog?
• Alert Sales and Marketing teams?

Docs update

N/A

🤖 Agent context

Authored by PostHog Code. Diagnosis path: traced the gateway's 400 from litellm.anthropic_messages for claude-opus-4-8 to get_llm_provider_logic.py:395, which checks litellm.anthropic_models rather than litellm.model_cost. Confirmed the v1.83.7-stable bundled model_prices_and_context_window_backup.json lacks both claude-opus-4-7 and claude-opus-4-8 (the live import-time fetch is what seeds known-good models today), and that neither refresh path called add_known_models after reassignment.

Considered prefixing anthropic/ on the model in _handle_anthropic_messages (mirroring the bedrock/ prefix already used). Rejected: only patches the Anthropic route; the same staleness affects any provider whose models land in the upstream JSON after startup. Fixing at the refresh boundary covers every provider.

Aside for a future PR: CostRefreshService and ModelCostService are two singletons racing on the same litellm.model_cost global — worth consolidating, out of scope here.

---

Created with PostHog Code

[greptile-apps]

Prompt To Fix All With AI

Fix the following 1 code review issue. Work through them one at a time, proposing concise fixes.

---

### Issue 1 of 1
services/llm-gateway/tests/test_cost_refresh.py:46-73
**Missing test coverage for `ModelCostService._refresh_cache`**

The regression test verifies `add_known_models` is called via `CostRefreshService.refresh()`, but the same change was made to `ModelCostService._refresh_cache()` with no corresponding test. If `ModelCostService` follows a different code path (e.g., its `_refresh_cache` silently swallows an error before or after `add_known_models`), there is no test to catch the regression.

_{Reviews (1): Last reviewed commit: "fix(llm-gateway): refresh provider sets ..." | Re-trigger Greptile}

[deployment-status-posthog]

Deploy status

Environment	Status	Deployed At	Workflow
dev	✅ Deployed	2026-05-29 10:52 UTC	Run
prod-us	✅ Deployed	2026-05-29 11:15 UTC	Run
prod-eu	✅ Deployed	2026-05-29 11:18 UTC	Run