feat(litellm): raise claude-code-subscription rate limits - #1503
Merged
Conversation
Measured 429s in LiteLLM logs at the old 10 rpm limit presented to the client as timeouts. Rewrites the sizing comment to reflect the real, current rationale instead of the stale "one interactive CLI user" justification.
Rpm-only fix left roughly a quarter of the observed 429s in place: logs show ~171 token-limit 429s alongside ~556 request-limit 429s in 24h. 7500000 keeps the same tokens-per-request headroom now that rpm is 300, since Claude Code requests can carry 50-200k token contexts.
Contributor
--- LiteLLMVirtualKey ai/claude-code-subscription
+++ LiteLLMVirtualKey ai/claude-code-subscription
@@ -12,8 +12,8 @@
models:
- claude-code-subscription
proxyRef: litellm
- rpmLimit: 10
+ rpmLimit: 300
secretKey: key
secretName: litellm-key-claude-code-subscription
- tpmLimit: 250000
+ tpmLimit: 7500000
|
|
✅ Automated recommendation: APPROVE Analysis engine: pr-review-local@http://litellm.ai.svc.cluster.local:4000/v1 (openai) RecommendationApprove. This PR correctly raises the Change-by-Change Findings1.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Intent
Raise the claude-code-subscription LiteLLM virtual key's rate limits in kubernetes/apps/base/ai/litellm/app/virtualkeys/claude-code-subscription.yaml. Captain reported client-side timeouts on the Claude Code subscription pass-through key; verified via LiteLLM proxy logs (litellm-99ccbcc74-dj4jp, last 24h) showing ~556 429s with 'Limit type: requests. Current limit: 10' and ~171 429s with 'Limit type: tokens. Current limit: 250000' (250000 is unique to this key's tpmLimit). Change: rpmLimit 10 -> 300 (captain-specified number). tpmLimit 250000 -> 7500000 (captain-approved after escalating concrete evidence that the token limit was also being hit; 7500000 preserves the same tokens-per-request headroom now that rpm is 300, since Claude Code requests can carry 50-200k token contexts). Rewrite the sizing comment to explain both numbers with the measured evidence (both 429 counts), replacing the stale 'one interactive CLI user, same order of magnitude as opencode.yaml' rationale which no longer applies. Preserve intact: the surrounding safety-reasoning comment explaining why this key deliberately has NO maxBudget (its model is priced at $0 so any budget would be silently inert) and that rate limits are therefore the whole guardrail - do not remove either limit, do not add a maxBudget. Do not widen the key's models list (stays scoped to claude-code-subscription only). Do not touch any other virtual key, the model catalog, fallback chains, the ai-pr-review key, or the proxy's own configuration. State plainly in the PR that this is a runaway-agent guardrail, not a throughput target: the real ceiling is Anthropic's own rate limiting against the captain's personal subscription, which this proxy neither sees nor can raise - if timeouts persist after this, the cause is upstream of the proxy and this change will not have fixed it. Also state plainly that the LiteLLMVirtualKey is server-side state reconciled by litellm-operator, so the new limits only take effect on the live key after this merges and the operator reconciles - verifying that the 429s actually stop is a post-merge check, not something confirmed in this PR.
What Changed
claude-code-subscriptionLiteLLM virtual key fromrpmLimit: 10/tpmLimit: 250000to300/7500000, keeping the model allow-list scoped toclaude-code-subscriptiononly and leavingmaxBudgetunset (rate limits remain the sole guardrail on the $0-priced pass-through model).LiteLLMVirtualKeyis server-side state reconciled by litellm-operator, so the new limits take effect only after merge + operator reconcile; confirming the 429s stop is a post-merge check.Risk Assessment
✅ Low: Single-file intentional rpm/tpm raise with preserved no-budget guardrail, scoped model list, and no other keys or proxy config touched.
Testing
Updated the stale CI pin for the new limits, then ran the focused subscription test plus a kustomize-emitted CR semantic proof; all checks passed showing 300 rpm / 7500000 tpm, preserved 25000 tokens-per-request headroom, no maxBudget, single-model scope, siblings untouched, and the rewritten evidence-backed sizing comment. Live 429 cessation remains intentionally post-merge after litellm-operator reconcile.
Evidence: kustomize-emitted CR rate-limit semantics + isolation
Evidence: litellm-claude-code-subscription-test.py full run (40/40)
Evidence: structured test results JSON
Evidence: kustomize-emitted LiteLLMVirtualKey YAML
apiVersion: litellm.home-operations.com/v1alpha1 kind: LiteLLMVirtualKey metadata: name: claude-code-subscription spec: keyAlias: claude-code-subscription models: [claude-code-subscription] rpmLimit: 300 tpmLimit: 7500000 (maxBudget absent)Pipeline
Updates from git push no-mistakes
✅ **intent** - passed
✅ No issues found.
✅ **Rebase** - passed
✅ No issues found.
✅ **Review** - passed
✅ No issues found.
✅ **Test** - passed
✅ No issues found.
UpdatedEXPECTED_RPM/EXPECTED_TPMinscripts/ci/litellm-claude-code-subscription-test.pyfrom 10/250000 to 300/7500000 to match the captain-approved CR valuesEVIDENCE_OUT=... python3 scripts/ci/litellm-claude-code-subscription-test.py(40/40 pass, including rpm/tpm sizing, no maxBudget, models scope, sibling isolation, kustomize emit)kubectl kustomize kubernetes/apps/base/ai/litellm/appconsumer parse asserting emitted LiteLLMVirtualKey rpmLimit=300 tpmLimit=7500000, models=['claude-code-subscription'], maxBudget absenttokens-per-request headroom check: 250000/10 == 7500000/300 == 25000Sibling virtual-key inventory isolation (ai-pr-review/demo/ha-demo/opencode/repo-wiki/router-demo unchanged)Operator comment contract: safety/no-maxBudget guardrail text preserved; sizing comment includes ~556/~171 evidence and 300/7500000git diffisolation: onlykubernetes/apps/base/ai/litellm/app/virtualkeys/claude-code-subscription.yamlunder litellm vs base✅ **Document** - passed
✅ No issues found.
✅ **Lint** - passed
✅ No issues found.
✅ **Push** - passed
✅ No issues found.