chore(deploy-dev): bump helm --burst-limit 200 --qps 100 - #245
Closed
lilyshen0722 wants to merge 1 commit into
Closed
chore(deploy-dev): bump helm --burst-limit 200 --qps 100#245lilyshen0722 wants to merge 1 commit into
lilyshen0722 wants to merge 1 commit into
Conversation
Helm upgrade has been failing with `client rate limiter Wait returned an error: context deadline exceeded` on every revision since rev 67. Initial diagnosis blamed the 5m timeout (#244 bumped to 10m) and then RBAC (deploy SA needed admin in commonly-dev — applied out-of-band). Both helped marginally, but the underlying cause is helm's own internal client QPS limiter throttling itself during --wait status polling. Helm 3.14 supports --burst-limit and --qps. Defaults are 100/50 (since 3.10) but a chart with 30+ resources times out at the default during --wait readiness polling. 200/100 doubles the budget — keeps helm from self-throttling without raising load on the API server (which is happy to serve at much higher rates). Reference: helm/helm#9805 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Contributor
Author
|
Squashed manually as the merge commit on main. |
lilyshen0722
added a commit
that referenced
this pull request
Apr 27, 2026
Helm --wait has been hitting its own client QPS limiter during readiness polling — not API server throttling. Defaults (qps=50, burst=100 since helm 3.10) are insufficient for a chart with 30+ resources. 200/100 doubles the polling budget without taxing the API server. Reference: helm/helm#9805 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
5 tasks
lilyshen0722
added a commit
that referenced
this pull request
Aug 5, 2026
…pod (#835) (#839) Every `Deploy Dev` run on 2026-08-04 reported failure while deploying successfully — five consecutively, helm revisions 420-424 all recorded `failed`. Cause: `--wait` waits for EVERY resource in the release. Two workloads in the chart have been unready for days — `litellm` (CrashLoopBackOff, 561 restarts over 2d10h) and `cloud-codex-cody` (Init:CrashLoopBackOff) — so the wait burned its full 10m timeout and exited non-zero while backend, frontend, clawdbot-gateway and commonly-bot had all rolled out correctly minutes earlier. Verified live: those four pass `kubectl rollout status` immediately; the other two never will. The error it surfaces reads like throttling and is not: client rate limiter Wait returned an error: context deadline exceeded The release already runs `--burst-limit 200 --qps 100` (#245). That is the context deadline expiring while waiting on pods that cannot become ready, so raising the limiter — the obvious fix, and the first one I proposed on the issue — could never have worked. The damage was not the red check. helm records a failed upgrade, so the newest revision helm considers `deployed` went stale by days: a bare `helm rollback commonly-dev` during an incident would have silently reverted every deploy since Aug 2, including two auth fixes and the heartbeat opt-in. Rollback safety depends on this job telling the truth. So: drop `--wait`, and verify exactly the four workloads whose image tags this job sets. Scoping is the point — an unrelated unhealthy workload is a real problem but it is not THIS job's failure, and conflating them is what made the signal worthless. `kubectl rollout status` still returns non-zero on a genuine stuck rollout, so a real regression fails as loudly as before. Per-deployment timeouts, because clawdbot-gateway carries two init containers and takes minutes longer than the rest. Workloads outside the deployed set are surfaced as a notice rather than failed on. Silence is how litellm reached 561 restarts unnoticed; failing on it is how five good deploys got marked bad. A notice is the correct middle. Deliberately NOT included: disabling litellm and cloudCodex in values-dev. Both are `enabled: true` while dead, and turning them off would fix this too — but that is a decision about the native runtime and cloud agents, not a CI fix, and it belongs to whoever owns that call. Verified against the live cluster: the unhealthy-workload query returns exactly `cloud-codex-cody litellm`, and the four-deployment gate passes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Helm `--wait` has been hitting its own client QPS limiter during readiness polling — not pod-rollout-time, not API server throttling. Defaults (qps=50, burst=100 since helm 3.10) are insufficient for a chart with 30+ resources. Bumping to 200/100 doubles the polling budget without taxing the API server.
This is a pragmatic try at fixing the recurring `client rate limiter Wait returned an error: context deadline exceeded` that's been present on every helm upgrade since rev 67 (Apr 21). Earlier attempts to fix this by bumping the timeout (#244) or granting the deploy SA RBAC perms (out-of-band manual rolebinding) helped marginally but didn't address the root cause.
What this is NOT
Not a guaranteed fix — if the QPS theory is wrong we're back to one of: drop `--wait` entirely, accept the cosmetic failure, or dig deeper. Either way this is a low-risk one-line change worth trying first.
Test plan
🤖 Generated with Claude Code