Stop retrying timeout error when dispatching worker command - #11851
Merged
Conversation
rkannan82
force-pushed
the
kannan/worker-cmd-no-retry-timeout
branch
from
August 28, 2026 20:23
d65cbbf to
3ba111c
Compare
When no worker is polling a control queue, the Nexus dispatch returns UpstreamTimeout after 10s. Previously this was retried up to 3 times, blocking a history goroutine for ~32s total per dead worker. At scale with many activity-worker deployments, this wastes goroutines on workers that are already gone. Return nil (non-retryable) on UpstreamTimeout instead. A single 10s attempt is sufficient to detect a missing poller. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
rkannan82
force-pushed
the
kannan/worker-cmd-no-retry-timeout
branch
from
August 28, 2026 21:18
427c893 to
855dd41
Compare
This was referenced Aug 28, 2026
rkannan82
marked this pull request as ready for review
August 29, 2026 18:23
Contributor
|
Claude finished @rkannan82's task in 2m 29s —— View job Reviewed 2 files, 2 findings. |
rkannan82
changed the title
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
rkannan82
changed the title
rkannan82
changed the title
rkannan82
changed the title
yycptt
approved these changes
Sep 1, 2026
yiminc
approved these changes
Sep 1, 2026
This was referenced Sep 1, 2026
…retry-timeout # Conflicts: # common/workercommands/dispatcher.go
rkannan82
enabled auto-merge (squash)
September 2, 2026 04:45
rkannan82
disabled auto-merge
September 2, 2026 04:46
rkannan82
enabled auto-merge (squash)
September 2, 2026 04:47
thestephenstanton
pushed a commit
that referenced
this pull request
Sep 2, 2026
## What Stop retrying worker command dispatch when the failure is a poller timeout (UpstreamTimeout). Transport errors (gRPC unreachable, connection refused) are still retried. ## Why When a worker is gone, each `DispatchNexusTask` attempt blocks a goroutine for the full 10s dispatch timeout waiting for a poller that will never arrive. With 3 max attempts, that's ~32s of blocked resources per command. At scale this adds up: if many workers go away simultaneously (e.g., deployment rollout), each dead worker can accumulate pending cancel commands. With per-destination concurrency of 100, this means up to 100 goroutines × N dead workers × 10s per attempt × 3 attempts — thousands of goroutines blocked on matching RPCs, causing memory pressure and connection buildup. Since worker commands are best-effort, there's no value in retrying after the first timeout. If no poller appeared in 10s, the worker is likely gone. ## How did you test it? - Unit tests: updated `TestExecute_UpstreamTimeout` and `TestHandleError_UpstreamTimeout_ReturnNil` to verify no retry on timeout while transport errors still retry. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Stop retrying worker command dispatch when the failure is a poller timeout (UpstreamTimeout). Transport errors (gRPC unreachable, connection refused) are still retried.
Why
When a worker is gone, each
DispatchNexusTaskattempt blocks a goroutine for the full 10s dispatch timeout waiting for a poller that will never arrive. With 3 max attempts, that's ~32s of blocked resources per command.At scale this adds up: if many workers go away simultaneously (e.g., deployment rollout), each dead worker can accumulate pending cancel commands. With per-destination concurrency of 100, this means up to 100 goroutines × N dead workers × 10s per attempt × 3 attempts — thousands of goroutines blocked on matching RPCs, causing memory pressure and connection buildup.
Since worker commands are best-effort, there's no value in retrying after the first timeout. If no poller appeared in 10s, the worker is likely gone.
How did you test it?
TestExecute_UpstreamTimeoutandTestHandleError_UpstreamTimeout_ReturnNilto verify no retry on timeout while transport errors still retry.🤖 Generated with Claude Code