Skip to content

Tune default values for dispatching worker command - #11900

Merged
rkannan82 merged 3 commits into
mainfrom
kannan/worker-cmd-config-defaults
Sep 3, 2026
Merged

Tune default values for dispatching worker command#11900
rkannan82 merged 3 commits into
mainfrom
kannan/worker-cmd-config-defaults

Conversation

@rkannan82

@rkannan82 rkannan82 commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

What

Change default dynamic config values for worker command dispatch:

  • WorkerCommandsDispatchTimeout: 10s → 5s
  • WorkerCommandsMaxAttempts: 3 → 30

Why

  • Shorter timeout (5s): Each dispatch attempt holds an outbound executor thread waiting for a poller. If the worker is gone, that thread is blocked for nothing. Cutting from 10s to 5s means we detect missing workers faster and free up the thread sooner.

  • More attempts (30): The main reason for retrying is to survive matching server restarts — during a rolling restart, dispatch RPCs can fail with transient transport errors until the new pod is ready. With only 3 attempts, commands could be dropped permanently during a routine restart. 30 attempts with the default backoff (initial=1s, coefficient=1.1) spreads retries over ~2 minutes, enough to ride out a restart. Transport errors fail fast (no blocking wait), so more attempts are cheap.

🤖 Generated with Claude Code

@rkannan82 rkannan82 changed the title Tune worker command dispatch defaults: 5s timeout, 30 max attempts Tune worker command dispatch defaults Sep 2, 2026
@rkannan82 rkannan82 changed the title Tune worker command dispatch defaults Tune default values for worker command dispatch rpc Sep 2, 2026
@rkannan82
rkannan82 marked this pull request as ready for review September 2, 2026 04:50
@rkannan82
rkannan82 requested a review from a team September 2, 2026 04:50
@rkannan82
rkannan82 requested review from a team as code owners September 2, 2026 04:50
@rkannan82
rkannan82 requested a review from yycptt September 2, 2026 04:50
@github-actions

github-actions Bot commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Claude finished @rkannan82's task in 2m 6s —— View job


Reviewed 1 file, 2 findings.
kannan/worker-cmd-config-defaults

@rkannan82 rkannan82 changed the title Tune default values for worker command dispatch rpc Tune default values for dispatching worker command Sep 2, 2026
@rkannan82
rkannan82 force-pushed the kannan/worker-cmd-config-defaults branch from 0e6923e to 3dadf82 Compare September 2, 2026 04:52
Comment thread common/dynamicconfig/constants.go
Comment thread common/dynamicconfig/constants.go
@rkannan82
rkannan82 force-pushed the kannan/worker-cmd-config-defaults branch from 3dadf82 to 9159e31 Compare September 2, 2026 04:54
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@rkannan82
rkannan82 force-pushed the kannan/worker-cmd-config-defaults branch from 9159e31 to a0a0134 Compare September 2, 2026 04:55
@rkannan82
rkannan82 requested a review from ShahabT September 3, 2026 19:00
@rkannan82 rkannan82 added the reliability-2026 Reliability related changes label Sep 3, 2026
@rkannan82
rkannan82 requested a review from yiminc September 3, 2026 20:08
@rkannan82
rkannan82 enabled auto-merge (squash) September 3, 2026 20:09
@rkannan82
rkannan82 merged commit 1f8442b into main Sep 3, 2026
57 checks passed
@rkannan82
rkannan82 deleted the kannan/worker-cmd-config-defaults branch September 3, 2026 20:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

reliability-2026 Reliability related changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants