Skip to content

[server] Add per-task timeout for RebalanceManager to prevent indefinite blocking#3429

Merged
swuferhong merged 3 commits into
apache:mainfrom
swuferhong:reblance-block
Jun 7, 2026
Merged

[server] Add per-task timeout for RebalanceManager to prevent indefinite blocking#3429
swuferhong merged 3 commits into
apache:mainfrom
swuferhong:reblance-block

Conversation

@swuferhong

Copy link
Copy Markdown
Contributor

Purpose

Linked issue: close #3096

  • Fix a liveness issue where RebalanceManager could block indefinitely if a rebalance bucket's ISR never converges (e.g., target replica is on a dead TabletServer).
  • Introduce a per-task timeout (2 minutes) with periodic checks (every 30 seconds). When an in-flight bucket exceeds the timeout, it is treated as COMPLETED, unblocking subsequent buckets and the overall rebalance.
  • Add RebalanceTaskTimeoutEvent to the coordinator event system for safe single-threaded state mutation.

Brief change log

Tests

API and Format

Documentation

@loserwang1024 loserwang1024 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I left a comment

@LiebingYu LiebingYu left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@swuferhong Thanks for the PR. I have some comments.

@swuferhong swuferhong merged commit 91bea7c into apache:main Jun 7, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[server] Add timeout mechanism for rebalance tasks to prevent indefinite blocking on missing completion events

3 participants