Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

limit citus_drain_node to drain the specified node only #6267

Closed
mtuncer opened this issue Aug 29, 2022 · 1 comment · Fixed by #6361
Closed

limit citus_drain_node to drain the specified node only #6267

mtuncer opened this issue Aug 29, 2022 · 1 comment · Fixed by #6361
Assignees

Comments

@mtuncer
Copy link
Member

mtuncer commented Aug 29, 2022

citus_drain_node tries to move data from multiple workers which have their shouldhaveshards value set to false.

this behavior is similar to rebalance_table_shards(drain_only: true) .

However, user explicitly wants to drain a single node, operation should be limited to that node only.

Scenario is

  • User wants to scale down their cluster with more than one node (say from 8 nodes to 5 nodes)
  • draining one by one is not optimal since, it will also move data to nodes that are going to be removed later
  • user sets shouldhaveshards value in pg_dist_node to false for nodes to be removed
  • user starts citus_drain_node in a single worker, hoping it would drain a single node in their allocated maintenance window
  • however citus tries to move data out from all 3 nodes, causing operation to take longer and not finishing on the maintenance window

Our documentation at https://docs.citusdata.com/en/v11.0/develop/api_udf.html#citus-drain-node does not mention this behavior, even hints that it moves data out from a specified (aka single) worker. However reality is different.

@JelteF
Copy link
Contributor

JelteF commented Aug 29, 2022

A second possible improvement that this issue highlights is that for draining nodes in general it would likely be optimal to do them one-by-one (even when draining them all at once).

gokhangulbiz added a commit that referenced this issue Oct 9, 2022
DESCRIPTION: Fixes citus_drain_node to drain the specified worker only.

Fixes #6267
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants