Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Consider allowing to change node address in one step instead of removing-then-adding #875

Closed
drmingdrmer opened this issue Jun 22, 2023 · 1 comment · Fixed by #876
Closed

Comments

@drmingdrmer
Copy link
Member

drmingdrmer commented Jun 22, 2023

When adding an existing node as a learner with a new address the old address remains unchanged.

There should be a way to update Node directly without removing-then-adding.
But this has risk of leading to brain split. Use it with caution.

Incorrect updating node could occur if the new address belongs to another node, leading to
two elected leaders.

Consider a 3-node cluster (a, b, c, with addresses x, y, z) and an
uninitialized node d with address w:

a: x
b: y
c: z

d: w

Allowing direct address changes (e.g., b's address from y to w) would
enable both x, y and z, w to form quorums and elect leaders.

  • c proposes ChangeMembership: {a:x, b:w, c:z};

  • c and d grant c;

  • c elects itself as leader

  • c and d confirm c as leader

  • a elects itself as leader

  • a and b confirm a as leader

It's best to avoid updating Node directly

@github-actions
Copy link

👋 Thanks for opening this issue!

Get help or engage by:

  • /help : to print help messages.
  • /assignme : to assign this issue to you.

drmingdrmer added a commit to drmingdrmer/openraft that referenced this issue Jun 22, 2023
During dynamic cluster changes, we sometimes need to update an existing
node, for example changing its network address.

Adding `SetNodes` variant to `ChangeMembers` allows replacing an
existing node directly.
However, this also carries risk of creating a split brain scenario if
used incorrectly.

- Fix: datafuselabs#875
drmingdrmer added a commit to drmingdrmer/openraft that referenced this issue Jun 22, 2023
During dynamic cluster changes, we sometimes need to update an existing
node, for example changing its network address.

Adding `SetNodes` variant to `ChangeMembers` allows replacing an
existing node directly.
However, this also carries risk of creating a split brain scenario if
used incorrectly.

- Fix: datafuselabs#875
drmingdrmer added a commit that referenced this issue Jun 22, 2023
During dynamic cluster changes, we sometimes need to update an existing
node, for example changing its network address.

Adding `SetNodes` variant to `ChangeMembers` allows replacing an
existing node directly.
However, this also carries risk of creating a split brain scenario if
used incorrectly.

- Fix: #875
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant