Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Raft: send Raft state depending on Tarantool version, not on mode #5438

Closed
Gerold103 opened this issue Oct 21, 2020 · 1 comment
Closed

Raft: send Raft state depending on Tarantool version, not on mode #5438

Gerold103 opened this issue Oct 21, 2020 · 1 comment
Assignees
@sergepetrenko
Labels
bug Something isn't working qsync replication raft RAFT protocol replication
Milestone
2.7.3

Comments

@Gerold103
Copy link
Collaborator

Currently Raft state is sent to a subscriber when it is connected and the election is enabled. This leads to a bug (or maybe just inconvenience), that if the node was a leader, was turned off, and couldn't deliver the 'off' message to all the followers, some of them would still think the node is a leader until their restart or a new leader election.

If the followers are just voters or also have Raft disabled, they treat the old node as a leader infinitely until a new one is elected somewhere.

The state isn't sent not to break connections with the old versions, which won't understand the Raft messages.

That should be fixed so the state is sent always when possible. When the subscriber has version >= 2.6.0.
@Gerold103 Gerold103 added bug Something isn't working qsync replication raft RAFT protocol labels Oct 21, 2020
@kyukhin kyukhin added this to the 2.7.1 milestone Oct 22, 2020
@kyukhin kyukhin modified the milestones: 2.7.1, 2.6.2 Nov 2, 2020
@sergepetrenko
Copy link
Collaborator

There's another problem with current approach:
It's possible that a running instance will never get is_leader notification from an instance it has just voted for.
Imagine you have a candidate, which lacks one vote to become a leader. Then you start a voter, and connect it to the candidate. box_process_subscribe involves sending complete raft state, which includes a vote request.
This vote request reaches the follower, and the follower votes for the candidate immediately.
If this vote reaches the candidate earlier than the candidate manages to start the relay thread and set relay->is_raft_enabled, the candidate will not send its state to the voter, leading to voter believing there's no leader until it's restarted.

@kyukhin kyukhin modified the milestones: 2.6.2, 2.6.3 Dec 25, 2020
@kyukhin kyukhin removed the tmp label Feb 12, 2021
@kyukhin kyukhin added the tmp label Apr 6, 2021
@kyukhin kyukhin modified the milestones: 2.6.3, 2.7.3 Apr 21, 2021
@kyukhin kyukhin removed the tmp label Apr 30, 2021
sergepetrenko added a commit that referenced this issue Jun 10, 2021
@sergepetrenko
replication: always send raft state to subscribers
409e72c 
Tarantool used to send out raft state on subscribe only when raft was
enabled. This was a safeguard against partially-upgraded clusters, where
some nodes had no clue about Raft messages and couldn't handle them
properly.

Actually, Raft state should be sent out always. For example, promote
bumps Raft term, even when raft is disabled, and it's important that
everyone in cluster has the same term, for the sake of promote at least.

So, send out Raft state to every subscriber with version >= 2.6.0
(that's when Raft was introduced).

Closes #5438
@sergepetrenko sergepetrenko self-assigned this Jun 10, 2021
sergepetrenko added a commit that referenced this issue Jun 10, 2021
@sergepetrenko
replication: always send raft state to subscribers
130aa88 
Tarantool used to send out raft state on subscribe only when raft was
enabled. This was a safeguard against partially-upgraded clusters, where
some nodes had no clue about Raft messages and couldn't handle them
properly.

Actually, Raft state should be sent out always. For example, promote
bumps Raft term, even when raft is disabled, and it's important that
everyone in cluster has the same term, for the sake of promote at least.

So, send out Raft state to every subscriber with version >= 2.6.0
(that's when Raft was introduced).

Closes #5438
sergepetrenko added a commit that referenced this issue Jun 10, 2021
@sergepetrenko
replication: always send raft state to subscribers
ee5c37f 
Tarantool used to send out raft state on subscribe only when raft was
enabled. This was a safeguard against partially-upgraded clusters, where
some nodes had no clue about Raft messages and couldn't handle them
properly.

Actually, Raft state should be sent out always. For example, promote
bumps Raft term, even when raft is disabled, and it's important that
everyone in cluster has the same term, for the sake of promote at least.

So, send out Raft state to every subscriber with version >= 2.6.0
(that's when Raft was introduced).

Closes #5438
sergepetrenko added a commit that referenced this issue Jun 17, 2021
@sergepetrenko
replication: always send raft state to subscribers
ef57e45 
Tarantool used to send out raft state on subscribe only when raft was
enabled. This was a safeguard against partially-upgraded clusters, where
some nodes had no clue about Raft messages and couldn't handle them
properly.

Actually, Raft state should be sent out always. For example, promote
will be changed to bump Raft term even when Raft is disabled, and it's
important that everyone in cluster has the same term for the sake of promote
at least.

So, send out Raft state to every subscriber with version >= 2.6.0
(that's when Raft was introduced).
Do the same for Raft broadcasts. They should be sent only to replicas
with version >= 2.6.0

Closes #5438
sergepetrenko added a commit that referenced this issue Jun 18, 2021
@sergepetrenko
replication: always send raft state to subscribers
0276f2f 
Tarantool used to send out raft state on subscribe only when raft was
enabled. This was a safeguard against partially-upgraded clusters, where
some nodes had no clue about Raft messages and couldn't handle them
properly.

Actually, Raft state should be sent out always. For example, promote
will be changed to bump Raft term even when Raft is disabled, and it's
important that everyone in cluster has the same term for the sake of promote
at least.

So, send out Raft state to every subscriber with version >= 2.6.0
(that's when Raft was introduced).
Do the same for Raft broadcasts. They should be sent only to replicas
with version >= 2.6.0

Closes #5438
sergepetrenko added a commit that referenced this issue Jun 21, 2021
@sergepetrenko
replication: always send raft state to subscribers
e354ec1 
Tarantool used to send out raft state on subscribe only when raft was
enabled. This was a safeguard against partially-upgraded clusters, where
some nodes had no clue about Raft messages and couldn't handle them
properly.

Actually, Raft state should be sent out always. For example, promote
will be changed to bump Raft term even when Raft is disabled, and it's
important that everyone in cluster has the same term for the sake of promote
at least.

So, send out Raft state to every subscriber with version >= 2.6.0
(that's when Raft was introduced).
Do the same for Raft broadcasts. They should be sent only to replicas
with version >= 2.6.0

Closes #5438
sergepetrenko added a commit that referenced this issue Jun 28, 2021
@sergepetrenko
replication: always send raft state to subscribers
099ee3a 
Tarantool used to send out raft state on subscribe only when raft was
enabled. This was a safeguard against partially-upgraded clusters, where
some nodes had no clue about Raft messages and couldn't handle them
properly.

Actually, Raft state should be sent out always. For example, promote
will be changed to bump Raft term even when Raft is disabled, and it's
important that everyone in cluster has the same term for the sake of promote
at least.

So, send out Raft state to every subscriber with version >= 2.6.0
(that's when Raft was introduced).
Do the same for Raft broadcasts. They should be sent only to replicas
with version >= 2.6.0

Closes #5438
cyrillos pushed a commit that referenced this issue Jul 10, 2021
@sergepetrenko @cyrillos
replication: always send raft state to subscribers
b415b98 
Tarantool used to send out raft state on subscribe only when raft was
enabled. This was a safeguard against partially-upgraded clusters, where
some nodes had no clue about Raft messages and couldn't handle them
properly.

Actually, Raft state should be sent out always. For example, promote
will be changed to bump Raft term even when Raft is disabled, and it's
important that everyone in cluster has the same term for the sake of promote
at least.

So, send out Raft state to every subscriber with version >= 2.6.0
(that's when Raft was introduced).
Do the same for Raft broadcasts. They should be sent only to replicas
with version >= 2.6.0

Closes #5438
sergepetrenko added a commit that referenced this issue Jul 14, 2021
@sergepetrenko
replication: always send raft state to subscribers
e6001e4 
Tarantool used to send out raft state on subscribe only when raft was
enabled. This was a safeguard against partially-upgraded clusters, where
some nodes had no clue about Raft messages and couldn't handle them
properly.

Actually, Raft state should be sent out always. For example, promote
will be changed to bump Raft term even when Raft is disabled, and it's
important that everyone in cluster has the same term for the sake of promote
at least.

So, send out Raft state to every subscriber with version >= 2.6.0
(that's when Raft was introduced).
Do the same for Raft broadcasts. They should be sent only to replicas
with version >= 2.6.0

Closes #5438
cyrillos pushed a commit that referenced this issue Jul 14, 2021
@sergepetrenko @cyrillos
replication: always send raft state to subscribers
0f9dbac 
Tarantool used to send out raft state on subscribe only when raft was
enabled. This was a safeguard against partially-upgraded clusters, where
some nodes had no clue about Raft messages and couldn't handle them
properly.

Actually, Raft state should be sent out always. For example, promote
will be changed to bump Raft term even when Raft is disabled, and it's
important that everyone in cluster has the same term for the sake of promote
at least.

So, send out Raft state to every subscriber with version >= 2.6.0
(that's when Raft was introduced).
Do the same for Raft broadcasts. They should be sent only to replicas
with version >= 2.6.0

Closes #5438
sergepetrenko added a commit that referenced this issue Jul 15, 2021
@sergepetrenko
replication: always send raft state to subscribers
7214585 
Tarantool used to send out raft state on subscribe only when raft was
enabled. This was a safeguard against partially-upgraded clusters, where
some nodes had no clue about Raft messages and couldn't handle them
properly.

Actually, Raft state should be sent out always. For example, promote
will be changed to bump Raft term even when Raft is disabled, and it's
important that everyone in cluster has the same term for the sake of promote
at least.

So, send out Raft state to every subscriber with version >= 2.6.0
(that's when Raft was introduced).
Do the same for Raft broadcasts. They should be sent only to replicas
with version >= 2.6.0

Closes #5438
cyrillos pushed a commit that referenced this issue Jul 16, 2021
@sergepetrenko @cyrillos
replication: always send raft state to subscribers
20aec30 
Tarantool used to send out raft state on subscribe only when raft was
enabled. This was a safeguard against partially-upgraded clusters, where
some nodes had no clue about Raft messages and couldn't handle them
properly.

Actually, Raft state should be sent out always. For example, promote
will be changed to bump Raft term even when Raft is disabled, and it's
important that everyone in cluster has the same term for the sake of promote
at least.

So, send out Raft state to every subscriber with version >= 2.6.0
(that's when Raft was introduced).
Do the same for Raft broadcasts. They should be sent only to replicas
with version >= 2.6.0

Closes #5438
cyrillos pushed a commit that referenced this issue Jul 19, 2021
@sergepetrenko @cyrillos
replication: always send raft state to subscribers
b710b13 
Tarantool used to send out raft state on subscribe only when raft was
enabled. This was a safeguard against partially-upgraded clusters, where
some nodes had no clue about Raft messages and couldn't handle them
properly.

Actually, Raft state should be sent out always. For example, promote
will be changed to bump Raft term even when Raft is disabled, and it's
important that everyone in cluster has the same term for the sake of promote
at least.

So, send out Raft state to every subscriber with version >= 2.6.0
(that's when Raft was introduced).
Do the same for Raft broadcasts. They should be sent only to replicas
with version >= 2.6.0

Closes #5438
cyrillos pushed a commit that referenced this issue Jul 22, 2021
@sergepetrenko @cyrillos
replication: always send raft state to subscribers
b176b78 
Tarantool used to send out raft state on subscribe only when raft was
enabled. This was a safeguard against partially-upgraded clusters, where
some nodes had no clue about Raft messages and couldn't handle them
properly.

Actually, Raft state should be sent out always. For example, promote
will be changed to bump Raft term even when Raft is disabled, and it's
important that everyone in cluster has the same term for the sake of promote
at least.

So, send out Raft state to every subscriber with version >= 2.6.0
(that's when Raft was introduced).
Do the same for Raft broadcasts. They should be sent only to replicas
with version >= 2.6.0

Closes #5438
sergepetrenko added a commit that referenced this issue Jul 23, 2021
@sergepetrenko
replication: always send raft state to subscribers
8030d34 
Tarantool used to send out raft state on subscribe only when raft was
enabled. This was a safeguard against partially-upgraded clusters, where
some nodes had no clue about Raft messages and couldn't handle them
properly.

Actually, Raft state should be sent out always. For example, promote
will be changed to bump Raft term even when Raft is disabled, and it's
important that everyone in cluster has the same term for the sake of promote
at least.

So, send out Raft state to every subscriber with version >= 2.6.0
(that's when Raft was introduced).
Do the same for Raft broadcasts. They should be sent only to replicas
with version >= 2.6.0

Closes #5438
sergepetrenko added a commit that referenced this issue Jul 29, 2021
@sergepetrenko
replication: always send raft state to subscribers
18ea6a6 
Tarantool used to send out raft state on subscribe only when raft was
enabled. This was a safeguard against partially-upgraded clusters, where
some nodes had no clue about Raft messages and couldn't handle them
properly.

Actually, Raft state should be sent out always. For example, promote
will be changed to bump Raft term even when Raft is disabled, and it's
important that everyone in cluster has the same term for the sake of promote
at least.

So, send out Raft state to every subscriber with version >= 2.6.0
(that's when Raft was introduced).
Do the same for Raft broadcasts. They should be sent only to replicas
with version >= 2.6.0

Closes #5438
cyrillos pushed a commit that referenced this issue Jul 30, 2021
@sergepetrenko @cyrillos
replication: always send raft state to subscribers
efe1877 
Tarantool used to send out raft state on subscribe only when raft was
enabled. This was a safeguard against partially-upgraded clusters, where
some nodes had no clue about Raft messages and couldn't handle them
properly.

Actually, Raft state should be sent out always. For example, promote
will be changed to bump Raft term even when Raft is disabled, and it's
important that everyone in cluster has the same term for the sake of promote
at least.

So, send out Raft state to every subscriber with version >= 2.6.0
(that's when Raft was introduced).
Do the same for Raft broadcasts. They should be sent only to replicas
with version >= 2.6.0

Closes #5438
sergepetrenko added a commit that referenced this issue Aug 2, 2021
@sergepetrenko
replication: always send raft state to subscribers
734a61e 
Tarantool used to send out raft state on subscribe only when raft was
enabled. This was a safeguard against partially-upgraded clusters, where
some nodes had no clue about Raft messages and couldn't handle them
properly.

Actually, Raft state should be sent out always. For example, promote
will be changed to bump Raft term even when Raft is disabled, and it's
important that everyone in cluster has the same term for the sake of promote
at least.

So, send out Raft state to every subscriber with version >= 2.6.0
(that's when Raft was introduced).
Do the same for Raft broadcasts. They should be sent only to replicas
with version >= 2.6.0

Closes #5438
sergepetrenko added a commit that referenced this issue Aug 4, 2021
@sergepetrenko
replication: always send raft state to subscribers
4b67a36 
Tarantool used to send out raft state on subscribe only when raft was
enabled. This was a safeguard against partially-upgraded clusters, where
some nodes had no clue about Raft messages and couldn't handle them
properly.

Actually, Raft state should be sent out always. For example, promote
will be changed to bump Raft term even when Raft is disabled, and it's
important that everyone in cluster has the same term for the sake of promote
at least.

So, send out Raft state to every subscriber with version >= 2.6.0
(that's when Raft was introduced).
Do the same for Raft broadcasts. They should be sent only to replicas
with version >= 2.6.0

Closes #5438
cyrillos pushed a commit that referenced this issue Aug 4, 2021
@sergepetrenko @cyrillos
replication: always send raft state to subscribers
029f23b 
Tarantool used to send out raft state on subscribe only when raft was
enabled. This was a safeguard against partially-upgraded clusters, where
some nodes had no clue about Raft messages and couldn't handle them
properly.

Actually, Raft state should be sent out always. For example, promote
will be changed to bump Raft term even when Raft is disabled, and it's
important that everyone in cluster has the same term for the sake of promote
at least.

So, send out Raft state to every subscriber with version >= 2.6.0
(that's when Raft was introduced).
Do the same for Raft broadcasts. They should be sent only to replicas
with version >= 2.6.0

Closes #5438
sergepetrenko added a commit that referenced this issue Aug 5, 2021
@sergepetrenko
replication: always send raft state to subscribers
9718091 
Tarantool used to send out raft state on subscribe only when raft was
enabled. This was a safeguard against partially-upgraded clusters, where
some nodes had no clue about Raft messages and couldn't handle them
properly.

Actually, Raft state should be sent out always. For example, promote
will be changed to bump Raft term even when Raft is disabled, and it's
important that everyone in cluster has the same term for the sake of promote
at least.

So, send out Raft state to every subscriber with version >= 2.6.0
(that's when Raft was introduced).
Do the same for Raft broadcasts. They should be sent only to replicas
with version >= 2.6.0

Closes #5438
sergepetrenko added a commit that referenced this issue Aug 5, 2021
@sergepetrenko
replication: always send raft state to subscribers
355fff0 
Tarantool used to send out raft state on subscribe only when raft was
enabled. This was a safeguard against partially-upgraded clusters, where
some nodes had no clue about Raft messages and couldn't handle them
properly.

Actually, Raft state should be sent out always. For example, promote
will be changed to bump Raft term even when Raft is disabled, and it's
important that everyone in cluster has the same term for the sake of promote
at least.

So, send out Raft state to every subscriber with version >= 2.6.0
(that's when Raft was introduced).
Do the same for Raft broadcasts. They should be sent only to replicas
with version >= 2.6.0

Closes #5438
@kyukhin kyukhin closed this as completed in 3bee3fe Aug 6, 2021
sergepetrenko added a commit that referenced this issue Aug 9, 2021
@sergepetrenko
replication: always send raft state to subscribers
f22e754 
Tarantool used to send out raft state on subscribe only when raft was
enabled. This was a safeguard against partially-upgraded clusters, where
some nodes had no clue about Raft messages and couldn't handle them
properly.

Actually, Raft state should be sent out always. For example, promote
will be changed to bump Raft term even when Raft is disabled, and it's
important that everyone in cluster has the same term for the sake of promote
at least.

So, send out Raft state to every subscriber with version >= 2.6.0
(that's when Raft was introduced).
Do the same for Raft broadcasts. They should be sent only to replicas
with version >= 2.6.0

Closes #5438
sergepetrenko added a commit that referenced this issue Aug 9, 2021
@sergepetrenko
replication: always send raft state to subscribers
53bd3cd 
Tarantool used to send out raft state on subscribe only when raft was
enabled. This was a safeguard against partially-upgraded clusters, where
some nodes had no clue about Raft messages and couldn't handle them
properly.

Actually, Raft state should be sent out always. For example, promote
will be changed to bump Raft term even when Raft is disabled, and it's
important that everyone in cluster has the same term for the sake of promote
at least.

So, send out Raft state to every subscriber with version >= 2.6.0
(that's when Raft was introduced).
Do the same for Raft broadcasts. They should be sent only to replicas
with version >= 2.6.0

Closes #5438
kyukhin pushed a commit that referenced this issue Aug 9, 2021
@sergepetrenko @kyukhin
replication: always send raft state to subscribers
8de012b 
Tarantool used to send out raft state on subscribe only when raft was
enabled. This was a safeguard against partially-upgraded clusters, where
some nodes had no clue about Raft messages and couldn't handle them
properly.

Actually, Raft state should be sent out always. For example, promote
will be changed to bump Raft term even when Raft is disabled, and it's
important that everyone in cluster has the same term for the sake of promote
at least.

So, send out Raft state to every subscriber with version >= 2.6.0
(that's when Raft was introduced).
Do the same for Raft broadcasts. They should be sent only to replicas
with version >= 2.6.0

Closes #5438

(cherry picked from commit 3bee3fe)
kyukhin pushed a commit that referenced this issue Aug 9, 2021
@sergepetrenko @kyukhin
replication: always send raft state to subscribers
0ea62ad 
Tarantool used to send out raft state on subscribe only when raft was
enabled. This was a safeguard against partially-upgraded clusters, where
some nodes had no clue about Raft messages and couldn't handle them
properly.

Actually, Raft state should be sent out always. For example, promote
will be changed to bump Raft term even when Raft is disabled, and it's
important that everyone in cluster has the same term for the sake of promote
at least.

So, send out Raft state to every subscriber with version >= 2.6.0
(that's when Raft was introduced).
Do the same for Raft broadcasts. They should be sent only to replicas
with version >= 2.6.0

Closes #5438
yanshtunder pushed a commit that referenced this issue Oct 4, 2021
@sergepetrenko @yanshtunder
replication: always send raft state to subscribers
118c86a 
Tarantool used to send out raft state on subscribe only when raft was
enabled. This was a safeguard against partially-upgraded clusters, where
some nodes had no clue about Raft messages and couldn't handle them
properly.

Actually, Raft state should be sent out always. For example, promote
will be changed to bump Raft term even when Raft is disabled, and it's
important that everyone in cluster has the same term for the sake of promote
at least.

So, send out Raft state to every subscriber with version >= 2.6.0
(that's when Raft was introduced).
Do the same for Raft broadcasts. They should be sent only to replicas
with version >= 2.6.0

Closes #5438
yanshtunder pushed a commit that referenced this issue Oct 4, 2021
@sergepetrenko @yanshtunder
replication: always send raft state to subscribers
2133bef 
Tarantool used to send out raft state on subscribe only when raft was
enabled. This was a safeguard against partially-upgraded clusters, where
some nodes had no clue about Raft messages and couldn't handle them
properly.

Actually, Raft state should be sent out always. For example, promote
will be changed to bump Raft term even when Raft is disabled, and it's
important that everyone in cluster has the same term for the sake of promote
at least.

So, send out Raft state to every subscriber with version >= 2.6.0
(that's when Raft was introduced).
Do the same for Raft broadcasts. They should be sent only to replicas
with version >= 2.6.0

Closes #5438
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@sergepetrenko sergepetrenko

Labels
bug Something isn't working qsync replication raft RAFT protocol replication
Projects
None yet
Milestone
2.7.3
Development

No branches or pull requests
3 participants
@Gerold103 @kyukhin @sergepetrenko