fix(raft): notify role change listener only when transition completed

[romansmirnov]

Description

• Removes calling ZeebePartition#onRoleChange() when registering the role change listener
• Raft notifies the role change listeners only when the transition is completed, and in the case of a leader transition, the initial entry gets committed
• Adjust some test cases to ensure that the actual test scenario is tested

Related issues

closes #7862

Definition of Done

Not all items need to be done depending on the issue and the pull request.

Code changes:

• The changes are backwards compatibility with previous versions
• If it fixes a bug then PRs are created to backport the fix to the last two minor versions. You can trigger a backport by assigning labels (e.g. backport stable/0.25) to the PR, in case that fails you need to create backports manually.

Testing:

• There are unit/integration tests that verify all acceptance criterias of the issue
• New tests are written to ensure backwards compatibility with further versions
• The behavior is tested manually
• The change has been verified by a QA run
• The impact of the changes is verified by a benchmark

Documentation:

• The documentation is updated (e.g. BPMN reference, configuration, examples, get-started guides, etc.)
• New content is added to the release announcement

[romansmirnov]

@deepthidevaki,

This PR implements the potential solution we have discussed to avoid calling the role change listener twice. While the solution works in general, I am still wondering if there are any alternative solutions you could think of?

Why am I asking? With changing the behavior of RaftContext#addRoleChangeListener(), i.e., submitting a command that runs with the Raft executor asynchronously which then adds the listener and executes the listener directly if no transition is currently ongoing, but now the semantic of some test cases might have changed. For example, the following test case:

https://github.com/camunda-cloud/zeebe/blob/b0413b9c03bbb8bce3b4e8f9c923e1b5343e9c85/atomix/cluster/src/test/java/io/atomix/raft/RaftTest.java#L344-L360

Before that change, it behaved pretty deterministically:

1. Create 3 Raft Servers so that a leader gets elected
2. Get the Leader
3. Add Role Change Listener (they are directly added to the set of listeners)
4. Let the leader step down
5. Await that a new leader gets elected and once the initial entry gets committed the listener is called

With the PR, now the behavior changes in a way that the listener gets called by the previous leader and not by the new leader. So while the test case itself is still fine - it still gets green, the code change potentially changes the semantic of this (and maybe other) test case(s).

Of course, I could now go through the test cases and adjust them accordingly 😄 But maybe there is an alternative you could think of?

[romansmirnov]

@deepthidevaki,

What do you think about the following approach? Basically, instead of pushing the case that the listener might be called twice to the Raft layer, the concern/issue should stay within the broker itself. Meaning, keep the way of registering and triggering the listener as it is in ZeebePartition, but handle the case more gracefully that a transition to the LEADER role is triggered twice here:

https://github.com/camunda-cloud/zeebe/blob/b0413b9c03bbb8bce3b4e8f9c923e1b5343e9c85/broker/src/main/java/io/camunda/zeebe/broker/system/partitions/impl/PartitionTransitionImpl.java#L64-L91

Basically, if the current role is equal to the role to transition to, then do nothing and return a completed future. In my opinion, this should not cause any issues also when calling follow-up listener like PartitionContext#notifyListenersOf...().

That way, we keep the "issue" and its handling in the broker and don't push it to the Raft layer.

What do you think?

[deepthidevaki]

Having a check in ZeebePartition to prevent multiple transitions to the same Role is a good idea. But I think this will not solve the issue completely. This will make sure that the second transition will not happen. But the main problem here is that the ZeebePartition is transitioning to Leader before InitialEntry is committed, because it actively queries Raft's current role, thus bypassing the listener notification. So in this case it can still happen that in the first transition, it reads the wrong lastEntry, because the entries are not committed yet. This will leads to the same error before the second transition happens. So we have to anyway remove the following call

 onRoleChange(context.getRaftPartition().getRole(), context.getRaftPartition().term());

from ZeebePartition#registerListenersAndTriggerRoleChange

[romansmirnov]

@deepthidevaki, true, you are right!

[deepthidevaki]

Of course, I could now go through the test cases and adjust them accordingly smile But maybe there is an alternative you could think of?

I cannot think of alternative solutions. Are there too many tests that needs to be adjusted because of this change?
It is ok to adjust test, but we should ensure that we are not breaking any critical guarantees.

[romansmirnov]

@deepthidevaki, thanks for your input and the quick discussion. Just a quick summary of what we have talked about:

• For now, we will not consider the following potential solution: Adding an additional guard in PartitionTransitionImpl#transitionTo() to abort the transition request when the current role is the same as the requested role. While in general, this guard could make sense it would require additional investigation to understand its impact to ensure that all Zeebe partition services are started correctly and only half of them.
• We will continue considering the solution when adding a role change listener to call the listener immediately if there is no ongoing transition. That way, the ZeebePartition does not need to trigger a role change on its own, and they will be notified once the transition is finished and in the case of the leader, the initial entry gets committed.

[romansmirnov]

@deepthidevaki,

I am done with the implementation of the fix. I tried to write a test case, therefore, but I couldn't find a way to reproduce the scenario in a test case and to control for example when the commit happens should happen. I think, the current test cases (i.e., the one that I adjusted a bit) implicitly ensure that the listener is notified correctly and especially in the case of a leader when the initial entry gets committed.

However, if you have an approach, I am happy to receive it!

[deepthidevaki]

🚀 LGTM.
May be we can also add a test to verify that the listener is notified after it is added, even if there was no role change. This is the semantics changed in this PR, right.

[romansmirnov]

@deepthidevaki, sure thing!

[deepthidevaki]

🚀 Thanks.

[romansmirnov]

bors merge

[ghost]

Build succeeded:

• continuous-integration/jenkins/branch

[github-actions]

Backport failed for stable/1.2, because it was unable to cherry-pick the commit(s).

Please cherry-pick the changes locally.

git fetch origin stable/1.2
git worktree add -d .worktree/backport-8285-to-stable/1.2 origin/stable/1.2
cd .worktree/backport-8285-to-stable/1.2
git checkout -b backport-8285-to-stable/1.2
ancref=$(git merge-base a36abef9d3d90ba000e169e03138bc4f4284fca1 9ae3cea7e69bdd8cf55b226e973c8933b9e6fe52)
git cherry-pick -x $ancref..9ae3cea7e69bdd8cf55b226e973c8933b9e6fe52