Do not reset snapshot replication when a single request timedout

[deepthidevaki]

Description

Previously, when replicating a snapshot chunk timedout, leader restarts snapshot replication from the first chunk. This can result in a never ending retry loop if one of the other chunk hits a timeout frequently. This can happen more frequent when network latency is high and/or snapshot is large. To ensure that snapshot replication can complete in such cases,

1. Leader do not reset snapshot replication, instead re-send the same chunk if it encounters a timeout
2. Follower accepts the chunk even if it is a duplicate.

Tests are added where we can inject timeout and observer the behavior. RaftRandomizedTest also covers snapshot replication.

Related issues

closes #11496

Definition of Done

Not all items need to be done depending on the issue and the pull request.

Code changes:

• The changes are backwards compatibility with previous versions
• If it fixes a bug then PRs are created to backport the fix to the last two minor versions. You can trigger a backport by assigning labels (e.g. backport stable/1.3) to the PR, in case that fails you need to create backports manually.

Testing:

• There are unit/integration tests that verify all acceptance criterias of the issue
• New tests are written to ensure backwards compatibility with further versions
• The behavior is tested manually
• The change has been verified by a QA run
• The impact of the changes is verified by a benchmark

Documentation:

• The documentation is updated (e.g. BPMN reference, configuration, examples, get-started guides, etc.)
• If the PR changes how BPMN processes are validated (e.g. support new BPMN element) then the Camunda modeling team should be informed to adjust the BPMN linting.

Other teams:
If the change impacts another team an issue has been created for this team, explaining what they need to do to support this change.

• Tasklist
• Web Modeler
• Desktop Modeler
• Optimize

Please refer to our review guidelines.

[npepinpe]

🚀

[npepinpe]

❓ What were the case where it would be the cause? Do you think it could also be more deeply nested?

[deepthidevaki]

error is usually CompletionException so we always have to get the cause. I don't think it can be more deeply nested.

[deepthidevaki]

This was not backported to older versions. But there was an incident today for which this would have helped. I don't see any reason not to backport this, although original issue was tagged as an improvement.

I'm backporting this. /cc @npepinpe

[deepthidevaki]

/backport

[backport-action]

Created backport PR for stable/8.2:

• [Backport stable/8.2] Do not reset snapshot replication when a single request timedout #19028 with remaining conflicts!

Please cherry-pick the changes locally and resolve any conflicts.

git fetch origin backport-16971-to-stable/8.2
git worktree add --checkout .worktree/backport-16971-to-stable/8.2 backport-16971-to-stable/8.2
cd .worktree/backport-16971-to-stable/8.2
git reset --hard HEAD^
git cherry-pick -x 281eb95c0f593c2cc4ffecc873f3738b895bd09b d32d65050d5c8a498ec6a60ddfa5b8dc3eb3ce89 be3c7d703e8ac9c84c3bf2ada4efcde1e39ba5da c16f3c8c9f66fdae85ca800fac4b978546d4f566 13e8d923c8892c03a155615eebc573bc96f079e4 fc7828a67a8cbe5b225abc83eb1d234c8502947e 18e6598d84e3944d10538ec6fe2813900ba7fe71 05ed2dce937338e936387fa6345993b78b14498b dd76387d68891fe97266a671cb4a9bc247610b5e
git push --force-with-lease

[backport-action]

Created backport PR for stable/8.3:

• [Backport stable/8.3] Do not reset snapshot replication when a single request timedout #19029 with remaining conflicts!

Please cherry-pick the changes locally and resolve any conflicts.

git fetch origin backport-16971-to-stable/8.3
git worktree add --checkout .worktree/backport-16971-to-stable/8.3 backport-16971-to-stable/8.3
cd .worktree/backport-16971-to-stable/8.3
git reset --hard HEAD^
git cherry-pick -x 281eb95c0f593c2cc4ffecc873f3738b895bd09b d32d65050d5c8a498ec6a60ddfa5b8dc3eb3ce89 be3c7d703e8ac9c84c3bf2ada4efcde1e39ba5da c16f3c8c9f66fdae85ca800fac4b978546d4f566 13e8d923c8892c03a155615eebc573bc96f079e4 fc7828a67a8cbe5b225abc83eb1d234c8502947e 18e6598d84e3944d10538ec6fe2813900ba7fe71 05ed2dce937338e936387fa6345993b78b14498b dd76387d68891fe97266a671cb4a9bc247610b5e
git push --force-with-lease

[backport-action]

Created backport PR for stable/8.4:

• [Backport stable/8.4] Do not reset snapshot replication when a single request timedout #19030 with remaining conflicts!

Please cherry-pick the changes locally and resolve any conflicts.

git fetch origin backport-16971-to-stable/8.4
git worktree add --checkout .worktree/backport-16971-to-stable/8.4 backport-16971-to-stable/8.4
cd .worktree/backport-16971-to-stable/8.4
git reset --hard HEAD^
git cherry-pick -x 281eb95c0f593c2cc4ffecc873f3738b895bd09b d32d65050d5c8a498ec6a60ddfa5b8dc3eb3ce89 be3c7d703e8ac9c84c3bf2ada4efcde1e39ba5da c16f3c8c9f66fdae85ca800fac4b978546d4f566 13e8d923c8892c03a155615eebc573bc96f079e4 fc7828a67a8cbe5b225abc83eb1d234c8502947e 18e6598d84e3944d10538ec6fe2813900ba7fe71 05ed2dce937338e936387fa6345993b78b14498b dd76387d68891fe97266a671cb4a9bc247610b5e
git push --force-with-lease