fix flaky test DeploymentClusteredTest.shouldRedistributeDeploymentWhenDeploymentPartitionIsRestarted #19358
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
**Background** Not receiving an acknowledgement command back from the partition that processed the distributed command is an expected scenario. To resolve that in production, we have a `CommandRedistributor` implemented. Every 10 seconds (with and exponential backoff), re-distributor simply re-tries the distributions that the partition didn't receive an acknowledgement back. **Flakiness** The logic explained above is already flaky by its nature. Simply, the partition is sometimes expected **not** to receive an acknowledgement back. This is exactly what happens in the test case. Sometimes, acknowledgement from partition 3 is not retrieved by partition 1 (deployment partition). Since the assertion waits for 5 seconds before failing, the re-distributor cannot retry the distribution in a given timeframe. **Solution** As a solution to that, we removed the assertion for `CommandDistributionIntent.ACKNOWLEDGED`. It is because there is already a method called `clientRule.waitUntilDeploymentIsDone()` later in the test, and it verifies if the distribution is completed within (partitionCount * 10L) seconds. Within that time frame the re-distribution can be retried and the distribution can be finished. This solution doesn't eliminate the flakiness as a whole, but it reduces its likelihood. Also, there is no way to reduce flakiness unless we accept to wait forever until the distribution is finished. In addition, `processDefinitionKey` is renamed to `deploymentKey` as the deployment event's key is a deployment key. That key also added to distribution assertion since we might have other distributions with different keys during the deployment along with deployment distribution in the future.
berkaycanbc
added
backport stable/8.3
berkaycanbc
deleted the
bcan-17303-fix-flaky-DeploymentClusteredTest.shouldRedistributeDeploymentWhenDeploymentPartitionIsRestarted
branch
|
Git push to origin failed for stable/8.3 with exitcode 1 |
|
Successfully created backport PR for |
|
Git push to origin failed for stable/8.4 with exitcode 1 |
|
Successfully created backport PR for |
|
Git push to origin failed for stable/8.5 with exitcode 1 |
|
Successfully created backport PR for |
github-merge-queue bot
pushed a commit
that referenced
this pull request
Jun 14, 2024
…distributeDeploymentWhenDeploymentPartitionIsRestarted (#19369) # Description Backport of #19358 to `stable/8.3`. relates to #17303 original author: @berkaycanbc
github-merge-queue bot
pushed a commit
that referenced
this pull request
Jun 14, 2024
…distributeDeploymentWhenDeploymentPartitionIsRestarted (#19370) # Description Backport of #19358 to `stable/8.4`. relates to #17303 original author: @berkaycanbc
github-merge-queue bot
pushed a commit
that referenced
this pull request
Jun 14, 2024
…distributeDeploymentWhenDeploymentPartitionIsRestarted (#19371) # Description Backport of #19358 to `stable/8.5`. relates to #17303 original author: @berkaycanbc
github-actions
bot
added
version:8.3.13
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Background
Not receiving an acknowledgement command back from the partition that processed the distributed command is an expected scenario. To resolve that in production, we have a
CommandRedistributorimplemented. Every 10 seconds (with an exponential backoff), re-distributor simply re-tries the distributions that the partition didn't receive an acknowledgement back.Flakiness
The logic explained above is already flaky by its nature. Simply, the partition is sometimes expected not to receive an acknowledgement back. This is exactly what happens in the test case. Sometimes, acknowledgement from partition 3 is not retrieved by partition 1 (deployment partition). Since the assertion waits for 5 seconds before failing, the re-distributor cannot retry the distribution in a given timeframe.
Solution
As a solution to that, we removed the assertion for
CommandDistributionIntent.ACKNOWLEDGED. It is because there is already a method calledclientRule.waitUntilDeploymentIsDone()later in the test, and it verifies if the distribution is completed within (partitionCount * 10L) seconds. Within that time frame the re-distribution can be retried and the distribution can be finished. This solution doesn't eliminate the flakiness as a whole, but it reduces its likelihood. Also, there is no way to reduce flakiness unless we accept to wait forever until the distribution is finished.In addition,
processDefinitionKeyis renamed todeploymentKeyas the deployment event's key is a deployment key. That key also added to distribution assertion since we might have other distributions with different keys during the deployment along with deployment distribution in the future.Related issues
closes #17303