Channel health check improvement for cancelled requests #36225
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
API change check API changes are not detected in this pull request. |
…annelHealthCheckImprovementForCancelledRequests
jeet1995
requested review from
kushagraThapar,
FabianMeiswinkel,
kirankumarkolli,
xinlian12,
milismsft,
aayush3011,
simorenoh and
Pilchie
as code owners
|
/azp run java - cosmos - tests |
|
Azure Pipelines successfully started running 1 pipeline(s). |
kushagraThapar
approved these changes
Aug 4, 2023
...os/src/main/java/com/azure/cosmos/implementation/directconnectivity/GatewayAddressCache.java
Outdated
Show resolved
Hide resolved
...om/azure/cosmos/implementation/directconnectivity/rntbd/RntbdClientChannelHealthChecker.java
Show resolved
Hide resolved
...om/azure/cosmos/implementation/directconnectivity/rntbd/RntbdClientChannelHealthChecker.java
Outdated
Show resolved
Hide resolved
|
/azp run java - cosmos - tests |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
/check-enforcer override |
jeet1995
added a commit
that referenced
this pull request
Sep 13, 2023
* Attempt to reproduce Walmart issue. * Added a cancellation gate to RntbdClientChannelHealthChecker. * Refactorings. * Refactorings. * Added tests. * Refactorings. * Fixed replica validation to open only 1 connection to replica. * Attempt at fixing tests. * Attempt at fixing tests. * Attempt at fixing tests. * Updated CHANGELOG.md. * Refactorings. * Refactorings. * Reacting to review comments. --------- Co-authored-by: Azure SDK Bot <53356347+azure-sdk@users.noreply.github.com> (cherry picked from commit e618ba1)
jeet1995
added a commit
that referenced
this pull request
Sep 13, 2023
* Attempt to reproduce Walmart issue. * Added a cancellation gate to RntbdClientChannelHealthChecker. * Refactorings. * Refactorings. * Added tests. * Refactorings. * Fixed replica validation to open only 1 connection to replica. * Attempt at fixing tests. * Attempt at fixing tests. * Attempt at fixing tests. * Updated CHANGELOG.md. * Refactorings. * Refactorings. * Reacting to review comments. --------- Co-authored-by: Azure SDK Bot <53356347+azure-sdk@users.noreply.github.com> (cherry picked from commit e618ba1)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
PR features
Add channel health check gate when the channel sees request cancellations
In service-side node-freeze scenarios, connections / channels may not be severed thus causing the SDK to not detect unhealthiness on such channels, and with end-to-end timeout also configured for the request, such requests do not hit transit timeout either which could have otherwise helped deem the channel as unhealthy.
In this PR, the SDK will track request cancellations and read delays at the channel level and if these properties exceed certain values, then the channel is deemed unhealthy. If such cancellations on a channel are a recurring symptom, the replica associated with this channel is treated as
Unhealthyand is removed from the list of replicas to send a request to. This can help reduce the volume of timeouts seen due to end-to-end timeout configured.Replica validation optimization
Previously replica validation would be triggered for replicas in the
Unknownhealth status despite the replica not being used by a container in the warm-up flow. This would create unnecessary connection attempts which could cause spikes in resource usage. Such replicas are excluded.Local testing done
Unhealthy.Unhealthyresults in a background address refresh with theforceAddressflag astrue.