ReplicaValidation

[xinlian12]

Task:
#28271
#25351
#27245

# Description
Customer has reported that during upgrade scenarios, they are experiencing increased request latency.
One of the reason being that during upgrade, replica still undergoing upgrade may still be returned back to SDK. As of today, the request will have 25% chance to hit the replica not ready yet, hence causing ConnectionTimeoutException, which contributes to the increased latency.

Solution:

• SDK will track the endpoint health based on client side metrics, and de-prioritize the replica which were marked as - unhealthy.
• SDK will validate the health of the replica by attempting to open connections to the backend. When SDK refresh addresses back from gateway for a partition, SDK is going to ONLY validate replicas were in unhealthy status by opening connection requests.
• It is best effort, which means:
  • 1. If the validation can not finish within 1 min by open connections, the de-prioritize will stop for certain status
  • 2. The selection of the replica will not be blocked by the validation process. For example, if a request needs to be sent to N replicas, and if there is only N-1 replica in good status, it will still go ahead selecting the Nth replica which needs validation
• It is opt in only for now, by setting System property COSMOS.REPLICA_ADDRESS_VALIDATION_ENABLED to true
• Added new fields in cosmos diagnostic for replica status

Replica status machine:

If replica validation is disabled, how the status transition happens:

If replica validation is enabled, how the status transition happens:

CosmosDiagnostic Change:

• replicaStatusList: it will capture the client side replica status when SDK making the replica selection to send the request.
  - rv in clientCfgs
  
  

Other Q&A:
1. The behavior of regional failover
The COSMOS.REPLICA_ADDRESS_VALIDATION_ENABLED will turn on replica validation for all regions the requests will be sent to. It will not proactively create connections to all partitions (different than openConnectionsAndInitCaches), it will only try to open connections to replicas which were marked as unhealthy as the request comes in.
2. How unhealthyPending will be transitioned to connected for higher level consistency level
It can be triggered by openConnectionRequest or RxDocumentServiceRequest.
As mentioned above, the selection of the replica will not be blocked by the validation process.
3. 408/503 handling
we are starting from internal 410&forceRefresh
4. What if the connection is closed due to idleConnectionTimeout configuration
The recommendation for customer would be to use idleConnectionTimeout 0 to use this feature.
There is no distinguishment in this PR.

Benchmark Tests:
RU: 100,000, Document Count: 100,000, Region: South Central US
Throughput:

P95 Latency:

P99 Latency:

P999 Latency:

Benchmark Tests:
RU: 6000, Document: 1000, Region: South Central US
Throughput:

P95 Latency:

P99 Latency:

P999 Latency:

Test22 upgrade test results:
CTL RunId: 789635e2-5219-4b9d-860f-e6f4bd3e4259
Pre-created documents: 10000, concurrency 10. Orange line: ReplicaValidation + #28270. Blue line: Master
P95 Latency:

P99 Latency:

Max Latency:

Following PR:

1. Add open connection validation for replicas in unknown status -> region will be limited by the openAsync region
2. Issue [Cosmos] ConnectionStateListener to invalidate the replica which is reset (not all replicas) #28270

[azure-sdk]

API change check

API changes are not detected in this pull request.

[FabianMeiswinkel]

LGTM Annie - great work - well structured implementation and PR description. Really appreciated!

[xinlian12]

/azp run java - cosmos - tests

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[xinlian12]

/azp run java - cosmos - tests

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[xinlian12]

/azp run java - cosmos - tests

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[xinlian12]

/azp run java - cosmos - tests

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[xinlian12]

Tested the following tests locally and passed:
openConnectionsAndInitCachesForDirectMode
orderbyContinuationOnUndefinedAndNull

[xinlian12]

/check-enforcer override