How and when exactly SDK makes primary read region switch?

[yar-shukan]

Hi,
Can I please get a clarification on this: https://docs.microsoft.com/en-us/azure/cosmos-db/high-availability#additional-information-on-read-region-outages
"The impacted region is automatically disconnected and will be marked offline. The Azure Cosmos DB SDKs will redirect read calls to the next available region in the preferred region list."

During some small issues we've seen in production with CosmosDB, I've seen retry patterns like these (so SDK was always sending requests to the affected region 1st, then retried to another region but I've never seen the cases SDK would switch region completely for 1st request. Usually, the e2e latency of such requests was high (like several 100s of ms to several seconds):

RequestStartTime: 2021-09-21T08:32:18.7390309Z, RequestEndTime: 2021-09-21T08:32:21.5446975Z,  Number of regions attempted:2
ResponseTime: 2021-09-21T08:32:20.5134282Z, StoreResult: StorePhysicalAddress: rntbd://<redacted1>.documents.azure.com:14001/apps/<guid1>/services/c534be01-5c1f-41cb-b368-b1f2fa0f8dc5/partitions/3b8087f1-8e3f-4f54-8e72-c59bb12cfe18/replicas/132762502484309495s/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: True, StatusCode: 410, SubStatusCode: 0, RequestCharge: 0, ItemLSN: -1, SessionToken: , UsingLocalLSN: True, TransportException: A client transport error occurred: The request timed out while waiting for a server response. (Time: 2021-09-21T08:32:20.5134282Z, activity ID: 3eef918d-c7be-46f8-82d0-2ca604f5369d, error code: ReceiveTimeout [0x0010], base error: HRESULT 0x80131500, URI: rntbd://<redacted1>.documents.azure.com:14001/apps/<guid1>/services/c534be01-5c1f-41cb-b368-b1f2fa0f8dc5/partitions/3b8087f1-8e3f-4f54-8e72-c59bb12cfe18/replicas/132762502484309495s/, connection: 10.218.0.178:11137 -> 13.69.226.10:14001, payload sent: True, CPU history: (2021-09-21T08:31:22.3127302Z 15.460), (2021-09-21T08:31:32.3196800Z 21.045), (2021-09-21T08:31:42.3212265Z 9.717), (2021-09-21T08:31:52.3209514Z 13.281), (2021-09-21T08:32:02.3094522Z 15.900), (2021-09-21T08:32:12.3110190Z 26.904), CPU count: 4), ResourceType: Document, OperationType: Read
ResponseTime: 2021-09-21T08:32:21.5290852Z, StoreResult: StorePhysicalAddress: rntbd://<redacted1>.documents.azure.com:14001/apps/<guid1>/services/c534be01-5c1f-41cb-b368-b1f2fa0f8dc5/partitions/3b8087f1-8e3f-4f54-8e72-c59bb12cfe18/replicas/132762502484309495s/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: True, StatusCode: 410, SubStatusCode: 0, RequestCharge: 0, ItemLSN: -1, SessionToken: , UsingLocalLSN: True, TransportException: A client transport error occurred: The request timed out while waiting for a server response. (Time: 2021-09-21T08:32:21.5290852Z, activity ID: 3eef918d-c7be-46f8-82d0-2ca604f5369d, error code: ReceiveTimeout [0x0010], base error: HRESULT 0x80131500, URI: rntbd://<redacted1>.documents.azure.com:14001/apps/<guid1>/services/c534be01-5c1f-41cb-b368-b1f2fa0f8dc5/partitions/3b8087f1-8e3f-4f54-8e72-c59bb12cfe18/replicas/132762502484309495s/, connection: 10.218.0.178:11137 -> 13.69.226.10:14001, payload sent: True, CPU history: (2021-09-21T08:31:22.3127302Z 15.460), (2021-09-21T08:31:32.3196800Z 21.045), (2021-09-21T08:31:42.3212265Z 9.717), (2021-09-21T08:31:52.3209514Z 13.281), (2021-09-21T08:32:02.3094522Z 15.900), (2021-09-21T08:32:12.3110190Z 26.904), CPU count: 4), ResourceType: Document, OperationType: Read
ResponseTime: 2021-09-21T08:32:21.5446975Z, StoreResult: StorePhysicalAddress: rntbd://<redacted2>.documents.azure.com:14119/apps/<guid2>/services/d192cc61-d768-4dfd-b7a9-accf71559eda/partitions/711989c0-f0a3-4fbf-90aa-88788e79343d/replicas/132755197351905377p/, LSN: 8899420928, GlobalCommittedLsn: 8899418977, PartitionKeyRangeId: 23, IsValid: True, StatusCode: 200, SubStatusCode: 0, RequestCharge: 1.05, ItemLSN: 8899404071, SessionToken: 13#8899419009#36=-1#3=-1#24=-1#7=-1#25=-1#12=-1#13=-1, UsingLocalLSN: True, TransportException: null, ResourceType: Document, OperationType: Read

Can please somebody explain:

• What will be the trigger for SDK to completely switch preferred read region ReadDocument/QueryPage requests, not to do initial request to 1st configured preferred read region?
• Does this mean SDK will switch 100% of requests to another available region?
• Will it still try the initially affected region?
• How does SDK determine the failback event when it should switch back?
• Does this applies to partial outages e.g. if some replica in partition or whole partition is down in 1 region?

[ealsur]

The diagnostics you are attaching are not for the V3 SDK (not for this repo), but align with https://docs.microsoft.com/en-us/azure/cosmos-db/sql/troubleshoot-sdk-availability#transient-connectivity-issues-on-tcp-protocol

That linked document describes the scenarios on which the SDK would do partial or complete failovers and cases when the SDK would go back to the original region if it becomes available again.

For specific error codes see: https://github.com/Azure/azure-cosmos-dotnet-v3/blob/master/Microsoft.Azure.Cosmos/src/ClientRetryPolicy.cs

If a particular partition is undergoing issues, normally these surface as TCP connectivity errors which eventually make that particular request retry on another region if possible (link above).

[yar-shukan]

It's not really crystal clear on nuance how exactly SDK detects "Regional outage" case
https://docs.microsoft.com/en-us/azure/cosmos-db/sql/troubleshoot-sdk-availability#regional-outage and how it will exactly look like.

Can there be a partial outage that results in a massive issue (e.g. let's say 10% of traffic) that looks as "Transient connectivity issues on TCP protocol" that won't be treated as "Regional outage"? I guess in this case SDK will still try 1st region without the full switch, won't it?

[ealsur]

The materialization of a regional outage depends on the language, so documenting all the potential error types is impractical in a conceptual document.

For C#, a regional outage is covered on HttpRequestException.

As for the partial outage, I do not know if that can happen, normally whenever a partition's health is having an issue, there are automations in place in the service to cover it. I cannot comment on the service internal mechanics. From the SDK perspective, if it's facing TCP connectivity errors due to network outage, Cosmos partition health issues or even the machine running the application, it will attempt that request on another region (not a full failover).