Speculative processing #34686

mbhaskar · 2023-04-27T08:13:59Z

Description

This PR adds support for speculative processing on top of end-to-end timeout to improve availability within timeout

Introduces Threshold based speculative execution

Only for Reads / Queries
If we don't get a successful response (either delayed response or error response) within the threshold value, we go to all preferred regions in parallel.
- We can try one region at a time, with some linear timeline.
- Whichever responds first, we will return, and cancel all the other ongoing requests.
Benefits:
1. Better chances of getting a response from either of those regions.
2. This provides Maximum availability at RUs / compute tradeoff (increased cost)

flowchart TD
    A[Request] -->B(Base Request with timeout policy)
    B --> Threshold{timespent < speculativeThreshold}
    Threshold -->| Yes | C{success <= EndToEndTimeout}
    Threshold --> | No | RemoteProcessing[Start Parallel remote requests to other regions]
    C -->|Yes| D[Return result]
    C -->|No response| E[Cancel and timeout]
    C --> |error response| F[Retry Flow]
    F --> G{timespent < EndToEndTimeout}
    G --> |No| E
    G --> |Yes| F

If speculative retries are enabled, based on the probability threshold Speculative retries may increase RU costs

When we have sparse data, the algo adds more variance around mean and this may increase the exploration even though the lower mean value is selected until the perf guarantee is increased. (The algo has proofs that this keeps cost under consideration and does only necessary exploration)

Current algo is tuned for 10sample requests. It can be tuned based on time/ number of requests and can be adjusted accordingly

Initial test resuls

Tests:

Read document 100 times

30s of delay faults on the local region

	P99 in ms	P95 in ms	P75 in ms
No Speculation	2078.12109375	2028.05859375	2016.05859375
Threshold based Speculation	1134.046875	616.015625	298.0

How to enable:

Following environment variables enable speculation when end to end timeout policy is specified
Integer value specifying the speculation type

0 - No speculation
1 - Threshold based speculation

COSMOS_SPECULATION_TYPE

The threshold int representing milli seconds
COSMOS_SPECULATION_THRESHOLD

The threshold step int representing milli seconds. This sets when speculation on subsequent regions is triggered
COSMOS_SPECULATION_THRESHOLD_STEP

example: Three regions r1, r2,r3 in preferred regions
Speculation threshold: 500
Specialization step: 100

At time t1 request to r1 is made
If there is no response in 500ms, request to r2 is made
If there is no response at 500+100ms, request to r3 is made

All SDK Contribution checklist:

The pull request does not introduce [breaking changes]
CHANGELOG is updated for new features, bug fixes or other significant changes.
I have read the contribution guidelines.

General Guidelines and Best Practices

Title of the pull request is clear and informative.
There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For more information on cleaning up the commits in your PR, see this page.

Testing Guidelines

Pull request includes test coverage for the included changes.

…led when the subscription gets cancelled. - Cleanup and refactoring

Adding more tests

…e-processing # Conflicts: # sdk/cosmos/azure-cosmos-tests/src/test/java/com/azure/cosmos/EndToEndTimeOutValidationTests.java # sdk/cosmos/azure-cosmos-tests/src/test/java/com/azure/cosmos/implementation/RxDocumentClientImplTest.java # sdk/cosmos/azure-cosmos-tests/src/test/java/com/azure/cosmos/implementation/caches/AsyncCacheNonBlockingTest.java # sdk/cosmos/azure-cosmos-tests/src/test/java/com/azure/cosmos/implementation/directconnectivity/BarrierRequestHelperTest.java # sdk/cosmos/azure-cosmos/src/main/java/com/azure/cosmos/CosmosAsyncClient.java # sdk/cosmos/azure-cosmos/src/main/java/com/azure/cosmos/CosmosClientBuilder.java # sdk/cosmos/azure-cosmos/src/main/java/com/azure/cosmos/implementation/AsyncDocumentClient.java # sdk/cosmos/azure-cosmos/src/main/java/com/azure/cosmos/implementation/RMResources.java # sdk/cosmos/azure-cosmos/src/main/java/com/azure/cosmos/implementation/RequestOptions.java # sdk/cosmos/azure-cosmos/src/main/java/com/azure/cosmos/implementation/RxDocumentClientImpl.java # sdk/cosmos/azure-cosmos/src/main/java/com/azure/cosmos/implementation/caches/AsyncCacheNonBlocking.java # sdk/cosmos/azure-cosmos/src/main/java/com/azure/cosmos/implementation/directconnectivity/ReplicatedResourceClient.java # sdk/cosmos/azure-cosmos/src/main/java/com/azure/cosmos/models/CosmosItemRequestOptions.java # sdk/cosmos/azure-cosmos/src/main/java/com/azure/cosmos/models/CosmosQueryRequestOptions.java

sdk/cosmos/azure-cosmos/pom.xml

sdk/cosmos/azure-cosmos/src/main/java/module-info.java

Fixing lint issues

...mos/azure-cosmos/src/main/java/com/azure/cosmos/implementation/RxDocumentServiceRequest.java

...c/main/java/com/azure/cosmos/implementation/directconnectivity/ReplicatedResourceClient.java

...mplementation/directconnectivity/speculativeprocessors/ThompsonSamplingBasedSpeculation.java

sdk/cosmos/azure-cosmos/src/main/java/com/azure/cosmos/models/CosmosQueryRequestOptions.java

Test cleanup Adding only ThresholdBasedSpeculation

mbhaskar · 2023-05-12T04:54:44Z

/azp run java - cosmos - tests

azure-pipelines · 2023-05-12T04:54:57Z

Azure Pipelines successfully started running 1 pipeline(s).

kushagraThapar

LGTM, thanks @mbhaskar

kushagraThapar · 2023-05-12T16:00:47Z

sdk/cosmos/azure-cosmos/src/main/java/com/azure/cosmos/CosmosDiagnostics.java

@@ -33,7 +33,7 @@
 */
 public final class CosmosDiagnostics {
    private static final Logger LOGGER = LoggerFactory.getLogger(CosmosDiagnostics.class);
-    private static final ObjectMapper OBJECT_MAPPER = new ObjectMapper();
+    static final ObjectMapper OBJECT_MAPPER = new ObjectMapper();


why not private anymore? Are we re-using this anywhere else?

Was using in tests to validate.

But could revert.

kushagraThapar · 2023-05-12T16:01:46Z

sdk/cosmos/azure-cosmos/src/main/java/com/azure/cosmos/implementation/Configs.java

+    }
+
+    public static int speculationThreshold() {
+        return getJVMConfigAsInt(SPECULATION_THRESHOLD, 500);


nit: we should define these default values in separate constant variables above like other methods.

kushagraThapar · 2023-05-12T16:06:40Z

...osmos/implementation/directconnectivity/speculativeprocessors/ThresholdBasedSpeculation.java

+
+    @Override
+    public void onResponseReceived(URI region, Duration latency) {
+        // unused


if it is unused, may be throw error if it gets called, something like not implemented exception. Can be done in a follow up PR.

kushagraThapar · 2023-05-12T16:07:18Z

sdk/cosmos/azure-cosmos/src/main/java/module-info.java

@@ -28,7 +28,7 @@
    requires java.logging;
 	requires HdrHistogram;

-	// public API surface area
+    // public API surface area


can revert this change,.

Will revert in follow up PR

xinlian12 · 2023-05-12T16:45:37Z

sdk/cosmos/azure-cosmos/src/main/java/com/azure/cosmos/implementation/Configs.java

@@ -300,6 +311,18 @@ public static int getSessionTokenMismatchMaximumBackoffTimeInMs() {
            DEFAULT_SESSION_TOKEN_MISMATCH_MAXIMUM_BACKOFF_TIME_IN_MILLISECONDS);
    }

+    public static int getSpeculationType() {
+        return getJVMConfigAsInt(SPECULATION_TYPE, 0);


should this by default be 1?

No speculation by default

xinlian12

LGTM, thanks

...zure-cosmos/src/main/java/com/azure/cosmos/implementation/DocumentServiceRequestContext.java

FabianMeiswinkel

LGTM - Thanks

mbhaskar · 2023-05-12T21:46:31Z

/check-enforcer override

kushagraThapar · 2023-05-12T21:49:53Z

/check-enforcer override

xinlian12 · 2023-05-12T21:56:49Z

/check-enforcer override

FabianMeiswinkel · 2023-05-12T21:57:56Z

/check-enforcer override

FabianMeiswinkel · 2023-05-12T22:03:08Z

/check-enforcer override

mbhaskar · 2023-05-12T22:05:01Z

/check-enforcer evaluate

kushagraThapar · 2023-05-12T22:12:16Z

/check-enforcer reset

kushagraThapar · 2023-05-12T22:13:40Z

/check-enforcer override

* end-end timeout implementation * cleanup * cleanup * Adding missing comments * Cleanup and fixing test * Adding more tests * - Moving asyncCache to background runnable so that it wont get cancelled when the subscription gets cancelled. - Cleanup and refactoring * Refactoring and clean up * Refactoring and clean up * Adding timeouts for writes. Adding more tests * Adding speculative execution support for read operations * Adding config options to speculation * Fixing issue with query where request is not cloning property * Shading apache math3 * Adding Portions license Fixing lint issues * refactoring * Test fix * Cleanup and refactoring Test cleanup Adding only ThresholdBasedSpeculation * changing default value * Removing unused test

mbhaskar added 11 commits April 19, 2023 09:31

end-end timeout implementation

c728cf3

cleanup

a0b28b1

cleanup

fdf144f

Adding missing comments

cbd223b

Cleanup and fixing test

95b2cb5

Adding more tests

4a7b0bf

- Moving asyncCache to background runnable so that it wont get cancel…

5637c13

…led when the subscription gets cancelled. - Cleanup and refactoring

Refactoring and clean up

ac6e87f

Refactoring and clean up

824688a

Adding timeouts for writes.

abbfb36

Adding more tests

Adding speculative execution support for read operations

5bb1627

github-actions bot added the Cosmos label Apr 27, 2023

mbhaskar changed the title ~~Mbhaskar/speculative processing~~ Speculative processing Apr 27, 2023

mbhaskar added 2 commits May 3, 2023 14:54

Adding config options to speculation

0b95bdc

mbhaskar marked this pull request as ready for review May 4, 2023 16:57

mbhaskar requested review from kushagraThapar, FabianMeiswinkel, kirankumarkolli, xinlian12, milismsft, aayush3011, simorenoh, jeet1995 and Pilchie as code owners May 4, 2023 16:57

mbhaskar commented May 4, 2023

View reviewed changes

sdk/cosmos/azure-cosmos/pom.xml Outdated Show resolved Hide resolved

mbhaskar commented May 4, 2023

View reviewed changes

sdk/cosmos/azure-cosmos/src/main/java/module-info.java Outdated Show resolved Hide resolved

mbhaskar added 3 commits May 4, 2023 15:17

Fixing issue with query where request is not cloning property

0ef7cb3

Shading apache math3

b9c8aea

Adding Portions license

82e608a

Fixing lint issues