ClientRetryPolicy broken when executing stored procedure

[dkroehan]

Describe the bug
When executing a stored procedure ( e.g. bulk-delete.js) via AsyncDocumentClient and the request runs into rate limiting then the retry mechanism throws an error.

Here is the log output:

2019-05-14 12:32:38.129 ERROR [-,,,] 28224 --- [o-eventloop-4-2] c.m.a.c.rx.internal.ClientRetryPolicy    : locationEndpoint is null because ClientRetryPolicy::onBeforeRequest(.) is not invoked, probably request creation failed due to invalid options, serialization setting, etc.
2019-05-14 12:32:38.131 ERROR [-,,,] 28224 --- [o-eventloop-4-2] i.RenameCollectionAwareClientRetryPolicy : onBeforeSendRequest is not invoked, encountered failure due to request being null

com.microsoft.azure.cosmosdb.DocumentClientException: Message: {"Errors":["Request rate is large"]}
...

To Reproduce
Execute bulk-delete.js stored procedure on a large enough dataset so that the execution runs into execution bounds (rate limiting).
In my case there were 500 documents to delete and the partition was configured with 400 RU/s. In the first execution 345 documents were successfully deleted. The described error occurred after executing the stored procedure a second time.

Expected behavior
In version com.microsoft.azure:azure-cosmosdb:2.4.4 everything is working correctly.
I get the following log output:

2019-05-14 13:03:33.363  WARN [-,,,] 28633 --- [o-eventloop-4-3] c.m.a.c.r.i.ResourceThrottleRetryPolicy  : Operation will be retried after 8569 milliseconds. Current attempt 1, Cumulative delay PT8.569S Exception: {}

The retry is performed automatically and the remaining documents are getting deleted.

Actual behavior
Error is thrown (see log output above) and no retry is performed.

Environment summary
Affected SDK Version: com.microsoft.azure:azure-cosmosdb:2.4.5. (Same for the 3.0.0-beta releases)

[christopheranderson]

@kushagraThapar - have we looked at this?

[kushagraThapar]

@dkroehan We did find some issues with retry policies, but we have fixed them in the latest version. Can you please re-confirm this in the latest versions, v2.6.3 and v3.3.1 ?

[dkroehan]

@kushagraThapar We recently updated to SDK v3 and I just checked it for the latest v3.3.1

Unfortunately the issue is not fixed. Here is the error output I get:

11:19:35.814 [reactor-http-epoll-4] ERROR c.a.d.c.internal.ClientRetryPolicy - locationEndpoint is null because ClientRetryPolicy::onBeforeRequest(.) is not invoked, probably request creation failed due to invalid options, serialization setting, etc.
11:19:35.817 [reactor-http-epoll-4] ERROR c.a.d.c.i.RenameCollectionAwareClientRetryPolicy - onBeforeSendRequest is not invoked, encountered failure due to request being null
com.azure.data.cosmos.CosmosClientException: Message: {"Errors":["Request rate is large"]}

Looks the same as in the first post.

[kushagraThapar]

@dkroehan Thanks for checking it, I will investigate it this week.

[dkroehan]

@kushagraThapar Do you have any update on this issue?

[kushagraThapar]

@kushagraThapar Do you have any update on this issue?

@dkroehan sorry this issue somehow got dropped from our tracking. I will track it now onwards.
We will pick it up in January.

As I get it, you are facing this issue both on V2 and V3 ?

[kushagraThapar]

@dkroehan Since v3 SDK got moved from this repository to here: https://github.com/Azure/azure-sdk-for-java/tree/master/sdk/cosmos

I strongly suggest please create an issue here as well: https://github.com/Azure/azure-sdk-for-java/issues
You can mention me, and mention this issue as well. Its easy to triage issues there as there is a microsoft-bot which notifies us on new issue creations.
Once you create it there, I will make sure the fix gets ported to v2 as well.

[dkroehan]

@kushagraThapar Created the new issue. Feel free to close it here.

[kushagraThapar]

@kushagraThapar Created the new issue. Feel free to close it here.

Thanks, will close this one.