Cosmos DB components should handle throttling

[ItalyPaleAle]

Expected Behavior

While making requests to Cosmos DB, in case of 429 responses (throttling), the components should automatically retry the requests after a back-off.

Actual Behavior

While making requests with the Cosmos DB components (state store and binding), if the service throttles the requests, the components just fail with an error without re-trying.

Release Note

RELEASE NOTE: FIX Cosmos DB components now handle throttling from database

[ItalyPaleAle]

/assign

[berndverst]

The 429 on init (due to metadata rate limit exceeded) are already being handled but operations are not.

The idea was it could be handled by resiliency. Of course we could add additional retry on 429 for operations directly in the component also.

Please do include the randomness factor provided by the cenkalti/backoff library -- see the retry on init for inspiration.

[ItalyPaleAle]

As per discussion, I think that re-trying on a 429 should be a core part of components, something that should be table stakes, and we should not expose that complexity to end-users. Most other components (including Redis, those who use the official Azure SDKs, etc) already handle 429's automatically. This should be the case for Cosmos DB too.

Agree on mirroring what's being done for init.

[armandvanderwalt]

Was this change reverted? Looks like automatic retries are not working when using cosmos db as a state store