Initialization: Fixes the SDK to retry if the initialization fails

[j82w]

Pull Request Template

Description

The SDK has an initialization task to get the account information and other info. If this task fails with a DocumentClientException it was never being recreated. This is a problem because if a 408 is thrown or some other transient problem the SDK will always returned the cached failure and will not actually retry the request.

Solution:
A new initialization function factory was created. This allows the initialization task to be recreated. The EnsureValidClientAsync now calls a new method that is thread safe to get or create a new task if the existing one failed.

Type of change

Please delete options that are not relevant.

• [] Bug fix (non-breaking change which fixes an issue)
• [] New feature (non-breaking change which adds functionality)
• [] Breaking change (fix or feature that would cause existing functionality to not work as expected)
• [] This change requires a documentation update

Closing issues

To automatically close an issue: closes #2990

[johngallardo]

Thanks for addressing this issue. I have found that this can also cause problems for psuedo-transient errors like permissions. E.g., if you construct a CosmosClient before setting SQL RBAC permissions for the identity accessing CosmosDB, the failed authorization exception can be returned to clients until a fresh CosmosClient object is created. These manifest as persistent exceptions that contain something like:

One or more errors occurred. (Request blocked by Auth jgalla-cosmosdb-test : Request is blocked because principal [<redacted>] does not have required RBAC permissions to perform action [Microsoft.DocumentDB/databaseAccounts/readMetadata] on resource [/]. Learn more: https://aka.ms/cosmos-native-rbac.This could be because the user's group memberships were not present in the AAD token.