[SDK-2687] Implement automatic rate-limit handling

[evansims]

Changes

This PR enables support for automatic retry of API requests that encounter a 429 rate-limit status response from the Auth0 API. It will retry, by default, up to 3 times, with a hard-coded cap of 10. It exponentially increases the delay between each request ("exponential backoff"), and uses a jitter technique to slightly randomize the variance of this delay.

• Adds a RestClientOptions object to carry a configuration throughout the Management calls of the SDK, from Auth0 instantiation down through to the RestClient calls. There was no existing way of gracefully applying RestClient options (telemetry, timeouts, and now retries) without having to write custom instantiation code. This can now all be done just from the Auth0 constructor.

Note: RestClientOptions is implemented in such a way as to provide this new cleaner configuration model without introducing breaking changes. All existing configuration arguments are still supported. If a RestClientOptions is not provided to RestClient during instantiation, one is created dynamically using any provided timeout or telemetry options passed as arguments. We should deprecate using the argument configurations in the future and transition developers toward using RestClientOptions directly.

• Updates Auth0 constructor to accept rest_options argument for a RestClientOptions configuration object. This is further passed on to the constructors for all the Management endpoint classes.
• Updates Blacklists, ClientGrants, Clients, Connections, CustomDomains, DeviceCredentials, EmailTemplates, Emails, Grants, Guardian, Hooks, Jobs, LogStreams, Logs, Organizations, ResourceServers, Roles, RulesConfigs, Rules, Stats, Tenants, Tickets, UserBlocks, UsersByEmail, and Users constructors to accept rest_options argument for a RestClientOptions configuration object.
• Updates RestClient constructor to accept options argument for a RestClientOptions configuration object.
• Updates RestClient.get() with automatic retry logic.
• Adds RestClient.MAX_REQUEST_RETRIES(), MAX_REQUEST_RETRY_JITTER(), MAX_REQUEST_RETRY_DELAY() and MIN_REQUEST_RETRY_DELAY() getters to read the hard limits from, to avoid using mutable values.
• Adds RestClient._metrics to aid in unit testing this new logic. This dict keeps track of the number of retry attempts, and the individual backoff windows applied to each retry.
• Adds RestClient._skip_sleep to avoid actually calling sleep() during unit tests and greatly slowing down the test process.
• Updates test_get_rate_limit_error and test_get_rate_limit_error_without_headers, and adds test_get_rate_limit_custom_retries, test_get_rate_limit_invalid_retries_below_min, test_get_rate_limit_invalid_retries_above_max and test_get_rate_limit_retries_use_exponential_backoff to unit tests to cover new functionality.
• Updates other tests to prefer reading from the new internal configuration model (although technically they would work without changes until the old model is deprecated.)

References

Please see the internal SDK-2653 epic on this topic.

Testing

Tests can be executed using python3 -m unittest -v.

• This change adds unit test coverage
• This change adds integration test coverage
• This change has been tested on the latest version of the platform/language or why not

Checklist

• I have read the Auth0 general contribution guidelines
• I have read the Auth0 Code of Conduct
• All existing and new tests complete without errors

[jimmyjames]

👍 LGTM, thanks for providing details on how this change is implemented to not introduce any breaking changes.

[daschi]

Thanks for adding this! I'm curious when the next release will be that will include the RestClientOptions?