Join GitHub today
GitHub is home to over 50 million developers working together to host and review code, manage projects, and build software together.Sign up
CloudRetry/AWSRetry backoff decorator with unit tests #17039
CloudRetry Base class inside of module_utils/cloud.py
The CloudRetry class can be implemented by any other cloud provider, that wants to implement a basic backoff algorithm in a decorator.
AWSRetry class overwrites 2 methods.
AWSRetry.backoff() decorator can be applied to any function that is making an aws boto/boto3 call.
The CloudRetry.backoff decorator takes on the following kwargs.
so instead of re implementing from scratch, would it not be better to expand the existing retry functions to allow for the custom conditions?
@bcoca I read api.py before writing my own. This one is purely focused on AWS and not just on the RateLimiting issue. This one also gets around the NotFound" (Eventual Consistency) issue as well. Could we add an argument to the decorator in API to take in Exception Strings. Sure, but I think this way the writer of the modules will not have to think of the multiple use cases beyond RequestLimitExceeded. Also if an exception is not part of the list of retries, it will just raise it.
I refactored it @bcoca. Now there is a base class called CloudRetry in module_utils/cloud.py. AWSRetry class in module_utils/ec2.py inherits from CloudRetry. Now only two staticmethods need to be overridden when another cloud provider wants to use this decorator. Test also updated.
@linuxdynasty If you want to run the unit tests locally with an environment similar to Shippable, use the following command:
That will run the unit tests on Python 2.7. You can use other Python versions as well, if you have them installed.
This base class CloudRetry can be reused by any other cloud provider. This decorator should be used in situations, where you need to implement a backoff algorithm and want to retry based on the status code from the exception.
* Added aws_retry decorator function with unit tests * Restructured the code to be used with a base class. This base class CloudRetry can be reused by any other cloud provider. This decorator should be used in situations, where you need to implement a backoff algorithm and want to retry based on the status code from the exception. * updated documentation * fixed tabs * added botocore and boto3 to requirements.txt * removed cloud.py from py24 tests, as it depends on boto3 * fix relative imports * updated test to be 2.6 compat * updated method name from retry to backoff * readded lxd * Updated default backoff from 2 seconds to 1.1s. This will be about a total of 48 seconds in 10 tries. This is configurable.