New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CloudRetry/AWSRetry backoff decorator with unit tests #17039

Merged
merged 11 commits into from Sep 13, 2016

Conversation

Projects
None yet
10 participants
@linuxdynasty
Contributor

linuxdynasty commented Aug 10, 2016

ISSUE TYPE
  • Feature Pull Request
COMPONENT NAME

CloudRetry Base class inside of module_utils/cloud.py
AWSRetry decorator inside of module_utils/ec2.py

ANSIBLE VERSION
ansible 2.1.1.0
SUMMARY

The CloudRetry class can be implemented by any other cloud provider, that wants to implement a basic backoff algorithm in a decorator.

AWSRetry class overwrites 2 methods.

  • status_code_from_exception (Parses the exception for the exact string)
  • found (Iterates over a list of exceptions to match against.

AWSRetry.backoff() decorator can be applied to any function that is making an aws boto/boto3 call.
It will only retry on the following Exceptions.
This list of failures is based on this API Reference.

  • RequestLimitExceeded
  • Unavailable
  • ServiceUnavailable
  • InternalFailure
  • InternalError
  • "\w+.NotFound" (Eventual Consistency)

The CloudRetry.backoff decorator takes on the following kwargs.

  • tries (number of times to try) default=10
  • delay (initial delay between retries in seconds) default=3
  • backoff (This is the multiplier, that will double on each retry) default=2

This change is Reviewable

@bcoca

This comment has been minimized.

Show comment
Hide comment
@bcoca

bcoca Aug 11, 2016

Member

so instead of re implementing from scratch, would it not be better to expand the existing retry functions to allow for the custom conditions?

https://github.com/ansible/ansible/blob/devel/lib/ansible/module_utils/api.py

Member

bcoca commented Aug 11, 2016

so instead of re implementing from scratch, would it not be better to expand the existing retry functions to allow for the custom conditions?

https://github.com/ansible/ansible/blob/devel/lib/ansible/module_utils/api.py

@linuxdynasty

This comment has been minimized.

Show comment
Hide comment
@linuxdynasty

linuxdynasty Aug 11, 2016

Contributor

@bcoca I read api.py before writing my own. This one is purely focused on AWS and not just on the RateLimiting issue. This one also gets around the NotFound" (Eventual Consistency) issue as well. Could we add an argument to the decorator in API to take in Exception Strings. Sure, but I think this way the writer of the modules will not have to think of the multiple use cases beyond RequestLimitExceeded. Also if an exception is not part of the list of retries, it will just raise it.

Contributor

linuxdynasty commented Aug 11, 2016

@bcoca I read api.py before writing my own. This one is purely focused on AWS and not just on the RateLimiting issue. This one also gets around the NotFound" (Eventual Consistency) issue as well. Could we add an argument to the decorator in API to take in Exception Strings. Sure, but I think this way the writer of the modules will not have to think of the multiple use cases beyond RequestLimitExceeded. Also if an exception is not part of the list of retries, it will just raise it.

@bcoca

This comment has been minimized.

Show comment
Hide comment
@bcoca

bcoca Aug 11, 2016

Member

@linuxdynasty i'm not saying modules should wrap the existing retries, i was suggesting this function could be re-implemented as a simple wrapper to an expanded 'generic retry'.

This would make 'cloud specific' retry functions much easier in general.

Member

bcoca commented Aug 11, 2016

@linuxdynasty i'm not saying modules should wrap the existing retries, i was suggesting this function could be re-implemented as a simple wrapper to an expanded 'generic retry'.

This would make 'cloud specific' retry functions much easier in general.

@linuxdynasty

This comment has been minimized.

Show comment
Hide comment
@linuxdynasty

linuxdynasty Aug 11, 2016

Contributor

I refactored it @bcoca. Now there is a base class called CloudRetry in module_utils/cloud.py. AWSRetry class in module_utils/ec2.py inherits from CloudRetry. Now only two staticmethods need to be overridden when another cloud provider wants to use this decorator. Test also updated.

Contributor

linuxdynasty commented Aug 11, 2016

I refactored it @bcoca. Now there is a base class called CloudRetry in module_utils/cloud.py. AWSRetry class in module_utils/ec2.py inherits from CloudRetry. Now only two staticmethods need to be overridden when another cloud provider wants to use this decorator. Test also updated.

@rodrickbrown

This comment has been minimized.

Show comment
Hide comment
@rodrickbrown

rodrickbrown Aug 11, 2016

+1 this looks good well needed feature request thanks @linuxdynasty

rodrickbrown commented Aug 11, 2016

+1 this looks good well needed feature request thanks @linuxdynasty

@linuxdynasty

This comment has been minimized.

Show comment
Hide comment
@linuxdynasty

linuxdynasty Aug 11, 2016

Contributor

Can we add botocore, boto3 to the ansible core testing framework. As one of the failures, is that botocore does not exist.

Contributor

linuxdynasty commented Aug 11, 2016

Can we add botocore, boto3 to the ansible core testing framework. As one of the failures, is that botocore does not exist.

@mattclay

This comment has been minimized.

Show comment
Hide comment
@mattclay

mattclay Aug 11, 2016

Member

@linuxdynasty You can add botocore and boto3 to test/utils/shippable/sanity-requirements.txt.

Member

mattclay commented Aug 11, 2016

@linuxdynasty You can add botocore and boto3 to test/utils/shippable/sanity-requirements.txt.

@mattclay

View changes

Show outdated Hide outdated lib/ansible/module_utils/cloud.py
@mattclay

This comment has been minimized.

Show comment
Hide comment
@mattclay

mattclay Aug 12, 2016

Member

@linuxdynasty My apologies, I had you put the dependencies in the wrong requirements file. You should add them to the two requirements files in the test/utils/tox directory instead.

Member

mattclay commented Aug 12, 2016

@linuxdynasty My apologies, I had you put the dependencies in the wrong requirements file. You should add them to the two requirements files in the test/utils/tox directory instead.

@mattclay

This comment has been minimized.

Show comment
Hide comment
@mattclay

mattclay Aug 12, 2016

Member

@linuxdynasty If you want to run the unit tests locally with an environment similar to Shippable, use the following command: TOXENV=py27 test/utils/shippable/sanity.sh

That will run the unit tests on Python 2.7. You can use other Python versions as well, if you have them installed.

Member

mattclay commented Aug 12, 2016

@linuxdynasty If you want to run the unit tests locally with an environment similar to Shippable, use the following command: TOXENV=py27 test/utils/shippable/sanity.sh

That will run the unit tests on Python 2.7. You can use other Python versions as well, if you have them installed.

@mattclay

View changes

Show outdated Hide outdated test/utils/shippable/sanity-requirements.txt
from ansible.module_utils.pycompat24 import get_exception
class CloudRetry(object):

This comment has been minimized.

@bcoca

bcoca Aug 12, 2016

Member

not exactly what i was asking, but good enough

@bcoca

bcoca Aug 12, 2016

Member

not exactly what i was asking, but good enough

This comment has been minimized.

@senorsmile

senorsmile Aug 16, 2016

I do think this could be useful for any module (not just cloud modules).

@senorsmile

senorsmile Aug 16, 2016

I do think this could be useful for any module (not just cloud modules).

@linuxdynasty linuxdynasty changed the title from aws_retry decorator function with unit tests to CloudRetry/AWSRetry decorator with unit tests Aug 12, 2016

@linuxdynasty linuxdynasty changed the title from CloudRetry/AWSRetry decorator with unit tests to CloudRetry/AWSRetry backoff decorator with unit tests Aug 12, 2016

@linuxdynasty

This comment has been minimized.

Show comment
Hide comment
@linuxdynasty

linuxdynasty Aug 12, 2016

Contributor

@mattclay thank you for all your help. Test are finally passing :D

Contributor

linuxdynasty commented Aug 12, 2016

@mattclay thank you for all your help. Test are finally passing :D

@linuxdynasty

This comment has been minimized.

Show comment
Hide comment
@linuxdynasty

linuxdynasty Aug 14, 2016

Contributor

tests are failing, but not do to this PR.

Contributor

linuxdynasty commented Aug 14, 2016

tests are failing, but not do to this PR.

@mattclay

This comment has been minimized.

Show comment
Hide comment
@mattclay

mattclay Aug 14, 2016

Member

I've restarted the build. The failure was due to a temporary timeout.

Member

mattclay commented Aug 14, 2016

I've restarted the build. The failure was due to a temporary timeout.

@AWSRetry.backoff(tries=2, delay=0.1)
def retry_once():
self.counter += 1
if self.counter < 2:

This comment has been minimized.

@fearphage

fearphage Aug 16, 2016

These tests would be a lot cleaner using Mock.

@fearphage

fearphage Aug 16, 2016

These tests would be a lot cleaner using Mock.

This comment has been minimized.

@bcoca

bcoca Aug 16, 2016

Member

Which is why the original 'retry' is in api.py, I might migrate this after merge

@bcoca

bcoca Aug 16, 2016

Member

Which is why the original 'retry' is in api.py, I might migrate this after merge

@linuxdynasty

This comment has been minimized.

Show comment
Hide comment
@linuxdynasty

linuxdynasty Aug 18, 2016

Contributor

@bcoca any updates on if you want me to move CloudRetry from cloud.py to api.py and still leave AWSRetry in ec2.py?

Contributor

linuxdynasty commented Aug 18, 2016

@bcoca any updates on if you want me to move CloudRetry from cloud.py to api.py and still leave AWSRetry in ec2.py?

@ryansb

View changes

Show outdated Hide outdated lib/ansible/module_utils/cloud.py
@linuxdynasty

This comment has been minimized.

Show comment
Hide comment
@linuxdynasty

linuxdynasty Aug 24, 2016

Contributor

@ryansb I pushed the changes. The build failed cause it could not spin a container.

Contributor

linuxdynasty commented Aug 24, 2016

@ryansb I pushed the changes. The build failed cause it could not spin a container.

Updated default backoff from 2 seconds to 1.1s.
This will be about a total of 48 seconds in 10 tries. This is
configurable.
@linuxdynasty

This comment has been minimized.

Show comment
Hide comment
@linuxdynasty

linuxdynasty Aug 30, 2016

Contributor

@ryansb does everything else look good to you?

Contributor

linuxdynasty commented Aug 30, 2016

@ryansb does everything else look good to you?

@ryansb ryansb merged commit b510abc into ansible:devel Sep 13, 2016

1 check passed

Shippable Run 2487 status is SUCCESS.
Details

sereinity added a commit to sereinity/ansible that referenced this pull request Jan 25, 2017

CloudRetry/AWSRetry backoff decorator with unit tests (#17039)
* Added aws_retry decorator function with unit tests

* Restructured the code to be used with a base class.

This base class CloudRetry can be reused by any other cloud provider.
This decorator should be used in situations, where you need to implement
a backoff algorithm and want to retry based on the status code from the
exception.

* updated documentation

* fixed tabs

* added botocore and boto3 to requirements.txt

* removed cloud.py from py24 tests, as it depends on boto3

* fix relative imports

* updated test to be 2.6 compat

* updated method name from retry to backoff

* readded lxd

* Updated default backoff from 2 seconds to 1.1s.

This will be about a total of 48 seconds in 10 tries. This is
configurable.

@ansibot ansibot added feature and removed feature_pull_request labels Mar 4, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment