CloudRetry/AWSRetry backoff decorator with unit tests #17039

linuxdynasty · 2016-08-10T23:40:30Z

ISSUE TYPE

Feature Pull Request

COMPONENT NAME

CloudRetry Base class inside of module_utils/cloud.py
AWSRetry decorator inside of module_utils/ec2.py

ANSIBLE VERSION

ansible 2.1.1.0

SUMMARY

The CloudRetry class can be implemented by any other cloud provider, that wants to implement a basic backoff algorithm in a decorator.

AWSRetry class overwrites 2 methods.

status_code_from_exception (Parses the exception for the exact string)
found (Iterates over a list of exceptions to match against.

AWSRetry.backoff() decorator can be applied to any function that is making an aws boto/boto3 call.
It will only retry on the following Exceptions.
This list of failures is based on this API Reference.

RequestLimitExceeded
Unavailable
ServiceUnavailable
InternalFailure
InternalError
"\w+.NotFound" (Eventual Consistency)

The CloudRetry.backoff decorator takes on the following kwargs.

tries (number of times to try) default=10
delay (initial delay between retries in seconds) default=3
backoff (This is the multiplier, that will double on each retry) default=2

This change is

bcoca · 2016-08-11T02:54:32Z

so instead of re implementing from scratch, would it not be better to expand the existing retry functions to allow for the custom conditions?

https://github.com/ansible/ansible/blob/devel/lib/ansible/module_utils/api.py

linuxdynasty · 2016-08-11T14:14:38Z

@bcoca I read api.py before writing my own. This one is purely focused on AWS and not just on the RateLimiting issue. This one also gets around the NotFound" (Eventual Consistency) issue as well. Could we add an argument to the decorator in API to take in Exception Strings. Sure, but I think this way the writer of the modules will not have to think of the multiple use cases beyond RequestLimitExceeded. Also if an exception is not part of the list of retries, it will just raise it.

bcoca · 2016-08-11T14:33:06Z

@linuxdynasty i'm not saying modules should wrap the existing retries, i was suggesting this function could be re-implemented as a simple wrapper to an expanded 'generic retry'.

This would make 'cloud specific' retry functions much easier in general.

linuxdynasty · 2016-08-11T19:25:31Z

I refactored it @bcoca. Now there is a base class called CloudRetry in module_utils/cloud.py. AWSRetry class in module_utils/ec2.py inherits from CloudRetry. Now only two staticmethods need to be overridden when another cloud provider wants to use this decorator. Test also updated.

rodrickbrown · 2016-08-11T19:27:40Z

+1 this looks good well needed feature request thanks @linuxdynasty

linuxdynasty · 2016-08-11T20:50:56Z

Can we add botocore, boto3 to the ansible core testing framework. As one of the failures, is that botocore does not exist.

mattclay · 2016-08-11T23:47:51Z

@linuxdynasty You can add botocore and boto3 to test/utils/shippable/sanity-requirements.txt.

mattclay · 2016-08-11T23:52:05Z

lib/ansible/module_utils/cloud.py

+                while max_tries > 1:
+                    try:
+                        return f(*args, **kwargs)
+                    except Exception as e:


This isn't compatible with Python 2.4, which is causing it to fail the py24 tests:

2016-08-11 20:25:51 + python2.4 -m compileall -fq -x 'module_utils/(a10|rax|openstack|ec2|gce|docker_common|azure_rm_common|vca|vmware|gcp|gcdns).py' lib/ansible/module_utils 2016-08-11 20:25:51 Compiling lib/ansible/module_utils/cloud.py ... 2016-08-11 20:25:51 File "lib/ansible/module_utils/cloud.py", line 86 2016-08-11 20:25:51 except Exception as e: 2016-08-11 20:25:51 ^ 2016-08-11 20:25:51 SyntaxError: invalid syntax

You can use this instead:

except Exception: e = get_exception()

mattclay · 2016-08-12T04:02:12Z

@linuxdynasty My apologies, I had you put the dependencies in the wrong requirements file. You should add them to the two requirements files in the test/utils/tox directory instead.

mattclay · 2016-08-12T04:09:26Z

@linuxdynasty If you want to run the unit tests locally with an environment similar to Shippable, use the following command: TOXENV=py27 test/utils/shippable/sanity.sh

That will run the unit tests on Python 2.7. You can use other Python versions as well, if you have them installed.

mattclay · 2016-08-12T04:30:08Z

test/utils/shippable/sanity-requirements.txt

@@ -2,3 +2,5 @@ tox
 pyyaml
 jinja2
 setuptools
+botocore
+boto3


Changes to this file aren't needed since the dependencies are in the test/utils/tox requirements files.

linuxdynasty · 2016-08-12T15:09:46Z

@mattclay thank you for all your help. Test are finally passing :D

linuxdynasty · 2016-08-14T21:48:05Z

tests are failing, but not do to this PR.

mattclay · 2016-08-14T22:22:41Z

I've restarted the build. The failure was due to a temporary timeout.

fearphage · 2016-08-16T00:20:16Z

test/units/module_utils/ec2/test_aws.py

+        @AWSRetry.backoff(tries=2, delay=0.1)
+        def retry_once():
+            self.counter += 1
+            if self.counter < 2:


These tests would be a lot cleaner using Mock.

Which is why the original 'retry' is in api.py, I might migrate this after merge

linuxdynasty · 2016-08-18T14:24:18Z

@bcoca any updates on if you want me to move CloudRetry from cloud.py to api.py and still leave AWSRetry in ec2.py?

ryansb · 2016-08-24T17:54:41Z

lib/ansible/module_utils/cloud.py

+        pass
+
+    @classmethod
+    def backoff(cls, tries=10, delay=3, backoff=2):


I'm not sure about these defaults - a delay of 3 and backoff of 2 for 10 tries would mean that, to fail, this retry decorator would wait for 3069 seconds (3 + 3*2 + 3*2*2 ...., or sum([3 * 2**i for i in range(10)])) or about 50 minutes. That seems like a really long time, especially since most modules make several calls.

A better default might be 4 tries, for a total default wait time of 45 seconds and having a max of, say, a minute between tries. That way, if someone wanted 10 tries it would only take about 7.5 minutes to fail.

Makes sense and I will update it shortly.

I'm thinking of changing the default backoff of 2 seconds to 1.1 seconds. I think this is more reasonable. sum([3 * 1.1**i for i in range(10)])
About a total of 48 seconds in 10 tries. @ryansb thoughts?

This should be configurable, because this isn't enough for the work I do with auto scale groups.

@jjshoe It is configurable, these values are just the defaults that will be used if the developer calls the decorator without arguments.

@linuxdynasty That seems reasonable to me - go for it. Should be plenty to wait out most rate limits.

This base class CloudRetry can be reused by any other cloud provider. This decorator should be used in situations, where you need to implement a backoff algorithm and want to retry based on the status code from the exception.

linuxdynasty · 2016-08-24T22:04:21Z

@ryansb I pushed the changes. The build failed cause it could not spin a container.

This will be about a total of 48 seconds in 10 tries. This is configurable.

linuxdynasty · 2016-08-30T19:26:35Z

@ryansb does everything else look good to you?

* Added aws_retry decorator function with unit tests * Restructured the code to be used with a base class. This base class CloudRetry can be reused by any other cloud provider. This decorator should be used in situations, where you need to implement a backoff algorithm and want to retry based on the status code from the exception. * updated documentation * fixed tabs * added botocore and boto3 to requirements.txt * removed cloud.py from py24 tests, as it depends on boto3 * fix relative imports * updated test to be 2.6 compat * updated method name from retry to backoff * readded lxd * Updated default backoff from 2 seconds to 1.1s. This will be about a total of 48 seconds in 10 tries. This is configurable.

linuxdynasty force-pushed the aws_retry_decorator branch from efba8c6 to 6c0e46c Compare August 11, 2016 19:23

linuxdynasty force-pushed the aws_retry_decorator branch from c82e70e to 7807dcc Compare August 11, 2016 19:50

mattclay reviewed Aug 11, 2016
View reviewed changes

nitzmahone added feature_pull_request aws labels Aug 12, 2016

linuxdynasty force-pushed the aws_retry_decorator branch from a9de44e to 1350fe2 Compare August 12, 2016 04:17

mattclay reviewed Aug 12, 2016
View reviewed changes

linuxdynasty changed the title ~~aws_retry decorator function with unit tests~~ CloudRetry/AWSRetry decorator with unit tests Aug 12, 2016

linuxdynasty changed the title ~~CloudRetry/AWSRetry decorator with unit tests~~ CloudRetry/AWSRetry backoff decorator with unit tests Aug 12, 2016

linuxdynasty mentioned this pull request Aug 14, 2016

Retrieve facts from Amazon Certificate Manager. ansible/ansible-modules-extras#2718

Closed

linuxdynasty force-pushed the aws_retry_decorator branch from ba62d5b to 519a504 Compare August 14, 2016 21:05

This was referenced Aug 15, 2016

Updated ec2_vpc_nat_gateway to use AWSRetry decorator ansible/ansible-modules-extras#2721

Closed

multi-file custom modules (with ModuleReplacer), outside /ansible/module... #10274

Closed

fearphage reviewed Aug 16, 2016
View reviewed changes

linuxdynasty force-pushed the aws_retry_decorator branch from 519a504 to 5b5c415 Compare August 18, 2016 13:56

ryansb reviewed Aug 24, 2016
View reviewed changes

linuxdynasty added 10 commits August 24, 2016 14:16

Added aws_retry decorator function with unit tests

0826d2f

Restructured the code to be used with a base class.

68835c1

This base class CloudRetry can be reused by any other cloud provider. This decorator should be used in situations, where you need to implement a backoff algorithm and want to retry based on the status code from the exception.

updated documentation

0c6923a

fixed tabs

ff73ab2

added botocore and boto3 to requirements.txt

67edc92

removed cloud.py from py24 tests, as it depends on boto3

c741cdc

fix relative imports

c526a8d

updated test to be 2.6 compat

28218fb

updated method name from retry to backoff

4e9f32e

readded lxd

02ffc76

linuxdynasty force-pushed the aws_retry_decorator branch from d048b06 to f807b6c Compare August 24, 2016 21:19

Updated default backoff from 2 seconds to 1.1s.

1acdd4e

This will be about a total of 48 seconds in 10 tries. This is configurable.

linuxdynasty force-pushed the aws_retry_decorator branch from f807b6c to 1acdd4e Compare August 24, 2016 22:14

ryansb merged commit b510abc into ansible:devel Sep 13, 2016

smiller171 mentioned this pull request Oct 11, 2017

ec2_elb: Failing with boto exception ( 400 - throttling ) #30229

Closed

ansibot added feature This issue/PR relates to a feature request. and removed feature_pull_request labels Mar 4, 2018

ansible locked and limited conversation to collaborators Apr 26, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CloudRetry/AWSRetry backoff decorator with unit tests #17039

CloudRetry/AWSRetry backoff decorator with unit tests #17039

linuxdynasty commented Aug 10, 2016 •

edited

Loading

bcoca commented Aug 11, 2016 •

edited

Loading

linuxdynasty commented Aug 11, 2016 •

edited

Loading

bcoca commented Aug 11, 2016

linuxdynasty commented Aug 11, 2016 •

edited

Loading

rodrickbrown commented Aug 11, 2016

linuxdynasty commented Aug 11, 2016

mattclay commented Aug 11, 2016

mattclay Aug 11, 2016

mattclay commented Aug 12, 2016

mattclay commented Aug 12, 2016

mattclay Aug 12, 2016

linuxdynasty commented Aug 12, 2016

linuxdynasty commented Aug 14, 2016

mattclay commented Aug 14, 2016

fearphage Aug 16, 2016

bcoca Aug 16, 2016

linuxdynasty commented Aug 18, 2016 •

edited

Loading

ryansb Aug 24, 2016

linuxdynasty Aug 24, 2016

linuxdynasty Aug 24, 2016

jjshoe Aug 24, 2016

ryansb Aug 24, 2016

linuxdynasty commented Aug 24, 2016

linuxdynasty commented Aug 30, 2016

CloudRetry/AWSRetry backoff decorator with unit tests #17039

CloudRetry/AWSRetry backoff decorator with unit tests #17039

Conversation

linuxdynasty commented Aug 10, 2016 • edited Loading

ISSUE TYPE

COMPONENT NAME

ANSIBLE VERSION

SUMMARY

bcoca commented Aug 11, 2016 • edited Loading

linuxdynasty commented Aug 11, 2016 • edited Loading

bcoca commented Aug 11, 2016

linuxdynasty commented Aug 11, 2016 • edited Loading

rodrickbrown commented Aug 11, 2016

linuxdynasty commented Aug 11, 2016

mattclay commented Aug 11, 2016

Choose a reason for hiding this comment

mattclay commented Aug 12, 2016

mattclay commented Aug 12, 2016

Choose a reason for hiding this comment

linuxdynasty commented Aug 12, 2016

linuxdynasty commented Aug 14, 2016

mattclay commented Aug 14, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

linuxdynasty commented Aug 18, 2016 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

linuxdynasty commented Aug 24, 2016

linuxdynasty commented Aug 30, 2016

linuxdynasty commented Aug 10, 2016 •

edited

Loading

bcoca commented Aug 11, 2016 •

edited

Loading

linuxdynasty commented Aug 11, 2016 •

edited

Loading

linuxdynasty commented Aug 11, 2016 •

edited

Loading

linuxdynasty commented Aug 18, 2016 •

edited

Loading