Generic: add retries when getting throttled by SWF API

[jbbarth]

At Botify we're getting to a point where sometimes we're being throttled by SWF API. This happened last week-end with a the rate-limit exceeded on a RespondDecisionTaskCompleted action, which is a bit unfortunate: the task did succeed, but because of the rate-limit, we retried it. Luckily this was only a 6 min task but still.

For the record the SWF rate-limit policy is described here: http://docs.aws.amazon.com/amazonswf/latest/developerguide/swf-dg-limits.html#throttle-limits-useast1

The error looks like this:

Traceback (most recent call last):
  File "/code/botify/backend/process/worker/base.py", line 225, in process_task
    self.complete(token, json.dumps(result, default=lambda v: repr(v)))
  File "/usr/local/lib/python2.7/dist-packages/swf/actors/worker.py", line 79, in complete
    raise ResponseError(e.body['message'])
ResponseError: Rate exceeded

After a quick test, throttling errors look like this:

>>> print e
SWFResponseError: 400 Bad Request
{u'message': u'Rate exceeded', u'__type': u'com.amazon.coral.availability#ThrottlingException'}

>>> print e.error_code
ThrottlingException

So we could add some retry logic via a decorator on most tasks if the exception has an error_code and error_code == "ThrottlingException"