exception introspection to determine retry

[kapilt]

in some cases to know if a backoff is retryable vs perm error, we need to inspect the exception instance to match against retryable error codes.

[bgreen-litl]

Maybe we could add a giveup_predicate keyword arg or similar to on_exception which would accept a function which takes the exception and returns True to give up immediately. Something like:

def is_fatal(http_error):
    return http_error.code == 404  # or whatever

@backoff.on_exception(backoff.expo,
                      urllib.error.HTTPError,
                      max_tries=8,
                      giveup_predicate=is_fatal)
def get_url(url):
    r = urllib.request.urlopen(url)
    return r.read()

Does this sound like it would work? Is it general enough? Is the issue limited to determining when to retry/give up?

Another solution, using the current API, might be to wrap the exception raising code with a function such that you're dealing with values rather than exceptions. Then backoff.on_predicate can be useful for this. For example, we have some code which looks like:

def keep_trying(resp):
    # keep retrying as long as the response was not OK and
    # it's a status code for which it makes sense to retry
    return (not resp.ok and
            resp.status_code in [408, 422, 429] or
            resp.status_code >= 500)

@backoff.on_predicate(backoff.expo, keep_trying, max_tries=8)
@backoff.on_exception(backoff.expo,
                      exceptions.RequestException,
                      max_tries=8)
def get_url(url):
    return requests.get(url)

[bgreen-litl]

@kapilt This PR adds a giveup kwarg to on_exception: #13

[bgreen-litl]

This feature is now released in backoff==1.3.1