step decorators: retry

[yaythomas]

Add a retry decorator, to allow this:

name: pypyr.steps.blah
in:
   blah: blahblah
retry:
  max: 3 # optional max no. of attempts. int. Defaults None (infinite).
  sleep: 0 # optional sleep between retries, in seconds. Decimals allowed. Defaults 0.
  retryOn: [BlahError, ValueError] # only retry these exceptions. Optional. Defaults None (retry any error).
  stopOn: [SeriousError, NoPointContinuingError] # stop retry on these exceptions.  Optional. Defaults None (don't abort retry on any error.)

retry keeps on looping over step until:

1. execute step run_step(context) returns success (doesn't raise error)
2. error raised back to the pypyr core.

Error raises back up to the pypyr core when:

1. max exhausted
2. any error in stopOn
3. retryOn is not None and error not in retryOn

retry should pump an int retryCounter into context on each retry iteration.

retryOn and stepOn deserves some thought: how to load exception dynamically - repr strings might well be too limiting input (e.g specific transient err msg that can't be predicted)? what if exception is outside stdlib? Prob just match on name, though this means if you had mymodule.ValueError that would also catch built-in ValueError...

try:
    raise ValueError()
except Exception as exception:
    assert type(exception).__name__ == 'ValueError'
    assert exception.__class__.__name__ == 'ValueError'

Implementation:
prob deserves its own class, similar to WhileDecorator. pypyr.dsl.RetryDecorator.

can prob borrow the plumbing for while_until_true(interval, max_attempts) in pypyr.utils.poll.
prob can do something like:

• retry_loop (self, context, step_method)
• exec_iteration(self, counter, context, step_method):. Return True if success (break out of retry loop), False if failed (continue retry loop).

exec_iteration contains the catches for retryOn - log and swallow if any of these, continue exit with False.

retry_loop will do something like this:

pypyr.utils.poll.while_until_true(interval=sleep,
                                     max_attempts=max)(
                                     self.exec_iteration)(context=context,
                                     step_method=step_method)

Rather than raise LoopMaxExhaustedError, raise the step_method error back up to the core if max exhausted.

The execution insertion point should probably be in invoke_step(self, context). Something like this:

if self.retry_decorator:
           self.retry_decorator.retry_loop(
                context,
                self.module.run_step(context)
            )
        else:
            logger.debug(f"running step {self.module}")
            self.module.run_step(context)
            logger.debug(f"step {self.module} done")

Note there's a bug here that might as well get fixed while we're here: #129

This means that retry happens inside the while and for loops. i.e a step can retry >1 within a single iteration of a while/for loop. Is this a valid assumption, or a good default behaviour? Or would a pipeline author instead expect the retries to be outside the while/for/conditional loops+checks?

Credit and thanks to @Reskov for an excellent idea!