retries: add support for "silent" retries

[Bogdanp]

No description provided.

[Bogdanp]

cc @takhs91

[jcass77]

In a similar vein, I've always wondered why RateLimitExceeded is logged as an error?

I often use ConcurrentRateLimiter as a distributed mutex lock if I want only one actor to be able to access a resource or perform some activity at a time. This usually results in a flood of RateLimitExceeded errors which seems unusual as actors are expected to back-off until they obtain the lock.

In the end I ended up muting all of the RateLimitExceeded exceptions, and added a custom logging middleware to ensure that message failures are logged once all of the retry attempts have ultimately been exhausted:

class LogFailuresMiddleware(dramatiq.Middleware):
    def after_process_message(self, broker, message, *, result=None, exception=None):
        """
        Ensure that message failures (e.g. as a result of timeout or retry limits) are logged.

        Note that a Dramatiq message 'fails' only after all of its retry attempts have been exhausted (i.e. failures
        on individual retries will not be logged, only the last one that causes Dramatiq to give up).
        """
        if message.failed:
            logger.error(
                f"'{message}' failed after {message._message.options.get('retries', 0)} retries - giving up!",
                extra={"result": str(result), "exception": str(exception)},
            )

[jcass77]

...maybe the canonical approach once this is merged will be to:

1. set the raise_on_failure parameter to false when acquiring the mutex
2. raise the new Retry exception provided by this PR if the lock cannot be acquired so that retry attempts are incremented but nothing is logged.
3. include a custom middleware (like above) for logging message failures once retries are exhausted

@Bogdanp, any comments on this?

[Bogdanp]

The Retries middleware already logs a warning when a message's retries are exhausted so I think your proposed middleware may be unnecessary. I've lowered the log level for the RateLimitExceeded case to debug since you're right that it is unnecessarily noisy in the general case.