Configuration Retry Policy

Retry Policy

Try a failed request again, with a delay before each retry, and choose which failures are worth it.

Configuration

SetRetryPolicy is called on the cfg of AddFxMap. Without it nothing is retried.

builder.Services.AddFxMap(cfg =>
{
    cfg.AddEntitiesFromAssemblyContaining<SomeEntityAssemblyMarker>();
    cfg.AddProfilesFromAssemblyContaining<SomeProfileAssemblyMarker>();
 
    cfg.SetRetryPolicy(
        retryCount: 3,                                              // retries after the first attempt (at most 4 attempts)
        sleepDurationProvider: n => TimeSpan.FromMilliseconds(      // waited BEFORE retry n (1 = the first retry)
            Random.Shared.Next(0, 200 * (1 << (n - 1)))),           // exponential backoff with jitter
        onRetry: (ex, delay) => logger.LogWarning(ex, "Retry in {Delay}", delay),
        shouldRetry: ex => ex is TimeoutException or IOException);  // retry only what can go away
});

Parameter	Meaning
retryCount	The maximum number of retries after the first attempt. The default is 3, which makes at most 4 attempts. It cannot be negative.
sleepDurationProvider	Returns the delay before a retry. Its parameter is the number of the retry, starting at 1 for the first retry. Without it the retries follow each other at once.
onRetry	Called before each retry, with the exception of the attempt that failed and the delay that is about to be waited. It is not called for the last failure, because no retry follows it.
shouldRetry	Since 5.1. Decides whether an exception is worth a retry. When it returns false the exception is thrown at once, without a delay. Without it every exception is retried, except when the caller cancelled.

What Happens, Step by Step

1. The request is sent. If it succeeds, that is all.
2. If it fails and retries are left, shouldRetry is asked (when you set it). If it says no, the exception is thrown at once.
3. Otherwise the delay of the retry is calculated, onRetry is called with the exception and the delay, and the delay is waited. The delay comes before the retry, and nothing is waited after the last failure.
4. The request is sent again. After the last attempt fails, the last exception is thrown.
5. If the caller cancels the request, nothing more is retried, and a delay that is being waited ends.

Choosing the Delay

Fixed

cfg.SetRetryPolicy(3, n => TimeSpan.FromMilliseconds(500));

Linear

cfg.SetRetryPolicy(3, n => TimeSpan.FromMilliseconds(200 * n));          // 200, 400, 600 ms

Exponential, with a cap

cfg.SetRetryPolicy(5, n => TimeSpan.FromSeconds(Math.Min(Math.Pow(2, n - 1), 10)));   // 1, 2, 4, 8, 10 s

Exponential, with jitter

cfg.SetRetryPolicy(3, n => TimeSpan.FromMilliseconds(Random.Shared.Next(0, 200 * (1 << (n - 1)))));

Jitter spreads the retries of many callers in time, so they do not all come back at the same moment.

What Is Retried

• A request that goes through a transport (gRPC, NATS, RabbitMQ, Kafka, Azure Service Bus, SQS) is retried.
• A request that a local handler answers, because the service owns the data, is not retried. Its errors are thrown at once. (In 5.0.0 a local handler was retried.)
• ThrowIfException() is not needed. Without it (the default) the error of a request is suppressed only after the last attempt, and the property stays empty, as before. With it, the last exception is thrown. Before 5.1 the policy did nothing when ThrowIfException() was not called, because the suppression swallowed every failure before the policy saw it.

The request timeout covers all the attempts

The request timeout (30 seconds by default, cfg.SetRequestTimeOut(...)) is counted for the whole request, the delays included. When it is over, nothing is retried. So a request that fails because nobody answered in time is usually not retried: the retries help with the failures that come back quickly, such as a connection that was refused or lost. If you add retries, keep the timeout short enough for the worst case you can accept.

Changed in 5.1

The retry policy of 5.0.0 did not do what its name says. 5.1 fixes it, so check your code if you counted on the old behavior:

Before 5.1	From 5.1
The delay was waited after a failed retry. The first retry followed at once, and the policy still slept after the last failure.	The delay is waited before each retry, and nothing after the last attempt.
onRetry was called after the wait, for the failure of a retry, and never for the first failure.	onRetry is called before each retry, with the exception of the failed attempt and the delay.
The first exception was thrown.	The last exception is thrown.
With ThrowIfException() not called, nothing was retried.	A request through a transport is retried, and its error is suppressed after the last attempt.
A local handler was retried.	A local handler is not retried.
A caller that cancelled did not stop the retries.	A cancel stops the retries, also while a delay is waited.
Every exception was retried.	shouldRetry can refuse a retry.

See the upgrade guide for 5.1.

Good Practice

• Retry only what can go away. Use shouldRetry to retry a timeout or a lost connection, and not an error that the same request will cause again.
• Log in onRetry, so retries do not happen silently.
• Keep retryCount small and add jitter to the delay.
• Remember that a retry sends the same request again: an owner can receive it more than once, so answering it must not have side effects.

A retry policy does not replace the supervisor: the policy retries a request, and the supervisor restarts a server of a message transport that failed.

---

Source: https://fxmapper.net/docs/configuration/retry-policy/ — the website is always the most up to date version.