Monitor loop is too aggressive

[obazoud]

Describe the bug
The monitor loop currently polls Redis at a high frequency (every 100ms), especially in development environments. When using a serverless Redis provider such as Upstash, the quota is quickly reached due to constant polling. If using ElasticCache Serverless, this aggressive polling can also lead to additional costs (few $/month)

From what I understand, this monitor loop is only used for retrying failed deliveries. It would be better to trigger the retry mechanism only when there are actually retries to process, instead of polling continuously.

To Reproduce
Steps to reproduce the behavior:

1. Start the API service
2. Check your Redis logs or monitoring
3. You should see the commands from the receiveMessage LUA Script every 100ms

Expected behavior

• Quick fix: the polling interval should be configurable. So in development and with serverless/pay-as-you-go Redis providers, the system should not quickly exhaust the quota or increase cost
• The monitor loop should avoid aggressive polling when there are no retry messages to process
• The retry mechanism should only be triggered when there are retries to process

Screenshots
N/A

Additional context
N/A

[alexbouchardd]

Good point; the serverless consideration here is one we hadn't considered. Those requests are extremely cheap when running Redis in VPC & just paying for the hardware, but if you pay for requests, I can see how that's problematic.

We can add a configuration or use a backoff algorithm. If there are no retries, we progressively back off and once we see one again, we can reset the delay between calls.

The alternative to polling would be to use something like pub/sub, which brings slightly more complexity but would work.

[obazoud]

A quick fix would already be very helpful 👍

The Lua script sends 7 commands per call par 100ms, ie 6M commands/day while the Upstash Free plan quota is 500 K commands/month

This applies to every replica: each one runs its own monitoring loop and sends Redis commands at this interval.