retrymq: reduce Redis load from idle retry scheduler polling

[alexluong]

Follow-up to #486, which added RETRY_POLL_BACKOFF_MS as a quick fix. This is the structural fix discussed there ("if there are no retries, we progressively back off").

Current behavior

scheduler.Monitor calls rsmq.ReceiveMessage in a loop, sleeping pollBackoff (default 100ms) whenever no message is returned. Every RSMQ method first calls getQueue (rsmq.go:299), a MULTI/HMGET/TIME/EXEC pipeline that fetches vt/delay/maxsize and the server clock, purely to compute the arguments for the EVALSHA that follows.

One idle poll:

RTT 1:  MULTI / HMGET {ns}:deliverymq-retry:Q vt delay maxsize / TIME / EXEC
RTT 2:  EVALSHA <receiveMessage> → ZRANGEBYSCORE finds nothing → {}

5 client-observable commands, 2 round trips, whether or not anything is queued.

	per second	per 30 days
Poll iterations	10	25.9M
Redis commands	50	129.6M
Round trips	20	51.8M

Counted as commands sent by the client. #486 cites 7/call, which counts redis.call()s inside the Lua body — the basis Upstash bills on. Both bases move by the same factor under this proposal.

This cost is per monitor instance and independent of traffic. Every replica runs its own monitor loop, so total Redis load scales with the number of running instances rather than with retry volume, and a queue that is empty for days costs exactly as much as a busy one.

vt/delay/maxsize are written once at CreateQueue and never mutated — outpost never calls SetQueueAttributes — so RTT 1 fetches three constants and a clock reading.

Why upstream does it this way

RSMQ (Node, 2013) predates redis.replicate_commands() (Redis 3.2) and effect-replication-by-default (Redis 5). Under verbatim script replication, calling TIME inside a script would desync replicas, so passing the timestamp in from the client was the only option. That constraint is long gone; the split round trip survived the port to go-rsmq and into this fork.

Proposal

Part A — make receiveMessage self-contained. Read vt from the :Q hash and call TIME inside the script. Eliminates RTT 1. 5 commands → 1, 2 round trips → 1.

Part B — sleep until the next due message. When nothing is due, the script also returns the earliest score in the sorted set. The monitor then sleeps clamp(nextDue − now, floor, cap) instead of a flat pollBackoff. An empty queue sleeps the full cap.

Two properties make this correct without special-casing:

• The sorted set holds both not-yet-due retries and in-flight messages hidden by vt — receiveMessage re-ZADDs at now+vt rather than removing — so the earliest score is the correct wake time in both cases.
• Every sleep is clamped to the cap, so worst-case lateness is bounded by the cap unconditionally, regardless of queue contents.

Lateness only occurs when a message is scheduled with a delay shorter than the monitor's remaining sleep, and is then bounded by that remainder. Consequently, if the cap is ≤ the shortest configured retry delay, added latency is exactly zero.

Throughput is unaffected: when a message is found the monitor polls again immediately with no sleep, so a backlog drains at full speed.

Config

RETRY_POLL_BACKOFF_MS is redefined as the maximum idle sleep, default 100 → 0, a sentinel meaning auto.

The knob keeps its user-facing meaning — how long the monitor waits when idle — and gains a contract that states both sides of the tradeoff: idle cost is one command per interval, and worst-case retry lateness is that interval, zero when it's ≤ your shortest retry delay.

For any fixed value, the new sleep min(nextDue − now, X) is ≤ the old flat X, so existing values keep working and only improve. The default move is the sole behavior change, and it affects only configs whose shortest retry delay is under 30s.

Auto (0/unset) resolves to min(30s, shortest configured retry delay), making zero added latency true by construction for every config, including a custom retry_schedule with sub-30s entries. An explicit positive value is honored as-is as a fixed maximum idle sleep.

Revised during PR #1026. The original proposal here clamped every configured value to the shortest retry delay. That cap could silently override an explicit user value downward — e.g. retry_schedule: [5, …] with an explicit retry_poll_backoff_ms: 10000 would be forced to 5s, doubling the user's intended idle Redis cost. The sentinel default keeps the zero-latency guarantee for the default config while leaving explicit values untouched. Startup validation now rejects retry_schedule entries < 1, retry_interval_seconds < 1 (when no schedule is set), and negative retry_poll_backoff_ms.

At a 30s cap: 129.6M → 86.4K commands per month, per monitor.

Also: decouple the consecutive-error backoff ladder (scheduler.go:179) from pollBackoff into internal constants, so the ~1 minute of transient-infra tolerance documented at scheduler.go:89-108 holds regardless of how this is configured.

Notes

Dragonfly strictly enforces declared KEYS[] — a script touching an undeclared key errors at runtime. Redis does not, so a script can pass Redis-only tests and fail on Dragonfly. internal/rsmq/rsmq_test.go already has a DragonflyRSMQSuite; both must cover the new script. TIME-then-write inside Lua is verified working on Redis and Dragonfly v1.39.0.

Key layout, field names, and score semantics are unchanged. No migration, no coordinated cutover — old and new instances can poll the same queue during a rolling deploy, and rollback is redeploying the previous image.

Follow-ups

Deliberately out of scope, each its own change:

• Fold getQueue into sendMessage (10 commands → ~2). Separated because of blast-radius asymmetry: a receive-path bug leaves the message queued for the next poll, while a send-path bug silently fails to schedule a retry for a delivery that has already failed. It also needs a decision on the generated-ID path, which derives the ID's timestamp from server time — proposal is to source the score from in-script TIME and leave the ID's cosmetic timestamp on the client clock, which scheduler/id.go already documents as unused for timing.
• Batch receive + bounded worker pool. The monitor currently returns one message per round trip and spawns an unbounded goroutine per message (scheduler.go:201).
• Single retry scheduler instance per process. initRetryScheduler is called by both BuildAPIWorkers and BuildDeliveryWorker, so a SERVICE=all container builds two rsmq clients and two Redis pools while only one runs Monitor. Sharing one instance also enables waking the monitor in-process when a retry is scheduled.
• time.Sleep at scheduler.go:193 is not ctx-aware, delaying shutdown. Fixed incidentally by Part B.

[alexbouchardd]

Part B — sleep until the next due message. When nothing is due, the script also returns the earliest score in the sorted set. The monitor then sleeps clamp(nextDue − now, floor, cap) instead of a flat pollBackoff. An empty queue sleeps the full cap.

Two properties make this correct without special-casing:

The sorted set holds both not-yet-due retries and in-flight messages hidden by vt — receiveMessage re-ZADDs at now+vt rather than removing — so the earliest score is the correct wake time in both cases.
Every sleep is clamped to the cap, so worst-case lateness is bounded by the cap unconditionally, regardless of queue contents.

That strikes me as incorrect. A newer retry can be scheduled before the currently "next" score in which case the sleep time isn't true