Harden Redis queue uniqueness with atomic Lua enqueue

[danielgerlag]

Describe the change
RedisQueueProvider.QueueWork de-duped pending ids with a non-atomic LINSERT + conditional RPUSH / LREM. Two concurrent enqueues of a missing id could both miss, both RPUSH, and leave duplicate pending LIST entries. Dequeue was (and remains) LPOP.

This change makes pending-id uniqueness atomic inside the Redis queue provider only. IQueueProvider is unchanged. Issue #1417 / PR #1444 (QueueConsumer._secondPasses after purge) are a separate consumer-side problem and are not addressed here.

Describe your implementation or design
Smallest upgrade-safe design: keep the existing LIST keys and the existing LINSERT/RPUSH/LREM algorithm, but run that sequence in one Lua EVAL so Redis serializes it.

Rejected alternative (SADD membership + RPUSH, LPOP+SREM): extra key, rolling-upgrade skew between old processes (LIST-only) and new processes (LIST+SET), and a crash window unless dequeue is also a single Lua. Safer for existing deployments is “no second key.”

DequeueWork stays a single LPOP. Because there is no companion SET, there is nothing to desync if a process dies after pop.

Tests
Added RedisQueueProviderFixture in WorkflowCore.Tests.Redis (existing Squadron Redis collection — this repo does not use Testcontainers for Redis):

• Concurrent QueueWork of a missing id leaves exactly one pending entry
• Re-QueueWork while pending is a no-op and preserves FIFO position
• Concurrent dequeue of unique ids returns each id once and leaves the LIST empty
• Dequeue then re-enqueue of the same id succeeds
• Pre-existing duplicate LIST entries are not grown by QueueWork and drain via LPOP

Breaking change
No required migration or flush. Same Redis keys, same intended uniqueness semantics, now actually atomic.

Compatibility

This design was chosen so existing Redis keys keep working. Details:

• Existing LIST keys that already contain duplicate entries: No migration. Duplicates stay until they are dequeued (LPOP drains them one by one). A new QueueWork of that id does not add another occurrence. Operators do not need a one-time flush; optionally deleting {prefix}-workflows|events|index would only drop in-flight work (not recommended).
• Rolling upgrades (mixed old/new RedisQueueProvider processes): Both generations share the same LIST. New processes enqueue atomically among themselves. An old process can still race its own multi-command sequence (or interleave LINSERT miss + RPUSH next to a new process) and insert a duplicate until every node is upgraded. After the rollout completes, uniqueness holds.
• Extra Redis keys: None. No companion SET, so no naming/TTL/cleanup and no SET/LIST drift.
• Crash window between pop and membership remove: None. Dequeue is one LPOP. An id cannot get stuck in a membership set or be lost because of a failed follow-up SREM.
• Semantic change (re-QueueWork while pending): Stronger no-op. The old code intended not to add a second pending entry, but concurrent misses could both RPUSH. After this change, re-queue of an id that is still pending is always a no-op. After dequeue, the same id may be queued again. FIFO position of an already-pending id is unchanged (it is not moved to the tail).
• One-time migration/flush: Not required.
• Redis version: Script uses LINSERT (2.2+) and Lua (2.6+). No LPOS / Redis 6 requirement.
• IQueueProvider / other providers: Unchanged. Production behavior outside RedisQueueProvider is unchanged.

Additional context
Queue key format is unchanged: {prefix}-workflows, {prefix}-events, {prefix}-index.