RedisWorker should query fleet size at startup before first heartbeat

[dkliban]

Problem

When a RedisWorker starts, it initializes self.num_workers = 1 and doesn't learn the actual fleet size until its first heartbeat cycle (which calls AppStatus.objects.online().count()). The worker's sleep interval between fetch attempts is calculated as num_workers * 10ms — so a new worker with num_workers=1 sleeps only 10ms between attempts.

During a KEDA auto-scaling event, many workers start simultaneously. Each one thinks it's the only worker and uses an aggressive fetch cadence. Combined with the doubling fetch strategy (querying 20 → 40 → 80 → 160+ tasks per attempt), this creates a thundering herd on the database.

Impact

During the 2026-07-24 incident, KEDA scaled workers from 25 to 150. All 150 new workers started with num_workers=1, hammering the database with large task queries before their first heartbeat. This caused:

• Database contention preventing heartbeat writes
• Workers failing and restarting (producing more thundering herd)
• Orphaned Redis locks from workers that died mid-task-acquisition

Current Code

# redis_worker.py __init__()
self.num_workers = 1  # <-- hardcoded, not updated until first beat()

# sleep()
base_sleep_ms = self.num_workers * 10.0  # <-- 10ms with num_workers=1

Proposed Fix

In __init__(), query the current worker count before entering the main loop:

self.num_workers = max(1, AppStatus.objects.online().filter(app_type="worker").count())

This ensures new workers immediately use an appropriate sleep interval based on the actual fleet size, reducing DB pressure during scale-up events.

References

• RCA: 2026-07-24/25 tasking degradation — incident where thundering herd from 150 workers caused cascading failures
• Related: redis_worker.py sleep() method and beat() method where num_workers is updated

[gerrod3]

Some questions to look into:

• Do workers not perform a heartbeat on their initialization? Should they?
• Can workers pickup a task before their first heartbeat?
• Are workers in the app-status records before their first heartbeat?