fix: replace KEYS with SCAN in RedisCache.ListQueues to prevent blocking

[Copilot]

Replace KEYS * with SCAN in Housekeeper to avoid Redis lock

Plan

• Explore the codebase and understand the issue
• Identify where KEYS * is used (redis_cache.go:200)
• Replace KEYS * with SCAN in RedisCache.ListQueues method
• Implement paginated iteration using SCAN cursor
• Ensure no functional changes to queue name processing
• Add unit tests for the new SCAN-based iteration
• Run existing tests to verify no regressions
• Run full test suite including housekeeper tests
• Run code review and security checks
• Address feedback on test coverage

Implementation Summary

Successfully replaced cache.Client.Keys(context.Background(), searchPattern) with SCAN in internal/queue/cache/redis_cache.go:

Key Changes:

1. SCAN Implementation: Uses cursor-based pagination with batch size of 1000 keys per iteration
2. Non-blocking: Iterates through all keys without blocking Redis operations
3. Backward Compatible: Maintains exact same functionality and behavior
4. Properly Tested: All 40+ existing tests pass, plus 2 new SCAN-specific tests

Test Results:

• ✅ All unit tests pass
• ✅ All integration tests pass (Redis 5.0.14)
• ✅ Housekeeper tests pass
• ✅ Security scan: 0 vulnerabilities found
• ✅ Code quality: No vet issues

New Tests Added:

1. TestListQueuesWithManyCachedQueuesIntegration - Creates 1500 queues to properly test SCAN pagination across multiple batches (batch size is 1000)
2. TestListQueuesWithSpecificPatternIntegration - Validates pattern matching with SCAN

Impact

This change improves Redis performance and stability by preventing potential locks caused by the KEYS * command, especially important when dealing with large numbers of queue keys.

Original prompt

---

This section details on the original issue you should resolve

<issue_title>Replace KEYS * with SCAN in Housekeeper to avoid Redis lock</issue_title>
<issue_description>The current iterator used by the Housekeeper component retrieves all queue names using the KEYS * command.
This approach can cause performance issues and potential Redis locks, especially when there are a large number of keys.

We should replace the KEYS * usage with SCAN, implementing proper pagination to iterate through keys efficiently and safely.

Acceptance Criteria:

• Replace KEYS * with SCAN in the Housekeeper iterator.
• Implement paginated iteration to fetch all queue names without blocking Redis.
• Ensure no functional changes to how queue names are processed.
• Add tests covering the new iteration logic.

Impact:
Improves Redis performance and stability by preventing potential locks caused by the KEYS * command.</issue_description>

Comments on the Issue (you are @copilot in this section)

• Fixes Replace KEYS * with SCAN in Housekeeper to avoid Redis lock #70

---

✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.