enqueue-cheat-check tool + faster Hermes backlog autoscaling

[owens1127]

Summary

• Adds tools/enqueue-cheat-check to publish instance IDs to instance_cheat_check for Hermes rechecks (activity, version, date range, instance IDs, skip-current, limit, dry-run).
• Speeds up Hermes autoscaling when queues spike:
  • New LargeBacklogThreshold fast-path in topic_manager — skips consecutive-check wait when depth is high.
  • 30s ScaleCheckInterval (was 5 min default) on instance_cheat_check, player_crawl, character_fill.
  • Tuned scale steps/cooldowns for those three topics.

Autoscaling changes

Queue	Max workers	Large backlog @	Check interval
instance_cheat_check	16 (was 4)	1000	30s
player_crawl	70	1000	30s
character_fill	15	1000	30s

Usage

go run ./tools/enqueue-cheat-check/ -dry-run -activity=102
go run ./tools/enqueue-cheat-check/ -activity=102 -since=2026-06-14T00:00:00Z

Deploy

cd /RaidHub/Services && git pull origin main
export PATH=/usr/local/go/bin:$PATH
make hermes && sudo systemctl restart hermes

Prod note

enqueue-cheat-check already used on prod to enqueue 819,743 activity-102 instances (tool scp'd pre-merge). Hermes autoscaling fix requires restart after merge.

Test plan

• go build ./apps/hermes/...
• go build ./tools/enqueue-cheat-check/
• Prod dry-run + enqueue for -activity=102
• CI green
• Post-merge: Hermes restart, confirm instance_cheat_check workers scale up within ~30s on deep queue

[sentry]

Bug: When -instance-ids is used with -activity or -version, the code applies both filters, contradicting the flag's documentation which states activity/version filters should be skipped.
_{Severity: MEDIUM}

Suggested Fix

Modify the logic in buildQuery to only perform the join and filter for activities/versions when no instance IDs are provided. Change the assignment of joinActivity to len(p.instanceIDs) == 0 && (len(p.activities) > 0 || len(p.versions) > 0). This will align the implementation with the documented behavior.

Prompt for AI Agent

Review the code at the location below. A potential bug has been identified by an AI
agent. Verify if this is a real issue. If it is, propose a fix; if not, explain why it's
not valid.

Location: tools/enqueue-cheat-check/main.go#L195

Potential issue: There is a logical contradiction between the documented behavior of the
`-instance-ids` flag and its implementation in the `buildQuery` function. The flag's
help text states it "skips activity/version filters". However, if a user provides both
`-instance-ids` and `-activity` or `-version` flags, the code generates a SQL query that
filters on both instance IDs and activity/version. This results in silently returning
fewer instances than the user expects based on the documentation. The issue stems from
the `joinActivity` variable being set to true even when instance IDs are present,
leading to an incorrect query.

_{Did we get this right? 👍 / 👎 to inform future reviews.}