Retry Bungie Cloudflare blocks via queue backoff

[owens1127]

Summary

• HTTP layer: One quick Cloudflare retry (~2s with 20% jitter), plus up to two fast transient retries (timeout/connection/5xx). No more multi-second in-process CF backoff loops that hold workers.
• Queue layer: Set MaxRetryCount: 0 on Bungie crawl queues (player_crawl, character_fill, activity_history, clan_crawl) so transient CF blocks republish to the delayed exchange with exponential backoff (capped at 30m) instead of being dropped after 3–5 attempts.

Addresses recurring SERVICES-1Q (MESSAGE_EXCEEDED_MAX_RETRIES on Bungie 403 Cloudflare pages).

Test plan

• go build ./...
• Deploy Hermes: make hermes && sudo systemctl restart hermes
• Confirm MESSAGE_EXCEEDED_MAX_RETRIES stops for player_crawl / character_fill during CF blips
• Spot-check a player crawl succeeds after a short CF block (message reappears from delayed exchange)

Deploy

Manual Hermes restart on prod after merge.

Made with Cursor

[sentry]

Bug: The condition if attempt > 1 is unreachable because the retry logic is configured with MaxAttempts: 1, making the code to add a retry query parameter dead.
_{Severity: LOW}

Suggested Fix

Either remove the unreachable if attempt > 1 block to eliminate the dead code, or if the retry query parameter is desired for observability, increase the MaxAttempts value in the CloudflareRetryConfig to be greater than 1.

Prompt for AI Agent

Review the code at the location below. A potential bug has been identified by an AI
agent. Verify if this is a real issue. If it is, propose a fix; if not, explain why it's
not valid.

Location: lib/web/bungie/client.go#L103-L105

Potential issue: In the retry logic for Bungie API requests, a condition `if attempt >
1` is used to add a `retry` query parameter to the URL. However, the underlying retry
mechanism is configured with `MaxAttempts: 1`. This configuration means the function
will be called with `attempt = 0` on the first try and `attempt = 1` on the single retry
attempt. The `attempt` variable will never be greater than 1, making the conditional
block unreachable dead code. This results in a minor loss of observability, as the
intended debugging parameter will never be added.

_{Did we get this right? 👍 / 👎 to inform future reviews.}