feat(challenges): make Phase 3 incremental + partial GIN for on_user_challenge cooldown trigger

[raymondjacobson]

Summary

Two related fixes for the wedge the IndexChallengesJob hit right after #842 deployed. The first post-deploy reconcile tick stalled (20+ min, no completion):

• The four new Phase 3 processors (m/r/rv/rd) were full-scan over 2.3M user_events rows.
• Every is_complete=true upsert fired the on_user_challenge trigger's cooldown-window check — an unindexed scan against the 8 GB notification table (~23.5M rows) that ran 19s+ per call (caught in pg_stat_activity, IO-bound on DataFileRead).

Block indexing stayed healthy throughout (the indexer uses separate connections) — this PR is purely a perf fix for the challenge job.

Two-pronged fix

(a) Phase 3 processors go incremental, #875-style

Each tick rescans only user_events rows whose blocknumber moved past a per-processor checkpoint, then re-derives state for the affected users. Same pattern the merged #875 used for profile_completion/track_upload/first_playlist/cosign.

Supporting migrations:

Migration	What it does
0209_user_events_blocknumber_idx.sql	CREATE INDEX CONCURRENTLY btree on user_events(blocknumber) so the dirty scan is an index range read instead of a 2.3M-row seq-scan.
0211_seed_phase_3_user_event_checkpoints.sql	Seeds the four checkpoints (challenges:m/r/rv/rd:last_blocknumber) to current max(user_events.blocknumber) so prod starts "from now" and skips re-deriving 2.3M historical rows. Python already populated user_challenges; the upserts are idempotent.

(b) Partial GIN on notification for the trigger's slow path

CREATE INDEX CONCURRENTLY ix_notification_cooldown_user_ids
    ON public.notification USING gin (user_ids)
    WHERE type = 'reward_in_cooldown';

The on_user_challenge trigger does, for every is_complete=true upsert where cooldown_days > 0:

SELECT id FROM notification
 WHERE type='reward_in_cooldown'
   AND new.user_id = ANY(user_ids)
   AND timestamp >= (new.completed_at - interval '1 hour')
 LIMIT 1

The full GIN on user_ids matches across the entire 8 GB table. The partial GIN restricted to type='reward_in_cooldown' is small (one type out of ~30) and lets the planner go straight to user-id matches within that slice. Benefits every cooldown_days>0 challenge (p, u, the Phase 2 ones, etc.), not just Phase 3.

Caveat

The r/rv dirty scan keys on user_events.blocknumber, so a referrer's verification flip (their users.is_verified going true) is not picked up until the referred user's user_events row changes again. Verification changes are rare and the old full-scan code only caught them on its next tick; this matches the precedent #875 set for the other incremental processors. Called out in the file-level docstring.

Test plan

• go build ./..., go vet ./jobs/challenges/..., gofmt clean.
• All 5 existing user_events processor tests pass — checkpoint defaults to 0 in the test DB, so the first run still catches the seeded block-101 rows (same end-state as the full-scan version).
• Added TestMobileInstall_SkipsRowsBelowCheckpoint — pre-seeds the checkpoint past the row's blocknumber and asserts no upsert, explicitly covering the dirty-set skip.
• Full jobs/challenges suite green (incl. feat(challenges): make heavy processors incremental (dirty-set) #875's incremental tests).
• Schema dump + migration tracker regenerated.

Cutover note

Prod is currently on cd94ede (#842) with the challenge job's first tick still grinding on the slow path. Once this deploys:

1. 0209 creates the user_events blocknumber index (CONCURRENTLY, no write blocking).
2. 0210 creates the partial GIN on notification (CONCURRENTLY).
3. 0211 seeds the four Phase 3 checkpoints to the current high-water mark.
4. The next tick of IndexChallengesJob picks up the new processors with checkpoint = max, so the dirty set is ~0 rows and the tick completes in milliseconds.

The currently-stuck in-flight tick on cd94ede will continue running until it finishes naturally (or until the pod is replaced by this deploy, which terminates it).

🤖 Generated with Claude Code