fix(resilience-ranking): chunked warm SET, always-on rebuild, truthful meta (Slice B)

[koala73]

Summary

Follow-up to PR #3121. Live behavior post-merge confirmed three coupled failures that left resilience:ranking:v9 absent even after Slice A landed (per-country score persistence) — the EMPTY_ON_DEMAND symptom in /api/health persisted.

Manual cron run log captured the smoking gun:

[Resilience-Scores] 0/222 scores pre-warmed
[Resilience-Scores] Warming 222 missing via ranking endpoint...
[Resilience-Scores] Ranking: 0 ranked, 222 greyed out          ← handler dropped publish
[Resilience-Scores] Rebuilt resilience:ranking:v9 with 222 countries (bulk-call race left ranking:v9 null)

Root causes

1. Pipeline body too big. warmMissingResilienceScores issued one 222-command pipeline SET (~600KB). It exceeded REDIS_PIPELINE_TIMEOUT_MS (5s) on Vercel Edge → runRedisPipeline returned [] → persistence guard correctly returned an empty warmed map → coverage 0/222 < 75% → handler skipped both ranking SET and meta SET. (Upstash actually persisted the writes a moment later — that's why the seeder's second HTTP call succeeded.)
2. Rebuild path gated on missing>0. seed-resilience-scores.mjs only probed/rebuilt the ranking inside if (missing > 0). Once per-country scores were warm from a previous cron, every subsequent cron skipped rebuild — but the ranking aggregate has the same 6h TTL and could expire alone, leaving multi-hour gaps.
3. Lying meta. writeRankingSeedMeta fired whenever recordCount > 0, with no check that resilience:ranking:v9 actually existed. Health endpoint then reported a fresh seedAge over a missing data key.

Fixes

• server/worldmonitor/resilience/v1/_shared.ts: split the warm pipeline SETs into SET_BATCH = 30-command chunks so each pipeline body fits well under timeout. Failed batches pad with empty entries to preserve per-command alignment.
• scripts/seed-resilience-scores.mjs: extract ensureRankingPresent({url, token, laggardsWarmed}) helper, call it from BOTH the warm and skip-warm branches so the rebuild fires every cron. Adds a post-rebuild STRLEN verification — rebuild HTTP can return 200 with a payload but still skip the SET internally.
• main(): gate writeRankingSeedMeta on result.rankingPresent. If the rebuild didn't actually produce a key, log and let the next cron retry.

Test plan

• Full resilience suite: 373/373 pass (3 new tests).
• resilience-ranking.test.mts: pipeline-size assertion (no single SET pipeline >30 commands).
• resilience-scores-seed.test.mjs: structural checks — ensureRankingPresent called from ≥2 sites, STRLEN-verify present, meta gated on result.rankingPresent.
• After deploy: live probe TTL resilience:ranking:v9 should be positive within one cron tick; /api/health should flip resilienceRanking to status: OK; the [Resilience-Scores] Skipping seed-meta write… warning should be rare.

Builds on PR #3121 (now on main).

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
worldmonitor	Ready	Preview, Comment	Apr 16, 2026 8:47am

[greptile-apps]

Greptile Summary

This PR addresses three coupled production failures that left resilience:ranking:v9 absent after the previous Slice A landing: (1) a 600KB single-pipeline SET timing out on Vercel Edge, (2) the ranking rebuild only running inside the missing > 0 branch, and (3) seed-meta being written even when the ranking key was absent.

The fixes — chunked SET batches (SET_BATCH = 30) in warmMissingResilienceScores, an ensureRankingPresent helper called from both cron branches, and a post-rebuild STRLEN verification gating the meta write — are logically sound and well-tested.

Confidence Score: 5/5

Safe to merge — all three root causes are addressed correctly with no new P0/P1 issues introduced.

All remaining observations are P2. The core logic — chunked SETs with index-alignment padding, always-on ranking probe/rebuild, and truthful meta gating — is sound and covered by both behavioral and structural tests. No data loss, security concerns, or broken contracts.

No files require special attention.

Important Files Changed

Filename	Overview
server/worldmonitor/resilience/v1/_shared.ts	Adds SET_BATCH=30 chunking in warmMissingResilienceScores with correct index-alignment padding for failed batches; the persistence guard correctly excludes any command whose batch returned [].
scripts/seed-resilience-scores.mjs	Extracts ensureRankingPresent helper (called from both warm and skip-warm branches), adds STRLEN post-rebuild verification, and gates writeRankingSeedMeta on result.rankingPresent. DEL-before-rebuild when laggardsWarmed>0 is an intentional tradeoff.
tests/resilience-ranking.test.mts	Adds two new behavioral tests: pipeline-size assertion (SET_BATCH ≤ 30) and persistence-guard regression (ranking not published when SETs return non-OK).
tests/resilience-scores-seed.test.mjs	Adds structural (source-text) assertions for the three invariants: ensureRankingPresent called ≥2 times, STRLEN verify present, and meta gated on rankingPresent.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A([Cron tick starts]) --> B[GET all country scores\nbulk pipeline]
    B --> C{preWarmed == total?}
    C -- missing > 0 --> D[Call /get-resilience-ranking\nto bulk-warm missing scores]
    D --> E[Re-check still-missing countries]
    E --> F{laggardsWarmed > 0?}
    F -- yes --> G[Warm laggards individually]
    G --> H[Final GET pipeline for intervals]
    F -- no --> H
    H --> I[computeAndWriteIntervals]
    I --> J[ensureRankingPresent\nlaggardsWarmed=N]
    C -- missing == 0 --> K[computeAndWriteIntervals\nwith preResults]
    K --> L[ensureRankingPresent\nlaggardsWarmed=0]
    J --> M{rankingExists != null\n&& laggardsWarmed == 0?}
    L --> M
    M -- yes --> P[return true]
    M -- no --> N{laggardsWarmed > 0?}
    N -- yes --> O[DEL resilience:ranking:v9]
    O --> Q[Rebuild via /get-resilience-ranking]
    N -- no --> Q
    Q --> R[STRLEN verify resilience:ranking:v9]
    R --> S{STRLEN > 0?}
    S -- yes --> T[return true]
    S -- no --> U[return false]
    P --> V{result.rankingPresent?}
    T --> V
    U --> V
    V -- true --> W[writeRankingSeedMeta]
    V -- false --> X[Log warning, next cron retries]

Loading

_{Reviews (1): Last reviewed commit: "fix(resilience-ranking): chunked warm SE..." | Re-trigger Greptile}

[greptile-apps]

Sequential batch awaits add round-trip latency

The batches are awaited one at a time. For 222 countries (⌈222/30⌉ = 8 batches), this serializes 7 extra Upstash round-trips (~100–500 ms each on Vercel Edge) that could run in parallel. Promise.all would reduce warm time to a single wave, though you'd need to flatten results in batch-offset order and keep the same alignment-padding logic.

Not blocking — the sequential approach is safer for backpressure and the total latency is still well within the function timeout.

[greptile-apps]

DEL before rebuild confirmation creates a brief absence window

When laggardsWarmed > 0 and the ranking key currently exists, the DEL fires before the rebuild HTTP call completes. If the rebuild request returns non-OK (transient edge timeout, deploy in progress, etc.), the STRLEN verify returns 0, rankingPresent is false, and the ranking stays absent until the next cron tick. The meta write is correctly skipped so the health endpoint won't lie, but users will hit EMPTY_ON_DEMAND for up to one cron interval.

A safer sequencing would DEL only after receiving a successful 200 from the rebuild call — though this does mean the stale ranking briefly serves read traffic during the rebuild, which may be acceptable. Worth documenting the tradeoff explicitly if not already captured elsewhere.