fix(gdelt-intel): share one timestamp validator between the fetch-ordering and health paths (#5858)

[Yigtwxx]

Summary

Addresses defect 2 of #5858. Defect 1 is deliberately not in this PR — see below.

scripts/seed-gdelt-intel.mjs had two readers of the same stored topic stamps, and only one of them clamped forward skew.

rankTopicsForFetch (:264-268) took the run clock:

const stampMs = (value) => {
  const parsed = Date.parse(value);
  if (!Number.isFinite(parsed)) return Number.NEGATIVE_INFINITY;
  return Number.isFinite(nowMs) ? Math.min(parsed, nowMs) : parsed;
};

contentMeta (:944-946) did not:

.map((t) => Date.parse(t?.fetchedAt))
.filter((ms) => Number.isFinite(ms) && ms > 0);

And contentMeta is the one that matters for the alarm: RUN_SEED_OPTS.maxContentAgeMin: 1440 (:966) is evaluated against the newestItemAt it returns. The path with the guard was the path that did not need it.

This extracts parseStampMs(value, nowMs) and routes both readers through it, which is what the issue asks for — "The two should share one timestamp validator", "ideally via a shared helper so the ordering path and the health path cannot diverge."

One correction to the issue's analysis, offered as evidence

The issue says a future-dated stamp keeps newestItemAt "falsely fresh indefinitely". That overstates it, and I would rather say so than quietly ship against it: api/health.js:1009 already computes isFutureDated from a negative content age and folds it into contentStale (:1015), with a comment saying exactly why. So a wildly future stamp does not hide staleness today — it surfaces as STALE_CONTENT.

What the missing clamp actually costs is narrower, and still worth closing:

1. The skew window. A stamp written a few hours ahead is not yet negative when health reads it minutes later, so the cohort reads fresh for the length of the skew. That is precisely the regime a real container clock drift produces — hours, not years — and it is the regime health's existing guard does not cover.
2. A newestItemAt an operator cannot reconcile. The value goes out on the wire. Two components in the same seeder resolving one stored stamp to two different instants is the divergence the issue is about, independent of whether either happens to alarm.
3. Date.parse accepted stamps at or before the epoch. contentMeta's ms > 0 filter caught exactly 0; a pre-epoch ISO string parses negative, passed Number.isFinite, and would have been silently taken as the oldestItemAt. parseStampMs rejects the whole range.

The clamp also has to be conditional: rankTopicsForFetch is called with an injected clock in tests and Date.now() in production, so a non-finite nowMs degrades to the raw parse rather than swallowing every stamp. That behaviour is preserved and pinned.

Why defect 1 is not here

Defect 1 — newestItemAt is a Math.max, so any single GDELT success holds the alarm green while five topics starve — is real and still open. I am not implementing it in this PR because it is a decision that should be yours, not mine, and the issue names two different shapes for it:

Suggested fix: alarm on per-topic staleness, not on the freshest topic. Either evaluate maxContentAgeMin against oldestItemAt, or add a distinct coverage/staleness field […] prefer a distinct field over ORing into the existing signal, and update the published schema description in the same change.

Here is what I found scoping each, so the decision is cheap when you make it:

• Evaluate maxContentAgeMin against oldestItemAt. Two lines, no contract change — and it is the shape your own note warns against, because one starved topic then holds the whole cohort at STALE_CONTENT with no detail about which topic or how many.
• A distinct coverage field. The right shape, and the expensive one. The content-age trio is a fixed contract: scripts/_seed-utils.mjs:555-557 and :620-623 on the write side, :723-727 on the read side, buildEnvelope in scripts/_seed-envelope-source.mjs plus its two mirrors (server/_shared/seed-envelope.ts, api/_seed-envelope.js) under scripts/verify-seed-envelope-parity.mjs, and api/health.js:997-1016 where it becomes contentAge. A fourth field touches all of them, and the new status it implies applies to every seeder that opted into content-age, not just this one.
• A third option worth considering, which I would pick: api/health.js already carries oldestItemAt through to contentAge (:1012) and does nothing with it. A distinct coverageStale derived there — separate from contentStale, so no existing signal is OR'd into — needs no envelope change at all, only a decision about what bound the starved end is judged against and whether a new status appears in the entry.

Happy to implement any of the three on your word, in this PR or a follow-up.

Verification

tests/seed-gdelt-intel-content-age.test.mjs   13 pass, 0 fail   (new)
+ seed-gdelt-intel-fetch-rotation, -fetch-budget,
  -merge-read-warns, -timeline-resilience, gdelt-fetch
                                              92 pass, 0 fail

Every guard is mutation-proven:

Mutant	Red
forward-skew clamp removed	6
epoch guard removed	1
contentMeta back to a bare Date.parse (the pre-fix reader)	4
contentMeta loses its run-clock default	1
clamp inverted to Math.max	21
articleless topics counted again	2

No survivors.

The fixture in the first contentMeta case is the starvation from #5848 itself — military 5 hours old, cyber 18 days, energy 29 days — so the test reads as the incident rather than as invented numbers. The last two tests exist only to pin the divergence closed: they feed one skewed fetchedAt to rankTopicsForFetch and to contentMeta and assert both resolve it to the same instant, and that ordinary past stamps still produce the same fetch order as before.

contentMeta now takes nowMs = Date.now(). runSeed invokes it as contentMeta(data) (scripts/_seed-utils.mjs:2001), so the default is the production path; there is a test asserting the one-argument call still clamps.

Other gates:

npm run typecheck        clean
npx biome check          clean (2 files)
npm run lint:boundaries  no violations
check-unicode-safety     2652 files scanned, clean

npm run test:data: identical failure set to origin/main — 47 failing test names on both, comm diff empty in both directions.

Out of scope

• Defect 1, for the reasons above.
• api/health.js. Its existing future-date handling is correct and untouched; this fix is upstream of it, at the seeder that writes the number.
• seed-gdelt-bulk-materializer (GDELT 429s are supply-side load shedding — proxy fixes are exhausted; migrate all consumers to the mirrored bulk datasets (bulk-first) #5843). As the issue's scope note says, the fetch-ordering code becomes dead once that lands but contentMeta and its health contract survive — which is why the shared validator is worth having on both sides now rather than only on the one that stays.
• No change to maxContentAgeMin, to the ranking comparator, or to any published field's meaning.

Type of change

• Bug fix
• New feature
• New data source / feed
• New map layer
• Refactor / code cleanup
• Documentation
• CI / Build / Infrastructure

Affected areas

• Map / Globe
• News panels / RSS feeds
• AI Insights / World Brief — the GDELT intel seeder feeding the analyst context; no payload shape change
• Market Radar / Crypto
• Desktop app (Tauri)
• API endpoints (/api/*)
• Config / Settings
• Other: scripts/seed-gdelt-intel.mjs (Railway seeder), health newestItemAt/oldestItemAt values

Checklist

• Tested on worldmonitor.app variant — N/A. The change is inside a Railway seeder's timestamp handling; there is no user-visible surface, and reproducing the original needs a container with a skewed clock. Verified through both exported readers with an injected run clock, and with mutation proof that each guard has teeth.
• Tested on tech.worldmonitor.app variant (if applicable) — N/A, no variant-specific behaviour.
• New RSS feed domains added to api/rss-proxy.js allowlist (if adding feeds) — N/A, no feeds added.
• No API keys or secrets committed
• TypeScript compiles without errors (npm run typecheck)

Documentation Alignment Checklist

N/A — no published documentation claim changes. newestItemAt and oldestItemAt keep their documented meanings; the fix only stops a skewed source stamp from producing a value ahead of the run clock. The schema-description update the issue asks for belongs with defect 1's new field, which is not in this PR. Listed for completeness:

• Claim ledger attached or linked — N/A, no documented claim changes.
• All required Audit Council role signoffs attached — N/A, no methodology or contract change.
• Generated docs regenerated from proto where applicable — N/A, no proto change.
• Fixture-backed examples recomputed — N/A, no published example depends on these stamps.
• Redis writers/readers enumerated for every documented key — no key is added, removed, or written differently. seed-meta: for this seeder keeps the same newestItemAt / oldestItemAt / maxContentAgeMin fields, written by runSeed from contentMeta and read by api/health.js:997-1016.

[vercel]

@Yigtwxx is attempting to deploy a commit to the World Monitor Team on Vercel.

A member of the Team first needs to authorize it.

[koala73]

@Yigtwxx can you allow edits on this PR please so I push fixes instead of creating a new PR and pulling your commit in

[Yigtwxx]

Done — maintainer_can_modify is now true on this PR, so you can push directly instead of opening a second one.

Branch state, so you don't have to check it yourself: head e50d59a07, mergeable: true, 39 commits behind main — and none of those 39 touch either file this PR changes (scripts/seed-gdelt-intel.mjs, tests/seed-gdelt-intel-content-age.test.mjs), so your patch lands clean without a rebase. Happy to rebase first if you'd rather push onto current main.

Sorry this cost you the extra round on #6042 — the box was the only thing missing there.

Still open for you in this PR, if it is what you were about to push: defect 1 of #5858 (newestItemAt being a Math.max) is deliberately not implemented here. The three shapes are costed with file:line in the PR body, because the one the issue prefers changes the seed-envelope content-age contract that is mirrored in three files and has a parity verifier.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
worldmonitor	Ready	Preview	Aug 10, 2026 12:18pm

[Yigtwxx]

Thanks for taking the branch over — 1f1cf0da is better than what it replaces. Two things below: the red check, and an independent audit of your commit.

The unit failure on 59a4b6c is not from this branch

One test: tests/rate-limit.test.mts:363 — "paid-provider endpoint policies abort a stalled Redis fetch before failing closed (#6236)", the Redis transport must be cancelled, not left pending, false !== true (job). It exercises server/_shared/rate-limit.ts; nothing in this PR touches it. 22600 pass, 1 fail.

It is a timing race with a very thin margin, not a regression:

• server/_shared/rate-limit.ts:30 — ENDPOINT_RATE_LIMIT_TIMEOUT_MS = 25 under NODE_TEST_CONTEXT
• server/_shared/rate-limit.ts:34 — ENDPOINT_REDIS_ABORT_TIMEOUT_MS = 20

In production those are 5_000 / 4_500 — a 500 ms margin. The test-context values compress it to 5 ms, and the two timers do not start together: AbortSignal.timeout(20) is a per-request factory (:560) the Upstash SDK only calls when it builds the fetch, while the SDK's timeout: 25 (:565) starts at rl.limit(). I instrumented that exact path with the test's own stub fetch, 10 runs on an idle machine:

fetch (and its abort signal) armed at    8.3 – 71.1 ms after the call starts
decision returned at                    30.7 – 98.1 ms
margin by which the abort won            1.1 – 16.3 ms   (minimum 1.1 ms)

The assertion reads fetchAborted on the same tick the awaited call returns, so any scheduling delay larger than that margin flips it to false. --test-concurrency=16 on a 2-core runner is exactly that. It passed 3/3 in isolation here and 3/3 in an 8-file concurrent batch — it needs full-suite load to show up.

So a re-run of unit should clear it. If you want it closed for good, the two candidates are widening the test-context split (e.g. 10 / 50, preserving the production ratio instead of inverting it) or having the assertion poll for the abort with a bounded deadline rather than reading it synchronously. Happy to send either as its own PR — it does not belong on this branch.

Audit of 1f1cf0da, independently verified

The fetchFn({ runStartedAtMs }) change is the widest part of it, and it is safe. scripts/_seed-utils.mjs:2132 now calls every seeder's fetcher with an argument where withRetry previously called fn() with none (:817-821), so before this commit every fetchFn received undefined. I enumerated all runSeed call sites: 109 pass a named function; 92 declare no parameter at all; 10 declare one, and every one of those is a destructured options object with defaults (seed-china-corporate-disclosures, seed-china-decision-signals, seed-china-policy-events, seed-china-stock-connect, seed-conflict-intel, seed-fatf-listing, seed-gdelt-bulk-materializer, seed-natural-events, seed-sec-cik-map, seed-trade-flows) — none declares a runStartedAtMs key, so nothing is shadowed and no default is displaced. The 6 imported ones (buildPayload ×3, buildGasPayload, buildOilPayload, fetchChinaReleaseCalendar) are zero-arg or options-object as well. No call site changes behaviour, and no fetchFn does a truthiness check on its first argument that the new object could flip.

The fail-closed direction is the right one and I confirmed where it lands. All stamps beyond tolerance → contentMeta returns null → newestItemAt: null in the envelope → api/health.js:1387 (contentStale: contentAgeMin == null || …) → STALE_CONTENT. So a poisoned persisted stamp now surfaces instead of buying the cohort a clamped-fresh reading. That is strictly better than the clamp I shipped, and the "cache merge mints fresh health evidence from a poisoned value" framing is the part I missed.

One consequence worth knowing, not a defect. contentMeta(data, startMs) clamps freshly fetched stamps too: scripts/seed-gdelt-intel.mjs:548 writes fetchedAt from the wall clock at fetch time, which is always after startMs, so every topic fetched during the run reports newestItemAt = run start. Published content age is inflated by the run duration (bounded by the 300 s soft budget). Against maxContentAgeMin: 1440 that is noise, and the immutable-clock property is worth more than the precision — but it does mean newestItemAt is now "run start", not "fetch instant".

One line that may deserve a comment. withRetry(() => fetchFn({ runStartedAtMs: startMs })) pins the same clock across retries, so deadlineAt = runStartedAt + _softBudgetMs (scripts/seed-gdelt-intel.mjs:355) became a whole-run budget rather than a per-attempt one. For this seeder that is the safer direction — it also keeps the fetch phase far inside the one-hour skew tolerance — but it is a behaviour change for any future fetchFn that derives a budget from the clock and expects a fresh one per attempt.

Local verification on 59a4b6c:

tests/seed-gdelt-intel-content-age.test.mjs + seed-content-age-contract.test.mjs   30 pass, 0 fail
gdelt-fetch, seed-gdelt-intel-fetch-budget / -fetch-rotation / -merge-read-warns /
  -timeline-resilience, seed-utils, seed-utils-with-retry, seed-contract,
  seed-utils-sigterm-cleanup, seed-utils-empty-data-failure                       197 pass, 0 fail, 4 skipped
tests/rate-limit.test.mts alone                                                    44 pass, 0 fail (×3)

Defect 1 of #5858 (newestItemAt as a Math.max, so one refreshed topic hides five starving ones) is still unimplemented, with the three shapes costed in the PR body. Say the word and I will send whichever you pick, here or as a follow-up.

[Yigtwxx]

Sent the flake fix as its own PR so it stays off this branch: #6405.

It widens the NODE_TEST_CONTEXT decision deadline from 25 to 250 (abort stays at 20), which takes the headroom from a measured 1.1–16.3 ms to 164–220 ms, and pins the gap with a mutation-proven guard so it cannot silently shrink again. Production values are untouched. "Allow edits by maintainers" is on from the start this time.

This PR still just needs a unit re-run.