Refresh each watchlist independently, show its real age, and alert when it goes stale

[hseshadr]

The incident this fixes

The daily publish-watchlist cron has been failing on EU_CONSOLIDATED: required feed fetch failed: EU request failed: 500 Internal Server Error. The EU webgate is still
500ing.

The publisher was right to refuse. It failed closed rather than sign a bundle missing
a required list, and that behaviour is preserved. The bug is the coupling: all four
feeds were required together, so a European outage blocked the OFAC refresh — and OFAC
is the list every visitor screens against by default. Users who never enabled EU were
getting stale data because of a European outage.

Claim touched

"You are screening against these lists", and the README's "runs daily".

Both were claims about coverage and freshness that nothing enforced.

---

1. Per-list independence, fail-closed preserved per list

Each list is now refreshed — and therefore fails — on its own. A list whose upstream is
unreachable is re-served from the bundle already published: fetched back through the
full trust chain (signed pointer against the pinned key → manifest content-address →
every chunk through the client's zstd-decompress → sha256 → each file against its
file_sha256), keeping its own older version and its own fetchedAt, stamped
stale with the upstream failure verbatim.

This is not "feeds are optional". A list that can be neither refreshed nor proven
from the published bundle still aborts the publish. Nothing is dropped from the catalog,
so a bundle can never assert coverage it does not have.

Contract reversed, deliberately. The test asserting that any unfetchable feed aborts
the bundle was inverted, not deleted — its fail-closed half survives as "cannot be
fetched AND cannot be carried".

No format change to the signed pointer

Per-list freshness rides inside the staged tree (catalog.json and each
<slug>/meta.json), so it inherits the existing trust chain for free:
chunk hash → file_sha256 → manifest hash → signed pointer. Nothing is added to
/latest, so its canonical signing preimage, its signature, and its monotonic
anti-rollback sequence are untouched.

Already-published bundles. Every bundle published before this carries no fetchedAt.
Carry-forward and the freshness guard both fall back to that bundle's generatedAt —
truthful, because in the old model all four lists refreshed in one run — and refuse
outright if neither can age the list. A present-but-malformed fetchedAt is corruption,
not migration, and still fails. No timestamp is ever invented.

2. The age is on screen

CatalogListInfo was {id, title}, which is precisely why no UI could show an age. It
now carries version, entity count and the four freshness fields. Settings renders
"updated 4 hours ago" when fresh and "Not updated for 3 days — the EU feed was
unreachable" when not, as text inside the checkbox's own label (part of its accessible
name, not colour alone). An unparseable fetchedAt renders "Age unknown", never "fresh".

The client guards now actually guard: assertBundleListMeta declared generatedAt and
model and never narrowed them. All four freshness fields are now required and validated.

3. The freshness guard

watchlist-freshness.yml runs 4×/day, verifies the live pointer's signature, reads
catalog.json back through the client's decode path, grades bundle-level and per-list
ages against a 26h ceiling, and opens/updates a single GitHub issue on breach.

It deliberately does not rely on being noticed — a red scheduled run is exactly the signal
that failed here. The alert step runs first; the failing step runs last.

---

Evidence

Claim	Proof
Live bundle age	version 2026-08-01, sequence 50, generatedAt 2026-08-01T18:23:54.891Z → 19.0h at check time. Pointer served application/json, signature-verified.
Carry-forward works on real production bytes	carryForwardList against aml-filter.com: 6,239 EU entities, vectors 2,395,776 = 6239×384, version kept at 2026-08-01, aged 18.8h via the generatedAt migration path, stale: true.
Guard passes when fresh	live prod, default 26h ceiling → exit 0, prints the per-list table.
Guard bites on real bytes	live prod, --max-age-hours 1 → exit 1, naming all four lists at their real 19.0h age.
Chunk-dedupe not regressed by the re-signed fixtures	delta-sync.spec.ts + scoped-sync-topup.spec.ts → 2 passed in real headless Chromium. Static: cold scoped sync 4/13 chunks ((root)+ofac only), EU top-up 3 chunks (eu only), re-fetched already-local chunks = [].
Demo bundles deterministic + residue-free	Full regen produces a byte-identical tree (git status empty) and leaves no .mutation.lock / chunks/ / manifests/.
Typecheck	publisher, browser, workstation, app — all clean under the commands CI runs.

The live red run

check-published-freshness: published bundle is STALE — aml-filter.com promises a daily
rebuild at most 1h old, and the LIVE origin (version 2026-08-01, sequence 50) breaches that:
  - the last successful refresh was 19.0h ago (generatedAt 2026-08-01T18:23:54.891Z), past the 1h ceiling — the daily rebuild is not running
  - OFAC_SDN (ofac): last refreshed 19.0h ago (aged from the bundle's generatedAt — pre-per-list-freshness bundle), past the 1h ceiling
  - UN_CONSOLIDATED (un): last refreshed 19.0h ago (…), past the 1h ceiling
  - EU_CONSOLIDATED (eu): last refreshed 19.0h ago (…), past the 1h ceiling
  - UK_OFSI (uk): last refreshed 19.0h ago (…), past the 1h ceiling
exit 1

Mutation proof — the decoupling tests can fail

mutation	result
carried list silently dropped from the bundle	2 failed / 39
carried list restamped fetchedAt: now (the "looks fresh" lie)	1 failed / 40
carried list reports stale: false	1 failed / 40
carried list restamped with today's version	1 failed / 40
restored	41 passed

The first row is the important one: a per-list staleness test that passed because the list
was absent would prove nothing. Dropping the list turns it red.

Client side: stubbing hasListFreshness to return true turns 19 of 36 browser tests
red.

---

Cron failure history — worse than it looked

46 scheduled runs since 2026-06-21, 25 failures:

cause	count	when
WATCHLIST_SIGNING_KEY secret is not set	22	2026-06-21 → 07-12, consecutive
pnpm minimum-release-age gate	1	07-21
OFAC 403 (pre-CSL Treasury WAF)	1	07-31
EU 500	1	08-02

Feed flakiness is 2 in 46. The headline is the other number: the publish job was
dead for 22 straight days and nobody noticed, because a failed scheduled run notifies no
one. That is the case for the guard, independent of the decoupling.

EU feed: transient outage, not a moved endpoint

Every FSD format 500s (xmlFullSanctionsList_1_1, xmlFullSanctionsList, both CSV
variants, xmlActiveSanctionsList_1_1); the path is unchanged; the static token is still
accepted (no 401/403 on file paths); the body is the webgate's own JSON error; and
08-01 succeeded through this exact URL. fetchWithTimeout already retries 3× with
backoff and treats 500 as retryable — those retries were exhausted in ~47s. Retrying
harder is not the fix; decoupling is.

Docs

The README/DEPLOY "runs daily" wording implied every list is under 24h old. Replaced with
what is true and now enforced: each list is refreshed daily when its source is reachable,
and every list shows its own last successful update time.

Deliberately not done

Per-list staleness has no hard ceiling at which coverage is withdrawn. Past some age a
carried list is arguably worse than an absent one — but refusing to publish would re-couple
the lists and age OFAC again, which is the defect being fixed. Left to the guard (red at
26h) on purpose; flagging it as the one policy call worth a second opinion.

🤖 Generated with Claude Code

https://claude.ai/code/session_0186xrrT9mYfCZTVkk8HN12p

Deploy hazard found and fixed mid-review

Making the client require per-list fetchedAt would have rejected the bundle production
is serving right now. Verified by running the live bytes through the branch's own
predicate, not by reading source:

LIVE bundle: version=2026-08-01 sequence=50  catalog.generatedAt=2026-08-01T18:23:54.891Z
  OFAC_SDN         fetchedAt=undefined  hasListFreshness=false
  UN_CONSOLIDATED  fetchedAt=undefined  hasListFreshness=false
  EU_CONSOLIDATED  fetchedAt=undefined  hasListFreshness=false
  UK_OFSI          fetchedAt=undefined  hasListFreshness=false

isBundleCatalogList gates on hasListFreshness, and assertBundleCatalog is on the
bundle-open path — so a strict client paired with a pre-freshness bundle stops loading the
watchlist. That pairing is reachable: deploy.yml's mirror-published-origin fallback
re-serves the OLD signed bundle whenever a fresh one cannot be built.

Fixed as a migration, not a flip — enforcement must never ship ahead of the data it
enforces. A legacy entry with no fetchedAt is aged from the catalog's generatedAt; a
fetchedAt that is present but malformed is still rejected, so the fallback cannot launder
corruption. The removal condition is recorded in the code: once every published bundle
carries per-list fetchedAt, the fallback comes out of all call sites together.

The same rule now lives in one shared predicate consumed by both the client and the
publisher, with a cross-implementation test running identical inputs through both and
asserting identical verdicts. Two copies of one rule diverge — that has already produced a
real defect in this portfolio.

The stale panel, rendered in a real browser

Not inferred from shared logic. A bundle variant with OFAC stale was published, served
under the production CSP, and /screen was driven in headless Chromium:

text                  : "⚠ OFAC SDN · Not updated for 3 days — the OFAC feed was unreachable"
role                  : alert
aboveTheFold          : true      (top=244px, height=43px, viewport 800px, scrollY=0)
hitAtCentre           : p.screen-list-age
panelOwnsItsOwnCentre : true      <- elementFromPoint at its centre returns the panel
contrast              : 7.60:1    (WCAG AAA for normal text)
console errors        : none

The elementFromPoint check is deliberate. A role="alert" panel can be perfectly
correct in logic and still be painted over by a sticky nav that comes later in the DOM —
that exact bug survived months in a sibling repo while every unit test passed. This one
owns its own centre point.

Screenshots: screen-STALE-panel.png (stale /screen), settings-freshness.png
(per-list ages in Settings, EU flagged red), screen-list-age.png (fresh /screen).

Chunk-dedupe blocker — closed, with before/after

The 46.6 → 28.2 MB cold-download cut rests on one property: enabling a list later
re-fetches nothing already local, asserted as an empty set intersection. The bundle
regeneration could have moved chunk boundaries. It did not:

	total chunks	cold scoped sync	cold lists	EU top-up	top-up lists	re-fetched
before (main)	13	4	(root)+ofac	3	eu	0
after (branch)	13	4	(root)+ofac	3	eu	0

Only the cold plaintext size moved, 6,043 → 6,642 bytes (+599) — the freshness fields
themselves. Chunk topology is identical because the tree is still one chunk per file.

Confirmed at runtime, not just statically: delta-sync.spec.ts +
scoped-sync-topup.spec.ts → 2 passed in real headless Chromium, on a pinned port
with reuseExistingServer disabled so no other agent's server could be measured by
mistake.

What was NOT verified — stated plainly

• The "Age unknown" path was never driven live. No bundle stages an unparseable
  fetchedAt, so that branch is covered by unit tests only.
• Headless Chromium at one desktop viewport only. No Safari, no mobile, no other
  viewport. Four live-driven cases; do not read them as six.
• Console cleanliness was measured, not assumed. Zero console errors during the
  /screen stale flow, and the listener was proven live by making the page emit a
  deliberate console.error and asserting it was caught. A zero from an unproven
  listener is worth nothing.

Follow-up with a dated trigger, not a TODO

The shared freshness predicate is deliberately permissive right now: an absent
per-list fetchedAt falls back to the catalog's generatedAt. That is required while
production still serves a pre-freshness bundle.

Flip it to strict when the live origin carries per-list fetchedAt — check with:

curl -s https://aml-filter.com/bundle/origin/latest
# then decode catalog.json and confirm every entry has a fetchedAt
pnpm --filter @amlfilter/publisher run check-published-freshness -- \
  --base-url https://aml-filter.com/bundle/origin --pubkey frontend/app/public/public.key
# the table's `source` column must read `fetchedAt`, not `generatedAt`, for all four lists

Because the rule lives in one shared predicate with two callers, that flip is one change
in one file and both sides move together. There is never a moment where the CI gate
accepts what the app rejects.

Hazard: before / after, same live bytes

https://aml-filter.com/bundle/origin, version=2026-08-01 sequence=50,
catalog.generatedAt=2026-08-01T18:23:54.891Z, no per-list fetchedAt on any entry.

	OFAC_SDN	UN	EU	UK_OFSI	publisher gate
before	REJECT	REJECT	REJECT	REJECT	ACCEPT ← the disagreement
after	ACCEPT (generatedAt)	ACCEPT	ACCEPT	ACCEPT	ACCEPT

One predicate, two callers, verified against production bytes rather than fixtures.
freshnessParity.test.ts runs identical inputs through both and asserts equal verdicts, so
they cannot drift apart again. Mutating the shared rule to let a malformed fetchedAt fall
through to generatedAt turns 23 tests red across both packages.

CI on the final commit: frontend pass (4m35s), gitleaks pass, GitGuardian pass.

[cursor]

Bugbot is not enabled for your account, so this pull request was not reviewed.

Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs.