Geofences become visible and their news becomes geographic: the
dashboard draws every AOI with its last-sweep counts, and AOI news is
searched by the place an area sits in rather than by its name. The
roadmap's last planned item ships here; along the way, every live
GDELT fetch was found to have been silently failing and was fixed.
Added
- Dashboard AOI layer: a toggleable "AOI Geofences" Leaflet layer draws
every defined area (circle, polygon, corridor) from the shared store,
with a tooltip and detail panel showing what the last collector sweep
counted inside it. Backed by a new/api/aoisendpoint that reads
the stored change snapshot and never gathers live, so drawing a shape
costs nothing upstream. An AOI that has never been swept says so
(last_sweep: null) instead of showing zeros; a broken store is a 503
with the reason, never an empty map that reads as "no areas defined".
Verified in headless Chromium: 2 shapes rendered, the 50 km circle at
50,000 m, no console errors. First endpoint tests for the dashboard
(Starlette TestClient); a real SQLite thread-affinity error surfaced
during them and was fixed by opening the store per request.
Changed
- AOI news is now scoped by where the area IS, not what it is named.
Every other AOI domain was filtered geometrically; news alone was a
GDELT search for the AOI's name, so an area called "Home" returned a
Chinese A-share IPO article and "PGH Square" an Indian op-ed on Gaza
(both measured live 2026-09-02, both for a fence around Pittsburgh).
Newsources/geocode.pyreverse-geocodes the AOI centre via OSM
Nominatim (no key; 1 req/s floor and an identifying User-Agent per
its usage policy; 30-day cache since places do not move) and the
brief/changes/digest search GDELT for the settlement and county
(("Pittsburgh" OR "Allegheny County")). The brief reports how news
was scoped innews_scoping; when geocoding is unavailable (geocoder
down, open ocean) it falls back to the AOI name and says so in
data_gaps, because that is exactly when articles may be unrelated.
Fixed
- Every live GDELT fetch had been failing and hiding behind stale
cache. Root cause, measured: api.gdeltproject.org TCP-connects in
0.03 s but its TLS handshake took 19.7 s, 17.4 s, then >45 s across
three samples (Nominatim and USGS: 0.05 s on the same network), and
the Fetcher's 15 s default counts TLS inside the connect phase, so
every request was aConnectTimeout. Both GDELT call sites (news
search, company profile) now use a shared 45 sGDELT_TIMEOUT;
live-verified afterwards with 10 real articles for the Pittsburgh
AOI. Handshakes slower than that still degrade to the existing honest
error/degradedshape. - GDELT DOC API requires OR clauses in parentheses: a bare
"a" OR "b"
returns nothing at all (measured live). The news query builder wraps
multi-term queries; a regression test pins the exact form. - A failed Nominatim request (e.g. HTTP 429) was reported as "returned
an unexpected response shape", pointing a reader at parsing when the
cause was rate limiting. A failed fetch and a genuinely odd payload
now carry distinct messages.
Testing
- The AOI test suite had briefly become non-hermetic: the shared
domain stub did not cover the new geocoder, so ~150 tests were each
making a live Nominatim request (suite time 0.9 s -> 15.8 s, and a
real 429 from OSM). The stub now covers it (back to 0.7 s), proven by
temporarily making any live geocode call raise: only the six tests
that mock Nominatim themselves touched it.