v0.12.17 - 🔍 City Pulse Review Pipeline: launch to hardened, weight-aware methodology (v0.12.0–v0.12.17)

🚀 New: the City Pulse review pipeline

An admin-triggered AI agent that audits City Pulse's own operation — cost, translation caching, severity weights, status/trend usefulness, prompt quality — and proposes concrete fixes as a draft PR.

• 🧮 Deterministic reducer (review/reduce.py) turns days of raw debug logs + radar.db into a compact, cheap-to-read digest — no LLM spend on re-parsing boilerplate
• 🤖 Gemini reviewer (review/reviewer.py) reasons over the digest only, producing a human-readable report plus machine-readable proposed edits
• 🔀 CI-mediated draft PR — proposals never auto-apply; a GitHub Actions workflow turns accepted findings into a reviewable PR
• 🏷️ Every pulse run stamped with pulse_config_version so output-quality changes can be attributed to a specific methodology snapshot and compared version-over-version

🛡️ Hardened against real production data

Validated with a gold-standard methodology — an independent human read of real digests vs. the actual Gemini report — catching real bugs before trusting the pipeline's own output:

• 🐳 Fixed missing review/+prompts/ in the web Docker image, and a missing GEMINI_API_KEY on the web container
• 🔧 Fixed the reducer's service filter polluting pulse_hours with extraction/daily records
• 📅 Fixed a cost-reconciliation month-boundary double-count
• 📊 Split paid_churn/cache_churn so a zero-cost flapping alert can't be mistaken for a cost driver
• 📈 Added a status_distribution histogram so the reviewer stops tallying status counts by eye
• ✍️ Corrected reviewer guidance on false "never reaches clear" findings
• 🏷️ Renamed a naming-collision field (prompt_samples → prompt_sample_texts) that caused a hallucinated setting

🎯 Reviewer can now actually judge and propose severity weights

The reviewer had no repo access and no visibility into today's actual weight values or thresholds — it could only guess. Fixed:

• ⚖️ weight_tables — the live SEVERITY_WEIGHTS_DWD/SERVICE_WEIGHTS_RMV/etc. values, read straight off pulse_categories.py
• 📐 Per-hour baseline (mean/p25/p75) alongside each category's score — the actual threshold a status was judged against
• 📝 top_drivers now carry title/source/body, not just id+weight, so a proposed change can cite what an alert actually said

🔧 Pulse-system fixes

• ✂️ Cap prompt alerts by severity rank instead of sending every alert (140 alerts / 68KB → bounded, ranked)
• 🌐 Difflib-based similarity threshold before paying for a retranslation on a cosmetic body edit
• 🕵️ Explicit root-cause-naming instruction so the pulse LLM connects correlated disruptions to their shared cause

🖥️ Admin dashboard cleanup

• 🎨 Colored pill for the review service in the Gemini log
• 🧹 Trimmed redundant status dots (RAM/Load — already on the health chart; StaticEvents — can't fail; Cron — duplicates "Last poll")
• 🏷️ Renamed "Extraction" health check to "Gemini"
• 🔴 "Last poll" text turns red once it exceeds the configured stale threshold
• 💾 Added current disk usage display
• 🗑️ Added 30-day retention to review_debug/ — previously the only debug directory with none

🐛 Fixes

• 💶 Fixed cost.monthly_budget_eur → cost.monthly_budget key mismatch (admin budget always silently fell back to a hardcoded €10 default)
• 🗂️ Moved translation_min_change_ratio under translator: in config.yaml (it's a caching knob, not pulse methodology)

v0.12.0 — 🔎 City Pulse Review Pipeline

🔎 v0.12.0 — City Pulse Review Pipeline

An admin-triggered AI agent that audits City Pulse's own operation — cost, translation, severity weights, status/trend usefulness, prompt quality, and cross-version comparison — and proposes concrete methodology improvements. Deterministic work is a script; judgment is the AI.

🏷️ Versioning

• 🧬 Stamp every pulse run with pulse_config_version — a hash of the prompt, weight table, and window/strategy config — so future reviews can attribute output changes to a specific methodology snapshot

⚙️ Deterministic reducer

• 📉 review/reduce.py turns days of cost_debug/translate_debug/pulse_debug logs plus radar.db into a compact digest — no LLM, no network
• 📊 Per-version metrics: status flap rate, trend-override rate, cost per pulse, coverage, and override rate (when human corrections exist)
• 🧾 Cost reconciliation against api_usage, pulse coverage gap detection, and event-log anomaly surfacing
• 🔒 subscribers table never queried — GDPR redaction by construction, not discipline

🤖 Gemini reviewer

• 📝 review/reviewer.py reasons over the digest only, producing a human-readable report plus machine-readable changes.json
• 💾 Reports and proposed edits written to the data volume (data/review_debug/), never the repo — reviewer never edits the running system

🖥️ Admin dashboard

• ✂️ Retire the standalone weight-review button — folded into the unified review
• 🎛️ New "Run review" control: days/drivers/samples knobs, high/low-detail presets, cost preview gating the run, and a past-reports list

🔀 CI draft PR

• 🔑 New X-Admin-Token header auth path (alongside session login) so CI can fetch a report's changes.json without a browser session — prod server still never holds a GitHub token
• 🌿 .github/workflows/review-pr.yml: manual workflow_dispatch job that applies unified-diff-shaped changes via git apply, writes anything else to PROPOSED_CHANGES.md for manual review, and opens a draft PR — never auto-merges

📚 Docs

• 📖 docs/review.md — full pipeline design, linked from analysis.md

v0.11.34 — 🩺 City Pulse charting overhaul, deterministic status hardening & mobile fixes

🩺 v0.11.34

Ten days of work on City Pulse's deterministic layer, a full charting migration to Chart.js, and a round of mobile/map polish.

📊 Charts & methodology page

• 🔄 Migrate category status/trend charts to Chart.js, replacing the old dot/range plot
• 🎨 Replace the thin p25–p75 whisker with a translucent baseline band
• 🗂️ Reorganize the pulse methodology page and add AI narrative/overrides evidence
• 🖼️ Render cited alerts as icon+title cards instead of raw links
• 📝 Link methodology page to analysis.md instead of duplicating detail; surface AI inputs
• 🐛 Restore histScores in selectCell, broken by the Chart.js migration
• 🐛 Expose lookahead/lead_alert in the pulse-methodology-data API

🧮 Deterministic status/trend engine

• 🛡️ Deterministic status/trend hardening: lagged baseline, hysteresis, weights versioning
• ⏱️ Give lead-alert its own window, decoupled from projected_score
• 📉 Rebase surge baseline to scheduled load instead of noisy ongoing score
• ⚡ Regenerate immediately when hysteresis flips a fast category's status during a skip window

🖥️ Admin dashboard

• 📈 Add a 24h system health chart with failure/recovery/restart events

📱 Mobile & map UX

• 🚫 Don't close pulse/weather overlays on scroll gestures when expanded
• 📐 Fixed 340px width for pulse/weather overlays; unify disclaimer font
• 🗺️ Consistent map popup width/dark-mode; extend incident-circle UX to strike/fire
• 🔗 Fix URL-shaped alert IDs breaking Telegram deep links (feuerwehr, polizei, strike)

🧹 Reliability & cleanup

• 🔁 Reconcile strike_duplicates cleanup with the active-strike check it must match
• 💰 Stop re-paying for polizei/strike extraction on entries that get discarded

📚 Docs

• 📖 Full audit and rewrite of README, docs/, and policy files

v0.11.24 — 🌆 Deterministic Pulse Engine, Admin Dashboard & Feuerwehr Alerts

🌆 City Pulse — deterministic status engine

• 🎯 Deterministic status + self-improving calibration loop
• 📊 Three-signal status judgment with statistical baseline
• 📈 p25 baseline stat as symmetric improving-trend signal
• 🔁 Redefined horizon score as forward projection over the full lookahead window
• ⏱️ Hour-slot comparison for adaptive pulse skip; narrow 1h override to transport/weather
• 📝 Rewrote pulse.md/analysis.md — consistent with deterministic status + LLM trend-only architecture
• 📄 New City Pulse methodology page with live data
• ⚖️ Refined severity weights — tram/bus/strike/police, RMV lines scaling (capped at 4)
• 🎨 Unified City Pulse status visuals across main page, admin, and methodology page

🛠️ Admin dashboard

• 🔍 Category score drill-down, translate log filter
• 🌓 Dark mode, themed scrollbars, transparent dark scrollbar track
• 📅 Frankfurt date picker default, chronological category grid, flipped grid (rows=time, cols=category)
• 📌 vs-last-month KPI, relative delta labels on mini charts, 6-column KPI row on desktop
• 📖 Detail panel — narrative at top, trend + status pills, score content
• 🐛 Fixed crashes (missing let body, bMean/bP75 used before declaration), variant-hit % denominator

🚒 New alert source

• 🆕 Feuerwehr Frankfurt (fire alerts) via Bluesky AT Protocol
• 🔑 Keyword subscriptions — receive any alert matching a location keyword

💰 Cost & translation

• ✂️ Cap translation length, dedup cross-alert-id duplicate translations
• 🕐 Correct cost_debug hourly snapshot timing; hourly deterministic pulse status
• 🚫 Skip retranslation when only the RMV "Stand:" timestamp changed in the title
• 🧹 Removed midnight daily admin report (visits + costs)

🌦️ Weather overlay

• 📏 Compact vertical spacing across the card
• 🌧️ Rainfall (mm) replaces cloud cover %; ↑/↓ high/low replaces dew point line
• 💬 Hover tooltips — humidity (RH + dew point), wind/rain 10-min values, hourly rain chance + mm, high/low
• 📱 Mobile: pushed below reset/close buttons

🐛 Other fixes

• 📍 Persist lat/lon/location_label on alert cache updates
• 🗺️ Correct Feuerwehr UI theming and map pin location
• 📋 Add Feuerwehr and Messe Frankfurt to info/disclaimer modals
• 📱 Mobile: always-visible sub-type pickers, emoji labels, overlay/button stacking, pulse-overlay height cap
• 🐛 Pulse methodology page shows last skipped pulse instead of last real pulse
• 🐛 Debounce City Pulse overlay show on reset view transition

Full Changelog: v0.11.0...v0.11.24

v0.11.0 — 🛠️ Admin Dashboard

🛠️ New feature

• 📊 Admin dashboard — cost monitoring, server status, and ban management in one place

Full Changelog: v0.10.16...v0.11.0

v0.10.16 — 🌦️ Weather Overlay Launch, Cost Controls & Spatial Pulse

🌦️ Weather overlay (new)

• 🆕 Weather overlay replaces the old DWD popup — current conditions + forecast
• 🇩🇪 Use German DWD alerts for richer content, translate to English
• 👆 Swipe gestures matching City Pulse behavior (down expands, up collapses)
• 📐 Auto-height layout, compact single-row stats, larger forecast, width matched to pulse overlay
• 📱 Mobile fixes — collapsible conditions section, no double border, full expansion, no stray scrollbar
• 🐛 Fixed display bug where inline clear was blocked by CSS display:none
• ✂️ Separated description/instruction paragraphs, collapsed stray newlines

💰 Cost & translation optimization

• 📉 Adaptive frequency + leaner input to reduce Gemini pulse costs
• 🧠 Separate thinking-token cost tracking with reduced thinking budgets
• 🔁 Translation variant cache eliminates flip-flop retranslation costs
• 🚫 Skip re-translation when only metadata changed (compare German text)
• ✂️ Drop DWD instruction_de from alert body to cut translation costs
• 🛑 Prevent mass retranslation on poller failure and strike-expiration loops
• 📊 /costs command (renamed from /cost) with hourly/daily/monthly breakdown + midnight auto-report
• 🗒️ Daily debug logs, translate_debug and pulse_debug folders logging every cycle

🌆 City Pulse — spatial awareness

• 🗺️ District-level spatial awareness for pulse synthesis
• 📍 Pass location_label to the pulse LLM, add spatial convergence guidance
• 🚫 Admin /pulse bypasses the adaptive frequency check
• 📝 Log skipped pulses with reason and category state

🌓 Dark mode & card selection fixes

• 🎨 Stronger neutral tint for card selection, no per-source colors
• 🔒 !important override to fix selected-card styling across all variants
• 🧩 Override src- attribute selector for selected card border

🐛 Other fixes

• 🕐 Fixed city pulse not delivered — UTC/Berlin hour mismatch
• 🌍 Normalize all dates to UTC on ingestion, display in Europe/Berlin
• ⏰ Move daily admin report cron to notifier container
• 🚓 Stop re-geocoding cached police alerts on every poll cycle
• 🔤 Fix cron broken by em dashes in comments; improve pulse Telegram format
• 🩹 Fix /costs missing get_api_usage import

Full Changelog: v0.10.0...v0.10.16

v0.10.0 — ✨ Cost Tracking & Pulse Overlay Polish

✨ New features

• 💰 /cost admin command — API usage tracking
• ☂️ Weather widget click now opens the latest DWD alert if one is available

🎨 UI polish

• 🗂️ Improved City Pulse overlay visual hierarchy and grouping
• 🔀 Reordered filter pills — roadworks before incidents
• 🏷️ Renamed toggle to "Show upcoming" to avoid ambiguity with "events"

🐛 Fixes

• ⌨️ City Pulse card now included in keyboard up/down navigation
• ⚡ Skip LLM re-extraction for cached strike alerts (cost/latency saving)

Full Changelog: v0.9.25...v0.10.0

v0.9.25 — 🌆 City Pulse Status Engine & Overlay Redesign

🌆 City Pulse status & trend engine

• 🔁 Replaced EWMA with a per-category time-series model judged by the LLM for status/trend
• 🎯 Replaced "upcoming score" with a projected score (net direction signal)
• 📈 Split trend into next-interval projection vs. horizon momentum, with horizon samples folded into history
• 🧭 Sigma-band thresholds decouple status from trend detection
• ⏱️ Ongoing-only temporal filtering + severity weighting for trend analysis
• 🧩 Status-level guidance added to the LLM prompt, with color-coded status in the overlay
• 🗒️ Score breakdown added to the pulse debug log
• 🏷️ Pulse title field added; disclaimer toggle cleaned up

🎨 Overlay & UI redesign

• 💓 ECG heartbeat animation in the City Pulse overlay header
• 👆 Swipe gestures, table layout, and legend spacing overhaul for the pulse overlay
• 🌓 Dark mode UX pass — unified navy theme, relative timestamps throughout
• 🧱 Pulse overlay redesigned as a centered single-column table with inline trend words
• 📍 Smart ending labels for status pills ("in 1 day", "starting soon", "X hrs")
• 🎨 Source-colored blink + ticker surfaces events starting/ending within 2 days
• 📌 De-pinned weather cards; added "Starts in" pills; grouped incidents filter

🎪 Content & filters

• 🎟️ Added Messe Frankfurt trade fairs, restructured events filter
• 📌 Pin support extended to Messe as a source

🐛 Fixes

• 🚫 Ticker excludes ONGOING pills — caps active alerts at 1hr before ending
• 🗃️ DB migration for projected_count/score columns on category_snapshots
• 🎭 Renamed pulse-summary pill class to pill-pulse to avoid a CSS collision
• 📵 Hide City Pulse card during search
• 📱 Unified mobile map close behavior; removed "Posted at" from the pulse card
• 🕐 "Posted" prefix on relative timestamps used consistently in the pulse overlay

Full Changelog: v0.9.5...v0.9.25

v0.9.5 — 📊 EWMA City Pulse, AI Transparency & Feed Integration

What's New

📈 EWMA-based status & trend detection

Replaced the 7-day rolling average with Exponential Weighted Moving Average (α=0.3) for both status classification and trend detection. Long-running alerts naturally fade into the baseline, eliminating the cross-hour comparison problem. Works from the first pulse — no 7-day warm-up needed.

🔗 Pulse references & AI transparency

• LLM returns the top 3 alerts that most influenced the summary — shown as clickable deep links with source icons in the pulse overlay
• Always-visible AI disclaimer toggle: "AI-generated summary · verify before acting → Click for sources & methodology"
• Collapsible section reveals referenced alerts, EWMA methodology explanation, and How it works link

📋 City Pulse feed card

City Pulse now appears as a pinned card at the top of the alert feed — same layout as other alerts with summary title, timestamp, and green ONGOING pill. Click resets the map to default view and opens the pulse overlay. Works on both desktop and mobile.

🤖 Deterministic pipeline improvements

• travel_ok, category statuses, and trends are all script-computed — the LLM only writes the summary and recommendation
• get_city_pulse MCP tool for programmatic access
• Prompt rewrite: synthesis hierarchy, spatial awareness, no alarmist tone, 300-char limit

🐛 Bug fixes

• Duplicate pulse notifications — pulse was sent every 10 minutes for the full hour due to missing dedup; now tracks last_pulse_at per subscriber and sends only once
• Stale pulse at hour mark — dispatch could send the previous hour's pulse before the new one was generated; now verifies generated_at matches the current hour
• Pulse references not rendering — window.alerts was undefined because alerts are let-scoped; fixed to use direct variable references
• Strike dark theme missing for filter pill and card hover states

🎨 UI polish

• Two-line centered disclaimer with dotted underline clickable affordance
• Mobile UX: city view toggle, pulse auto-show, fade animation
• Dark theme consistency across all components

v0.9.0 — 📊 City Pulse: AI Situational Summary

📊 City Pulse

Frankfurt Radar now generates an hourly AI-powered situational summary using Gemini 2.5 Flash with extended thinking. Instead of scanning individual alert cards, users get a concise synthesis of what's happening, what's changing, and what to do about it.

✨ Highlights

• Hourly pulse — AI synthesis of all active alerts, displayed as a floating overlay on the map
• Daily digest — 24 hourly pulses compressed into a one-paragraph summary at 23:00, providing multi-day trend context
• Telegram delivery — subscribers choose when to receive the pulse (08:00, 12:00, or 18:00)
• 5 trend categories — Weather ⛈️ · Transport 🚇 · Roadworks 🚧 · Incidents 🚨 · Events 🎉
• 3 trend directions — ↗ worsening · ↘ improving · → stable
• Smart prioritization — new alerts emphasized, old low-severity alerts deprioritized
• Proactive recommendations — suggests alternative routes during disruptions, events during good weather
• Editable prompts — YAML frontmatter + template in prompts/, seeded to data volume
• Admin /pulse command — trigger generation manually via Telegram
• One pulse per hour — dedup prevents history pollution from repeated manual triggers

🔒 Security Audit Remediation (#264–#268)

• Gitleaks allowlist for false positives (+ config format fix)
• SECURITY.md ↔ security.html sync
• Caddyfile subdomain security headers
• BIND_ADDR env var — 127.0.0.1 for production behind Caddy, 0.0.0.0 for self-hosters
• _allow_manual_poll safe default

🐛 Bug Fixes

• Deep link auto-opens cleared section for removed alerts
• SOURCE_URL fallbacks for baustellen/sports/strike + synced officialUrl in popups
• Radar frames re-fetched on each open (stale "now" marker fix)

📝 Documentation

• README: City Pulse feature section + updated architecture diagram
• Architecture, user guide, self-hosting docs updated
• PRIVACY.md + privacy.html: Gemini API data processing disclosure

🚀 Deploy

# Add to .env
BIND_ADDR=127.0.0.1

# Add to data/config.yaml
pulse:
  enabled: true

# Deploy
rm -rf data/prompts
docker compose pull && docker compose up -d