perf(dashboard): trim logs to first+last 3 and slim top-level news#62
Merged
Conversation
Daily JSON files were 65% larger than necessary (877KB → 305KB on the
largest day) due to two sources of bloat:
1. Logs: every incident shipped its FULL log timeline (up to 11 logs with
139+ news items for long-running incidents like Venezuela). Now trimmed
to first log (genesis) + last 3 logs (recent activity). Frontend shows
a ⋮ separator with a tooltip indicating how many logs were hidden.
Header reads '4 of 11 log(s)' instead of '11 log(s)'.
2. Top-level news array: carried full 7-field dicts (url, body, image,
outlet...) per news item, but the frontend only uses headline (search
haystack) and published_date (recent activity filter) when logs exist.
Now slimmed to {headline, published_date} when logs exist — the full
dicts in logs[].news[] are unchanged (drawer timeline reads from there).
Added logs_total field to incident dict so the frontend can detect
trimming. SCHEMA_VERSION bumped to 1.5.
nullhack
added a commit
that referenced
this pull request
Jul 24, 2026
Dashboard data regenerated with log trimming and news slimming. 65% size reduction on the largest daily JSON (877KB → 305KB).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Daily JSON files were 65% larger than necessary (877KB → 305KB on the largest day). Two sources of bloat:
Logs: every incident shipped its FULL log timeline (up to 11 logs with 139+ news items for long-running incidents like Venezuela). Drawer timeline was correspondingly very long.
Top-level news array: carried full 7-field dicts per news item, but the frontend only uses
headline(search haystack) andpublished_date(recent activity filter) when logs exist. The full dicts are never rendered — the drawer timeline reads fromlogs[].news[]instead.Changes
scripts/generate_dashboard_data.pyMAX_RECENT_LOGS = 3constant[first_log] + last_3_logswhenlen > 4logs_totalfield to incident dict (original count before trimming)newsarray slimmed to{headline, published_date}when logs exist; full dicts kept when no logs (drawer fallback)SCHEMA_VERSIONbumped to1.5dashboard/app.js_logEntryHtml(log)+_timelineHtml(logs, logsTotal)helpersTimeline · 4 of 11 log(s)when trimmedN log(s) hiddendashboard/styles.css.log-entry--gap: centered muted ⋮ with bottom borderImpact
Only 24 of 1065 incident-appearances across all daily JSONs have >4 logs — most incidents are unaffected. Search and recent activity tabs work unchanged (they only need headline + published_date).