Future Roadmap

Future Roadmap

Planned enhancements in priority order. Full design documents for major features are in the docs/ directory of the repository.

---

Phase 2 — Weekly intelligence summary

Status: ✅ Complete (May 2026). See Weekly-Reports for full documentation.

What was built

Every Sunday at 11:00 PM, a separate launchd job runs the weekly synthesis pipeline (Pass 6). It reads the past 7 days of daily briefs and correlation analyses from signal.db and makes a single LLM call to produce a weekly intelligence brief.

The weekly brief contains:

• Week in Review — the dominant political dynamic of the week as a whole
• Story Arc Tracker — how the top 3–5 stories evolved day by day
• What Escalated — items that grew in significance as the week progressed
• What Was Buried — stories that appeared then disappeared without resolution
• Blindspot of the Week — the most significant story both sides systematically underreported
• Watch List: Next Week — concrete forward-looking items with named entities and deadlines
• Analyst Note — a weekly assessment of trajectory

Key design decisions made

• Read from DB, not HTML files — the structured JSON in correlation_analyses and cluster_analyses is richer than re-parsing HTML. Each day's SITUATION OVERVIEW, WATCH LIST, narrative patterns, and anomalies are extracted and fed to Pass 6.
• Single LLM call — unlike daily Pass 1 (one call per article), the weekly synthesis is one large-context call. Claude handles this well; Ollama would struggle with the context window.
• Gold/amber visual identity — weekly reports use a gold accent scheme (#e3b341) instead of the daily blue, making them immediately distinguishable in the archive and on the landing page.
• Separate weekly_briefs DB table — weekly runs never interfere with daily data.
• Schedule: Sunday 11:00 PM — captures the full week including Sunday morning news, and publishes before Monday morning.

Files added

File	Purpose
pipeline/weekly.py	Pass 6 — reads DB, formats daily summaries, calls LLM
scripts/run_weekly_and_publish.sh	Shell wrapper: runs weekly pipeline + git push
scripts/com.flexrpl.signal.weekly.plist	launchd configuration for Sunday 11 PM

---

Phase 3 — Monthly intelligence summary

Status: Planned (target: late June 2026, once four weekly reports exist as inputs)

Every month, a monthly synthesis pass will read the four weekly briefs and produce a higher-order analysis focused on month-scale trends — story arc completion, what materialized from the watch list, and what dominated the month overall.

Design

The pattern mirrors Phase 2 exactly. A new pipeline/monthly.py module (Pass 7) reads from the weekly_briefs table instead of briefs, condenses each week's brief into key sections, and makes a single LLM call.

Dimension	Daily (Passes 1–5)	Weekly (Pass 6)	Monthly (Pass 7)
Input	RSS articles	7 daily briefs	4 weekly briefs
LLM calls	~150	1	1
DB source	articles, briefs	briefs, correlation_analyses	weekly_briefs
Schedule	Daily 5:00 AM	Sunday 11:00 PM	1st of month, 12:01 AM
Output	brief_YYYYMMDD_HHMM.html	weekly_YYYYWNN_HHMM.html	monthly_YYYY-MM_HHMM.html
Theme accent	Blue #58a6ff	Gold #e3b341	Green #3fb950

Proposed monthly brief sections

• Month in Review — the dominant political dynamic of the month as a whole
• Story Arc Completion — which stories that opened in the month resolved, escalated, or stalled
• What the Watch List Got Right — items from weekly watch lists that materialized
• What the Watch List Missed — significant developments that weren't anticipated
• Blindspot of the Month — the most significant story systematically underreported across the full month
• Watch List: Next Month — forward-looking items with named entities and expected timelines
• Analyst Note — monthly trajectory assessment

Files to add

File	Purpose
pipeline/monthly.py	Pass 7 — reads weekly_briefs, formats weekly summaries, calls LLM
pipeline/prompts.py	Add MONTHLY_BRIEF prompt constant
scripts/run_monthly_and_publish.sh	Shell wrapper: runs monthly pipeline + git push
scripts/com.flexrpl.signal.monthly.plist	launchd configuration for 1st of month, 12:01 AM

Prerequisite

At least four weekly reports must exist in signal.db before the monthly pipeline can run meaningfully. The first full month of weekly data will be available at the end of June 2026.

---

Pass 1 parallelization

Status: Deferred. Not necessary for current usage — the ~22 minute daily run completes well before the 5:00 AM window closes.

Current: Pass 1 runs 140–160 Claude CLI subprocess calls sequentially, taking ~22 minutes.

Proposed: Use concurrent.futures.ThreadPoolExecutor to run multiple calls in parallel:

from concurrent.futures import ThreadPoolExecutor, as_completed

with ThreadPoolExecutor(max_workers=10) as executor:
    futures = {
        executor.submit(_llm_call_claude, prompt, timeout): (article, db_id)
        for article, db_id, prompt in work_items
    }
    for future in as_completed(futures):
        article, db_id = futures[future]
        result = future.result()
        # process result...

Expected improvement: 10 workers × ~9 sec/call = Pass 1 completes in ~2–3 minutes instead of ~22. Full run time drops from 22 minutes to ~5–7 minutes.

Considerations:

• Claude API rate limits — Anthropic's API allows high concurrency on Pro/Max plans, but sustained 10-parallel subprocess calls may hit limits. Start with max_workers=5 and tune.
• The Rich progress bar needs updating to work with async completion order.

---

Formal test suite

Status: ✅ Complete (May 2026). See Testing for full documentation.

164 tests across all five pipeline modules. 91% overall coverage. analyzer.py at 100%. All LLM calls and network I/O are mocked — the suite runs in under two seconds.

---

Source expansion

Status: Ongoing as needed

Potential additions to improve spectrum coverage:

Source	Bias	Notes
New York Times	left	Requires RSS subscription
The Atlantic	center-left	Good long-form analysis
The Dispatch	center-right	Conservative anti-Trump perspective
The Intercept	far-left	Investigative/adversarial framing
Just the News	right	Alternative to Breitbart
Substack Politics aggregator	varies	Would require custom parser

---

Cross-run deduplication

Status: Planned

Currently, if an article is published at 11 PM, it may appear in both the Day 1 run (within the 24-hour window) and the Day 2 run (still within the window at 5 AM). URL deduplication is only within-run.

Proposed fix: Before saving articles in store.py, query for URLs already seen in recent runs:

SELECT url FROM articles
WHERE collected_at > datetime('now', '-48 hours')
AND url = ?

This would prevent re-analyzing the same article across consecutive days.

---

Email/push delivery

Status: Idea stage

Instead of only publishing to GitHub Pages, optionally email the daily brief to a subscriber list or push a notification with the report link.

Options:

• Email: Send the HTML report via smtplib or a service like SendGrid
• Push notification: Use a service like Pushover or ntfy.sh
• Slack/Discord webhook: Post a summary and link to a channel

This would be controlled by a new delivery: section in sources.yaml.

---

SonarQube quality gates

Status: In progress

Remaining issues are cognitive complexity warnings in reporter.py and main.py — primarily due to the large HTML template functions. Test coverage is no longer a blocker (91% overall). Key remaining categories:

• Code smells — overly complex functions in reporter.py (_render_story_cards, _render_brief_sections) and _update_index() in main.py
• Duplications — repeated HTML generation patterns across daily and weekly templates