Future Roadmap

Future Roadmap

Planned enhancements in priority order. Full design documents for major features are in the docs/ directory of the repository.

---

Phase 2 — Weekly intelligence summary

Status: Planned. Design doc: docs/phase2-weekly-summary.md

Concept

Every Sunday, a new pipeline pass reviews all daily briefs from the preceding week and synthesizes a weekly intelligence summary. This is a higher-order analysis — the LLM reads across seven days of analysis rather than a single day's raw articles.

What the weekly brief will contain

• Week in review — the dominant themes and how they evolved day by day
• Storylines that developed — how major stories progressed, escalated, or resolved over the week
• Watch list tracking — items from daily watch lists: what materialized, what stalled, what surprised
• Structural patterns — recurring narratives, coordinated messaging campaigns, sustained blindspots
• Week-ahead outlook — forward-looking assessment of what to watch next week

Implementation plan

1. New CLI flag — python3 main.py --weekly triggers the weekly summary mode
2. Data source — queries the briefs table in signal.db for the last 7 completed runs
3. New prompt — WEEKLY_SUMMARY in prompts.py, designed for multi-day synthesis
4. New report template — reporter.py gets a generate_weekly_report() function
5. Separate output directory — weekly reports go in reports/weekly/
6. Archive integration — archive.html gets a "Weekly" section
7. launchd scheduling — new plist entry for Sunday 6:00 AM

Key design decisions

• The weekly brief analyzes the brief texts (Pass 5 outputs), not raw articles — this keeps the context window manageable and focuses on synthesized intelligence rather than raw content
• The weekly run should use Claude (same rationale as daily runs)
• Weekly reports should be clearly distinguished from daily reports in the archive — different color scheme or badge

---

Pass 1 parallelization

Status: Planned. Priority: High (largest performance improvement available)

Current: Pass 1 runs 140–160 Claude CLI subprocess calls sequentially, taking ~22 minutes.

Proposed: Use concurrent.futures.ThreadPoolExecutor to run multiple calls in parallel:

from concurrent.futures import ThreadPoolExecutor, as_completed

with ThreadPoolExecutor(max_workers=10) as executor:
    futures = {
        executor.submit(_llm_call_claude, prompt, timeout): (article, db_id)
        for article, db_id, prompt in work_items
    }
    for future in as_completed(futures):
        article, db_id = futures[future]
        result = future.result()
        # process result...

Expected improvement: 10 workers × ~9 sec/call = Pass 1 completes in ~2–3 minutes instead of ~22. Full run time drops from 22 minutes to ~5–7 minutes.

Considerations:

• Claude API rate limits — Anthropic's API allows high concurrency on Pro/Max plans, but sustained 10-parallel subprocess calls may hit limits. Start with max_workers=5 and tune.
• The Rich progress bar needs updating to work with async completion order.

---

Formal test suite

Status: Planned

Proposed structure:

tests/
├── test_collector.py     # Feed parsing, full-text extraction, deduplication
├── test_analyzer.py      # Clustering logic, JSON parsing, LLM dispatch mocking
├── test_store.py         # SQLite operations, schema creation
├── test_reporter.py      # HTML rendering, markdown conversion
└── conftest.py           # Fixtures: sample articles, mock LLM responses

Key areas to cover:

• _parse_json_response() with malformed inputs
• Union-Find clustering with known article sets
• _article_is_fresh() boundary conditions
• _update_index() with various report path formats

---

Source expansion

Status: Ongoing as needed

Potential additions to improve spectrum coverage:

Source	Bias	Notes
New York Times	left	Requires RSS subscription
The Atlantic	center-left	Good long-form analysis
The Dispatch	center-right	Conservative anti-Trump perspective
The Intercept	far-left	Investigative/adversarial framing
Just the News	right	Alternative to Breitbart
Substack Politics aggregator	varies	Would require custom parser

---

Cross-run deduplication

Status: Planned

Currently, if an article is published at 11 PM, it may appear in both the Day 1 run (within the 24-hour window) and the Day 2 run (still within the window at 5 AM). URL deduplication is only within-run.

Proposed fix: Before saving articles in store.py, query for URLs already seen in recent runs:

SELECT url FROM articles
WHERE collected_at > datetime('now', '-48 hours')
AND url = ?

This would prevent re-analyzing the same article across consecutive days.

---

Email/push delivery

Status: Idea stage

Instead of only publishing to GitHub Pages, optionally email the daily brief to a subscriber list or push a notification with the report link.

Options:

• Email: Send the HTML report via smtplib or a service like SendGrid
• Push notification: Use a service like Pushover or ntfy.sh
• Slack/Discord webhook: Post a summary and link to a channel

This would be controlled by a new delivery: section in sources.yaml.

---

SonarQube quality gates

Status: In progress

The SonarLint extension currently flags ~27 issues across the codebase. These need to be resolved to meet the production deployment quality standard. Key categories:

• Security hotspots — any subprocess calls with unsanitized input
• Code smells — overly complex functions (particularly in reporter.py which has a very large HTML template)
• Duplications — repeated HTML generation patterns
• Test coverage — currently 0% (blocked by the formal test suite work above)