-
Notifications
You must be signed in to change notification settings - Fork 0
Architecture
Garot Conklin edited this page Jun 10, 2026
·
3 revisions
Signal has three pipelines sharing the same codebase, database, and publishing infrastructure:
- Daily pipeline (Passes 1–5): Runs every morning at 4:00 AM. Fetches articles, extracts entities, clusters stories, runs cross-spectrum correlation analysis, and synthesizes a daily intelligence brief.
- Weekly pipeline (Pass 6): Runs every Monday at 6:00 AM. Reads the past week's daily briefs from the database and synthesizes a weekly intelligence summary focused on change over time.
- Monthly pipeline (Pass 7): Runs on the 1st of each month at 7:00 AM. Reads daily briefs and weekly summaries for the previous calendar month and synthesizes a monthly intelligence summary. Partial months are auto-detected and labeled.
Both pipelines are deliberately simple: no web framework, no task queue, no daemon — just Python scripts that run on a schedule and produce self-contained HTML files.
signal/
├── main.py # Entry point, venv bootstrap, index/archive generation
├── config/
│ └── sources.yaml # All configuration: sources, LLM, collection settings
├── pipeline/
│ ├── __init__.py # Initialization file
│ ├── collector.py # RSS ingestion, full-text extraction, deduplication
│ ├── analyzer.py # Passes 1–5: entity extraction through brief synthesis
│ ├── weekly.py # Pass 6: weekly synthesis (reads DB, no fetching)
│ ├── monthly.py # Pass 7: monthly synthesis (reads DB, no fetching)
│ ├── prompts.py # All LLM prompt templates (DAILY_*, WEEKLY_BRIEF, MONTHLY_BRIEF)
│ ├── reporter.py # HTML report generation (daily + weekly + monthly templates)
│ └── store.py # SQLite persistence layer
├── scripts/
│ ├── run_and_publish.sh # Daily shell wrapper: pipeline + git push
│ ├── run_weekly_and_publish.sh # Weekly shell wrapper: synthesis + git push
│ ├── run_monthly_and_publish.sh # Monthly shell wrapper: synthesis + git push
│ ├── com.flexrpl.signal.plist # launchd config: daily 4:00 AM
│ ├── com.flexrpl.signal.weekly.plist # launchd config: weekly Monday 6:00 AM
│ └── com.flexrpl.signal.monthly.plist # launchd config: monthly 1st 7:00 AM
├── reports/
│ ├── brief_YYYYMMDD_HHMM.html # Daily reports (tracked in git → GitHub Pages)
│ ├── weekly_YYYYWNN_HHMM.html # Weekly reports (tracked in git → GitHub Pages)
│ └── monthly_YYYYMM[_partial]_HHMM.html # Monthly reports (tracked in git → GitHub Pages)
├── docs/ # Operational documentation (this file)
├── logs/
│ └── cron.log # Runtime log (gitignored except .gitkeep)
├── images/
│ └── signal_banner.png # Banner image used on landing page
├── index.html # Landing page (regenerated on each run)
├── archive.html # Full report archive (regenerated on each run)
├── signal.db # SQLite database (gitignored)
└── requirements.txt # Python dependencies (gitignored)sources.yaml
│
▼
┌─────────────────────────────────────────┐
│ collector.py — collect_feeds() │
│ │
│ • feedparser: parse all 18 RSS feeds │
│ • Filter by age (< 24 hours) │
│ • Deduplicate by URL │
│ • httpx + BeautifulSoup: full text │
│ • Returns: List[ArticleDict] │
└──────────────────┬──────────────────────┘
│ ~140–160 articles/day
▼
┌─────────────────────────────────────────┐
│ store.py — init_db(), save_articles() │
│ │
│ • Creates signal.db tables if missing │
│ • Inserts all articles for this run │
│ • Returns: List[article_db_id] │
└──────────────────┬──────────────────────┘
│
▼
┌─────────────────────────────────────────┐
│ analyzer.py — Pass 1: extract_entities │
│ │
│ • One LLM call per article │
│ • Extracts: topic, key_claim, │
│ entities{people,orgs,bills,locs}, │
│ sentiment, framing │
│ • Updates DB with extracted fields │
│ • Returns: articles (augmented) │
└──────────────────┬──────────────────────┘
│
▼
┌─────────────────────────────────────────┐
│ analyzer.py — Pass 2: cluster_articles │
│ │
│ • Union-Find algorithm │
│ • Groups articles by: │
│ - Title similarity ≥ 0.70 (rapidfuzz)│
│ - Shared entity count ≥ 2 │
│ • Same source articles not clustered │
│ • Returns: List[ClusterDict] │
│ (multi-source clusters + singletons) │
└──────────────────┬──────────────────────┘
│
▼
┌─────────────────────────────────────────┐
│ analyzer.py — Pass 3: analyze_clusters │
│ │
│ • One LLM call per multi-source cluster│
│ • Extracts: headline, consensus_facts, │
│ contested_points, left/right/center │
│ framing, omissions, assessment, │
│ significance │
│ • Saves cluster analyses to DB │
└──────────────────┬──────────────────────┘
│
▼
┌─────────────────────────────────────────┐
│ analyzer.py — Pass 4: correlate_stories│
│ │
│ • Single LLM call over all clusters │
│ • Builds entity co-occurrence network │
│ • Identifies left-only/right-only │
│ coverage blindspots │
│ • Reads previous run's watch list │
│ for delta detection │
│ • Returns: correlation analysis dict │
└──────────────────┬──────────────────────┘
│
▼
┌─────────────────────────────────────────┐
│ analyzer.py — Pass 5: synthesize_brief │
│ │
│ • Single LLM call │
│ • Inputs: all cluster analyses + │
│ correlation analysis │
│ • Outputs: markdown brief with 6 │
│ structured sections │
│ • Saves brief text to DB │
└──────────────────┬──────────────────────┘
│
▼
┌─────────────────────────────────────────┐
│ reporter.py — generate_report() │
│ │
│ • Renders markdown → HTML sections │
│ • Builds story cards with framing data │
│ • Generates connections, patterns, │
│ anomalies, blindspot panels │
│ • Writes: reports/brief_YYYYMMDD.html │
└──────────────────┬──────────────────────┘
│
▼
┌─────────────────────────────────────────┐
│ main.py — _update_index() │
│ │
│ • Regenerates index.html (landing page)│
│ • Regenerates archive.html (full list) │
└──────────────────┬──────────────────────┘
│
▼
┌─────────────────────────────────────────┐
│ run_and_publish.sh │
│ │
│ • git add reports/ index.html │
│ archive.html │
│ • git commit + git push origin main │
│ → GitHub Actions deploys to Pages │
└─────────────────────────────────────────┘signal.db (past 7 days of complete runs)
│
▼
┌─────────────────────────────────────────┐
│ store.py — get_runs_for_weekly() │
│ │
│ • Queries runs WHERE status='complete' │
│ AND has a brief AND within 7 days │
│ • Deduplicates: one run per calendar │
│ date (latest run wins) │
│ • Returns: List[run metadata] │
└──────────────────┬──────────────────────┘
│ 5–7 qualifying runs
▼
┌─────────────────────────────────────────┐
│ store.py — get_weekly_source_data() │
│ │
│ • For each run: fetches brief_text, │
│ narrative_patterns, anomalies, │
│ watch_list, delta_from_previous │
│ • Returns: enriched list sorted by │
│ date (oldest → newest) │
└──────────────────┬──────────────────────┘
│
▼
┌─────────────────────────────────────────┐
│ weekly.py — Pass 6: run_weekly() │
│ │
│ • Extracts SITUATION OVERVIEW and │
│ WATCH LIST from each daily brief │
│ • Builds watch list evolution timeline │
│ • Formats single WEEKLY_BRIEF prompt │
│ • One LLM call (Claude, large context) │
│ • Saves to weekly_briefs table in DB │
└──────────────────┬──────────────────────┘
│
▼
┌─────────────────────────────────────────┐
│ reporter.py — generate_weekly_report() │
│ │
│ • Gold/amber theme (not blue) │
│ • Day chips: one per run included │
│ • 7 structured sections │
│ • Writes: reports/weekly_YYYYWNN.html │
└──────────────────┬──────────────────────┘
│
▼
┌─────────────────────────────────────────┐
│ main.py — _update_index() │
│ │
│ • Adds "Weekly Summary" card to │
│ index.html landing page │
│ • Adds "Monthly Summary" card when │
│ a monthly report exists │
│ • Adds sections to archive.html │
└──────────────────┬──────────────────────┘
│
▼
┌─────────────────────────────────────────┐
│ run_weekly_and_publish.sh │
│ │
│ • git add reports/weekly_*.html │
│ index.html archive.html │
│ • git commit + git push origin main │
│ → GitHub Actions deploys to Pages │
└─────────────────────────────────────────┘signal.db (daily briefs + weekly summaries for calendar month)
│
▼
┌─────────────────────────────────────────┐
│ store.py — get_runs_for_month() │
│ get_weekly_briefs_for_month() │
│ │
│ • One complete run per calendar date │
│ • Weekly briefs overlapping the month │
│ • Auto-detects partial coverage │
└──────────────────┬──────────────────────┘
│
▼
┌─────────────────────────────────────────┐
│ monthly.py — Pass 7: run_monthly() │
│ │
│ • Condenses weekly + daily briefs │
│ • Builds watch list evolution timeline │
│ • Formats single MONTHLY_BRIEF prompt │
│ • One LLM call (Claude, large context) │
│ • Saves to monthly_briefs table in DB │
└──────────────────┬──────────────────────┘
│
▼
┌─────────────────────────────────────────┐
│ reporter.py — generate_monthly_report()│
│ │
│ • Purple theme (#a371f7) │
│ • Partial badge when coverage incomplete│
│ • Writes: reports/monthly_YYYYMM.html │
└──────────────────┬──────────────────────┘
│
▼
┌─────────────────────────────────────────┐
│ run_monthly_and_publish.sh │
│ │
│ • git add reports/monthly_*.html │
│ index.html archive.html │
│ • git commit + git push origin main │
│ → GitHub Actions deploys to Pages │
└─────────────────────────────────────────┘Pass 2 needs to group N articles where any two articles satisfying the similarity criteria belong in the same cluster — including transitively (if AB and BC, then A, B, C are one cluster). Union-Find solves this in near-linear time without building an explicit graph.
Each article generates an independent LLM call. Currently these are sequential subprocess calls to the Claude CLI. This is the primary bottleneck (~9 seconds/article × 146 articles = ~22 minutes). A future optimization is to use a thread pool (concurrent.futures.ThreadPoolExecutor) to parallelize calls. See Future Roadmap.
SQLite enables delta detection across runs and serves as the data source for weekly and monthly synthesis. Pass 4 queries the previous run's watch list to track what has materialized or escalated. Pass 6 and Pass 7 query the briefs, correlation_analyses, and weekly_briefs tables directly — the rich structured data stored there is more useful than re-parsing the HTML reports.
Reports are static files committed to git and served by GitHub Pages. There is no server, no database exposed publicly, no authentication. Each report is fully standalone — it can be saved, shared, or viewed offline.
When invoked as python3 main.py, the script creates .venv/, installs dependencies, then re-executes itself as venv/bin/python main.py --no-venv. This means users never need to manually activate a virtual environment.
runs (
id INTEGER PRIMARY KEY,
started_at TEXT,
finished_at TEXT,
article_count INTEGER,
cluster_count INTEGER,
status TEXT -- 'running' | 'complete'
)
articles (
id INTEGER PRIMARY KEY,
run_id INTEGER REFERENCES runs(id),
title TEXT, url TEXT, source_name TEXT, bias TEXT,
published_at TEXT, collected_at TEXT,
text_snippet TEXT, full_text TEXT,
entities_json TEXT, -- JSON: {people, organizations, legislation, locations}
key_claim TEXT,
topic TEXT, -- legislation|executive_action|election|...
sentiment TEXT, -- neutral|positive|negative|alarming|celebratory
framing TEXT
)
clusters (
id INTEGER PRIMARY KEY,
run_id INTEGER REFERENCES runs(id),
story_title TEXT, article_ids TEXT, -- JSON array
source_count INTEGER, bias_spread TEXT -- JSON: {bias: count}
)
cluster_analyses (
id INTEGER PRIMARY KEY,
cluster_id INTEGER REFERENCES clusters(id),
run_id INTEGER REFERENCES runs(id),
analysis_json TEXT, -- full Pass 3 output
created_at TEXT
)
correlation_analyses (
id INTEGER PRIMARY KEY,
run_id INTEGER REFERENCES runs(id),
analysis_json TEXT, -- full Pass 4 output
created_at TEXT
)
briefs (
id INTEGER PRIMARY KEY,
run_id INTEGER REFERENCES runs(id),
brief_text TEXT, -- Pass 5 markdown output
created_at TEXT
)
weekly_briefs (
id INTEGER PRIMARY KEY,
week_start TEXT, -- ISO date of first day covered (e.g. '2026-05-11')
week_end TEXT, -- ISO date of last day covered (e.g. '2026-05-15')
run_ids TEXT, -- JSON array of daily run IDs included
brief_text TEXT, -- Pass 6 markdown output
created_at TEXT
)
monthly_briefs (
id INTEGER PRIMARY KEY,
month_label TEXT, -- e.g. 'May 2026'
month_start TEXT, -- first day of coverage (may differ from calendar start if partial)
month_end TEXT, -- last day of coverage
run_ids TEXT, -- JSON array of daily run IDs included
weekly_ids TEXT, -- JSON array of weekly brief IDs included
brief_text TEXT, -- Pass 7 markdown output
partial INTEGER, -- 1 if coverage does not span full calendar month
created_at TEXT
)| Pass | Calls | Model input | Notes |
|---|---|---|---|
| Pass 1 | ~146 | Article title + snippet (≤1500 chars) | One per article |
| Pass 3 | ~5–10 | Formatted cluster articles | One per multi-source cluster |
| Pass 4 | 1 | All cluster summaries + entity network | Single cross-story call |
| Pass 5 | 1 | Correlation analysis + story analyses | Single synthesis call |
| Total | ~153–158 |
Pass 5 uses a 3× timeout multiplier (min(timeout * 3, 360)) since the synthesis prompt is the largest and most complex call.
| Pass | Calls | Model input | Notes |
|---|---|---|---|
| Pass 6 | 1 | 5–7 days of condensed briefs + watch list evolution | Single large-context call |
| Total | 1 | No fetching, no per-article work |
Pass 6 uses a 2× timeout multiplier (360 seconds) to accommodate the larger context. The weekly synthesis deliberately reads only condensed sections from each daily brief (SITUATION OVERVIEW, WATCH LIST, ANALYST NOTE, and correlation patterns) rather than full brief texts, keeping the context window manageable.
| Pass | Calls | Model input | Notes |
|---|---|---|---|
| Pass 7 | 1 | Weekly summaries + condensed daily briefs + watch evolution | Single large-context call |
| Total | 1 | No fetching, partial months OK |
Pass 7 reuses _llm_call() from weekly.py and shares _extract_section() and _build_watch_list_evolution() helpers.
Signal · Repository · fleXRPL · Daily political intelligence — powered by local AI