Architecture

Architecture

Signal is a sequential five-pass pipeline. Each pass builds on the structured output of the previous one. The architecture is deliberately simple: no web framework, no task queue, no daemon — just a Python script that runs once per day and produces a self-contained HTML file.

---

Repository structure

signal/
├── main.py                         # Entry point, venv bootstrap, index/archive generation
├── config/
│   └── sources.yaml                # All configuration: sources, LLM, collection settings
├── pipeline/
│   ├── __init__.py                 # Initialization file
│   ├── collector.py                # RSS ingestion, full-text extraction, deduplication
│   ├── analyzer.py                 # Passes 1–5: entity extraction through brief synthesis
│   ├── prompts.py                  # All LLM prompt templates
│   ├── reporter.py                 # HTML report generation
│   └── store.py                    # SQLite persistence layer
├── scripts/
│   ├── run_and_publish.sh          # Shell wrapper: runs pipeline + git push
│   └── com.flexrpl.signal.plist    # launchd configuration file
├── reports/
│   └── brief_YYYYMMDD_HHMM.html    # Generated reports (tracked in git → GitHub Pages)
├── docs/                           # Operational documentation (this file)
├── logs/
│   └── cron.log                    # Runtime log (gitignored except .gitkeep)
├── images/
│   └── signal_banner.png           # Banner image used on landing page
├── index.html                      # Landing page (regenerated on each run)
├── archive.html                    # Full report archive (regenerated on each run)
├── signal.db                       # SQLite database (gitignored)
└── requirements.txt                # Python dependencies (gitignored)

---

Data flow

sources.yaml
    │
    ▼
┌─────────────────────────────────────────┐
│  collector.py — collect_feeds()         │
│                                         │
│  • feedparser: parse all 18 RSS feeds   │
│  • Filter by age (< 24 hours)           │
│  • Deduplicate by URL                   │
│  • httpx + BeautifulSoup: full text     │
│  • Returns: List[ArticleDict]           │
└──────────────────┬──────────────────────┘
                   │ ~140–160 articles/day
                   ▼
┌─────────────────────────────────────────┐
│  store.py — init_db(), save_articles()  │
│                                         │
│  • Creates signal.db tables if missing  │
│  • Inserts all articles for this run    │
│  • Returns: List[article_db_id]         │
└──────────────────┬──────────────────────┘
                   │
                   ▼
┌─────────────────────────────────────────┐
│  analyzer.py — Pass 1: extract_entities │
│                                         │
│  • One LLM call per article             │
│  • Extracts: topic, key_claim,          │
│    entities{people,orgs,bills,locs},    │
│    sentiment, framing                   │
│  • Updates DB with extracted fields     │
│  • Returns: articles (augmented)        │
└──────────────────┬──────────────────────┘
                   │
                   ▼
┌─────────────────────────────────────────┐
│  analyzer.py — Pass 2: cluster_articles │
│                                         │
│  • Union-Find algorithm                 │
│  • Groups articles by:                  │
│    - Title similarity ≥ 0.70 (rapidfuzz)│
│    - Shared entity count ≥ 2            │
│  • Same source articles not clustered   │
│  • Returns: List[ClusterDict]           │
│    (multi-source clusters + singletons) │
└──────────────────┬──────────────────────┘
                   │
                   ▼
┌─────────────────────────────────────────┐
│  analyzer.py — Pass 3: analyze_clusters │
│                                         │
│  • One LLM call per multi-source cluster│
│  • Extracts: headline, consensus_facts, │
│    contested_points, left/right/center  │
│    framing, omissions, assessment,      │
│    significance                         │
│  • Saves cluster analyses to DB         │
└──────────────────┬──────────────────────┘
                   │
                   ▼
┌─────────────────────────────────────────┐
│  analyzer.py — Pass 4: correlate_stories│
│                                         │
│  • Single LLM call over all clusters    │
│  • Builds entity co-occurrence network  │
│  • Identifies left-only/right-only      │
│    coverage blindspots                  │
│  • Reads previous run's watch list      │
│    for delta detection                  │
│  • Returns: correlation analysis dict   │
└──────────────────┬──────────────────────┘
                   │
                   ▼
┌─────────────────────────────────────────┐
│  analyzer.py — Pass 5: synthesize_brief │
│                                         │
│  • Single LLM call                      │
│  • Inputs: all cluster analyses +       │
│    correlation analysis                 │
│  • Outputs: markdown brief with 6       │
│    structured sections                  │
│  • Saves brief text to DB               │
└──────────────────┬──────────────────────┘
                   │
                   ▼
┌─────────────────────────────────────────┐
│  reporter.py — generate_report()        │
│                                         │
│  • Renders markdown → HTML sections     │
│  • Builds story cards with framing data │
│  • Generates connections, patterns,     │
│    anomalies, blindspot panels          │
│  • Writes: reports/brief_YYYYMMDD.html  │
└──────────────────┬──────────────────────┘
                   │
                   ▼
┌─────────────────────────────────────────┐
│  main.py — _update_index()              │
│                                         │
│  • Regenerates index.html (landing page)│
│  • Regenerates archive.html (full list) │
└──────────────────┬──────────────────────┘
                   │
                   ▼
┌─────────────────────────────────────────┐
│  run_and_publish.sh                     │
│                                         │
│  • git add reports/ index.html          │
│              archive.html               │
│  • git commit + git push origin main    │
│  → GitHub Actions deploys to Pages      │
└─────────────────────────────────────────┘

---

Key design decisions

Why Union-Find for clustering?

Pass 2 needs to group N articles where any two articles satisfying the similarity criteria belong in the same cluster — including transitively (if AB and BC, then A, B, C are one cluster). Union-Find solves this in near-linear time without building an explicit graph.

Why LLM calls are sequential in Pass 1

Each article generates an independent LLM call. Currently these are sequential subprocess calls to the Claude CLI. This is the primary bottleneck (~9 seconds/article × 146 articles = ~22 minutes). A future optimization is to use a thread pool (concurrent.futures.ThreadPoolExecutor) to parallelize calls. See Future Roadmap.

Why SQLite and not a flat file

SQLite enables delta detection across runs. Pass 4 queries the previous run's watch list to track what has materialized or escalated. Future Phase 2 weekly summaries will query the briefs table directly to synthesize across multiple days.

Why the report is a self-contained HTML file

Reports are static files committed to git and served by GitHub Pages. There is no server, no database exposed publicly, no authentication. Each report is fully standalone — it can be saved, shared, or viewed offline.

Why main.py bootstraps a venv

When invoked as python3 main.py, the script creates .venv/, installs dependencies, then re-executes itself as venv/bin/python main.py --no-venv. This means users never need to manually activate a virtual environment.

---

SQLite schema

runs (
    id INTEGER PRIMARY KEY,
    started_at TEXT,
    finished_at TEXT,
    article_count INTEGER,
    cluster_count INTEGER,
    status TEXT  -- 'running' | 'complete'
)

articles (
    id INTEGER PRIMARY KEY,
    run_id INTEGER REFERENCES runs(id),
    title TEXT, url TEXT, source_name TEXT, bias TEXT,
    published_at TEXT, collected_at TEXT,
    text_snippet TEXT, full_text TEXT,
    entities_json TEXT,   -- JSON: {people, organizations, legislation, locations}
    key_claim TEXT,
    topic TEXT,           -- legislation|executive_action|election|...
    sentiment TEXT,       -- neutral|positive|negative|alarming|celebratory
    framing TEXT
)

clusters (
    id INTEGER PRIMARY KEY,
    run_id INTEGER REFERENCES runs(id),
    story_title TEXT, article_ids TEXT,  -- JSON array
    source_count INTEGER, bias_spread TEXT  -- JSON: {bias: count}
)

cluster_analyses (
    id INTEGER PRIMARY KEY,
    cluster_id INTEGER REFERENCES clusters(id),
    run_id INTEGER REFERENCES runs(id),
    analysis_json TEXT,   -- full Pass 3 output
    created_at TEXT
)

correlation_analyses (
    id INTEGER PRIMARY KEY,
    run_id INTEGER REFERENCES runs(id),
    analysis_json TEXT,   -- full Pass 4 output
    created_at TEXT
)

briefs (
    id INTEGER PRIMARY KEY,
    run_id INTEGER REFERENCES runs(id),
    brief_text TEXT,      -- Pass 5 markdown output
    created_at TEXT
)

---

LLM call budget per run

Pass	Calls	Model input	Notes
Pass 1	~146	Article title + snippet (≤1500 chars)	One per article
Pass 3	~5–10	Formatted cluster articles	One per multi-source cluster
Pass 4	1	All cluster summaries + entity network	Single cross-story call
Pass 5	1	Correlation analysis + story analyses	Single synthesis call
Total	~153–158

Pass 5 uses a 3× timeout multiplier (min(timeout * 3, 360)) since the synthesis prompt is the largest and most complex call.