Configuration

Configuration

All pipeline configuration lives in a single file: config/sources.yaml.

---

Full annotated reference

# ── LLM provider ────────────────────────────────────────────────────────────
llm:
  provider:
    claude # Default provider. Options: "ollama" | "claude"
    # Override at runtime with SIGNAL_LLM_PROVIDER env var.
    # See: LLM Providers wiki page.
  ollama:
    model: "qwen2.5:14b" # Ollama model to use. Must be installed via `ollama pull`.
    base_url: "http://localhost:11434" # Ollama server address.
    timeout: 120 # Seconds to wait per LLM call before timing out.
  claude:
    timeout: 180 # Claude CLI calls include network round-trip; give extra headroom.

# ── Article collection ───────────────────────────────────────────────────────
collection:
  max_articles_per_source: 20 # Cap articles taken from each RSS feed per run.
  article_age_hours: 24 # Ignore articles older than this. 24 = today's news only.
  fetch_full_text:
    true # If true, fetches full article body via HTTP.
    # If false (--no-fetch), uses RSS snippet only (~800 chars).
  fetch_timeout: 10 # Seconds to wait per HTTP article fetch.

# ── Clustering thresholds (Pass 2) ──────────────────────────────────────────
analysis:
  min_cluster_size:
    2 # Articles needed to form a cluster (not used to filter,
    # but singletons are tracked separately).
  entity_similarity_threshold: 0.75 # Reserved for future use.
  title_similarity_threshold:
    0.70 # Two articles cluster if their titles score ≥ this
    # on rapidfuzz token_sort_ratio (0.0–1.0).

# ── News sources ─────────────────────────────────────────────────────────────
sources:
  - name: "AP Politics"
    url: "https://feeds.apnews.com/rss/politics"
    bias: "center"
    type: rss
  # ... additional sources ...

---

LLM provider settings

Switching providers

Three layers of precedence (highest first):

1. SIGNAL_LLM_PROVIDER environment variable
2. llm.provider in sources.yaml
3. Hard-coded fallback: "ollama"

# One-off override
SIGNAL_LLM_PROVIDER=ollama python3 main.py

# Permanent change
# Edit config/sources.yaml: provider: ollama

Claude timeout

Claude CLI calls include process startup + HTTPS round-trip + inference. 180 seconds is generous; most calls complete in 5–30 seconds. Increase if you see subprocess.TimeoutExpired errors in Pass 1.

Ollama timeout

120 seconds per local inference call. Increase for larger models (e.g. qwen2.5:32b).

---

Collection settings

article_age_hours

Controls how far back the pipeline looks for articles. At 24 hours, only today's news is processed. Increasing to 48 would pick up more articles but risk reprocessing yesterday's stories (which are already in the DB from the previous run — the URL deduplication in store.py prevents duplicate storage, but the LLM still processes them).

Note: URL deduplication is at the collection stage only — seen_urls is a set built within a single run. There is no cross-run deduplication currently, which means an article published at 11 PM could appear in both the Day 1 and Day 2 runs.

fetch_full_text

When true, each article URL is fetched via httpx and the main body text is extracted using BeautifulSoup. The extractor tries common article containers (article, [role="main"], .article-body, etc.) before falling back to the longest paragraph block. Text is capped at 4000 characters.

When false (or via --no-fetch), only the RSS feed's <summary>/<description> is used (~800 chars). This is faster but reduces analysis quality.

---

Clustering thresholds

title_similarity_threshold

Uses rapidfuzz.fuzz.token_sort_ratio — a fuzzy string match that is order-insensitive. A score of 0.70 means two titles must share 70% of their word-tokens to cluster. This is well-calibrated for political news (e.g. "Senate Passes Budget Bill" and "Budget Bill Passes Senate" score ~0.95).

Lower this value to cluster more aggressively (more multi-source clusters, fewer singletons). Raise it to require tighter title matches.

Entity overlap

Two articles also cluster if they share ≥ 2 named entities. This catches stories covered under different headlines but discussing the same actors (e.g. an article about "Trump signs executive order" and "EO on trade signed by president" would share "Trump" + "executive order").

---

Adding a news source

Add a new entry to the sources: list:

- name: "New York Times Politics"
  url: "https://rss.nytimes.com/services/xml/rss/nyt/Politics.xml"
  bias: "left"
  type: rss

Valid bias values

Value	Examples
center	AP, Reuters, C-SPAN, The Hill, Axios
center-left	NPR, PBS, Politico
left	Washington Post, The Guardian
far-left	Mother Jones
center-right	Wall Street Journal, RealClearPolitics
right	Fox News, Washington Examiner, National Review
far-right	Breitbart
libertarian	Reason

The bias value is used in Pass 3 (framing analysis prompt) and Pass 4 (blindspot detection). It is also displayed as a tag on each article in the report's source index.

Testing a new source

python3 main.py --collect-only

This fetches feeds without running any LLM passes. Check the output for errors from the new source URL.

---

Current sources (18 total)

Source	Bias
AP Politics	center
Reuters Politics	center
C-SPAN	center
NPR Politics	center-left
PBS NewsHour Politics	center-left
Politico	center-left
The Hill	center
Axios Politics	center
Washington Post Politics	left
The Guardian US	left
Mother Jones	far-left
Wall Street Journal Politics	center-right
Fox News Politics	right
Washington Examiner	right
National Review	right
Breitbart	far-right
RealClearPolitics	center-right
Reason	libertarian