Development

Development

---

Project structure

signal/
├── main.py                        # Entry point + venv bootstrap + index/archive generation
├── config/sources.yaml            # All runtime configuration
├── pipeline/
│   ├── __init__.py                # Initialization file
│   ├── collector.py               # Feed ingestion, full-text extraction
│   ├── analyzer.py                # Passes 1–5 + LLM dispatch
│   ├── prompts.py                 # All LLM prompt templates
│   ├── reporter.py                # HTML report rendering
│   └── store.py                   # SQLite persistence
├── scripts/
│   ├── run_and_publish.sh         # Automation wrapper
│   └── com.flexrpl.signal.plist   # launchd configuration file
├── docs/                          # Operational docs
├── reports/                       # Generated HTML reports (committed)
├── logs/                          # Runtime logs (gitignored)
└── images/                        # Static assets

---

Development environment

git clone https://github.com/fleXRPL/signal.git
cd signal

# Dependencies install automatically on first run, OR manually:
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

---

Dependencies

Package	Version	Purpose
feedparser	≥6.0.11	RSS/Atom feed parsing
httpx	≥0.27.0	Async-capable HTTP client for full-text fetching
beautifulsoup4	≥4.12.3	HTML parsing for article body extraction
lxml	≥5.2.0	Fast HTML/XML parser backend for BeautifulSoup
python-dateutil	≥2.9.0	Robust date parsing from feed entries
rapidfuzz	≥3.9.0	Fast fuzzy string matching for title clustering
rich	≥13.7.0	Terminal progress bars and colored output
pyyaml	≥6.0.1	YAML config file parsing
ollama	≥0.3.3	Ollama Python client (used when provider=ollama)
jinja2	≥3.1.4	Available but not currently used in templates

---

Code style and quality

The project targets SonarQube Cloud quality gates. All code must pass:

• No critical or major SonarQube issues
• No security hotspots
• Line length ≤ 120 characters
• Docstrings on all public functions
• Type hints on all function signatures

Running SonarQube locally

SonarQube scanning runs automatically on push via the SonarLint VSCode extension. The SonarQube tab in Cursor/VS Code shows live issues.

Key patterns to follow

Error handling in LLM calls:

try:
    raw = _llm_call(prompt, model, base_url, timeout, provider)
    parsed = _parse_json_response(raw)
except Exception as exc:  # noqa: BLE001
    console.print(f"  [red]LLM error:[/red] {exc}")
    parsed = None

All LLM errors are caught, logged, and handled gracefully. The pipeline never aborts due to a single failed LLM call.

JSON parsing:

Always use _parse_json_response() rather than json.loads() directly on LLM output. It handles markdown fences, leading text, and embedded JSON.

---

Adding a new analysis pass

If you want to add a sixth pass (e.g. sentiment trend tracking):

1. Add a prompt template in pipeline/prompts.py:

   SENTIMENT_TREND = """\
   You are an analyst tracking sentiment trends...
   Return ONLY a valid JSON object: {{ ... }}"""

2. Add a function in pipeline/analyzer.py:

   def track_sentiment_trends(
       articles: List[Dict[str, Any]],
       run_id: int,
       model: str,
       base_url: str,
       timeout: int,
       provider: str = "claude",
   ) -> Dict[str, Any]:
       """Pass 6: Track sentiment trends across sources."""
       ...

3. Wire it into run_pipeline() in pipeline/analyzer.py:

   # Pass 6
   trends = track_sentiment_trends(articles, run_id, model, base_url, timeout, provider)
   return brief, clusters, correlation, trends

4. Update run_signal() in main.py to accept and use the new return value.

5. Add a DB table in store.py if persistence is needed.

6. Add a rendering function in reporter.py to display the output in the HTML report.

---

Modifying the HTML report

The report template is HTML_TEMPLATE in pipeline/reporter.py — a large multi-line string with Python .format() placeholders. It uses inline CSS (GitHub-dark theme) with CSS custom properties.

Key rendering functions:

Function	Output
_render_brief_sections()	Parses markdown → HTML for the 6 brief sections
_render_story_cards()	Story cluster cards with framing analysis
_render_connections()	Hidden connections list
_render_list_items()	Generic list renderer for patterns/anomalies/watch items
_render_side_coverage()	Left-only/right-only blindspot columns
_render_source_index()	Collapsible source index

---

Modifying the landing page or archive

index.html and archive.html are generated inline by _update_index() in main.py. There is no separate template file — the HTML is an f-string in the function body. This was a deliberate choice to keep the pipeline self-contained in a single repository with no build step.

---

Testing

There is no formal test suite currently. The recommended testing workflow:

# 1. Verify feed connectivity (no LLM, instant)
python3 main.py --collect-only

# 2. Fast end-to-end test with Claude (uses RSS snippets)
python3 main.py --no-fetch

# 3. Full production run
python3 main.py

# 4. Test the full automation script
bash scripts/run_and_publish.sh
tail -f logs/cron.log

A formal pytest suite is planned. See Future Roadmap.

---

Environment variables

Variable	Values	Default	Effect
SIGNAL_LLM_PROVIDER	claude | ollama	claude (via sources.yaml)	Selects the LLM backend

---

Git workflow

The main branch is the production branch. Every push to main triggers GitHub Actions to redeploy GitHub Pages.

Reports and HTML pages are committed directly to main by run_and_publish.sh. There is no separate gh-pages branch — the site is served from the repo root on main.

Never force-push to main — it would rewrite the report history visible on the archive page.