A next-generation subjectivity-aware RAG pipeline built for investment-style questions like "Should I buy HDFC Bank stock right now?" or "Is Tesla a good long-term investment?" — with multi-layer verification so the answer is never just vibes.
- 🔍 SERP-powered web search — fetches top results via SerpAPI
- 🌐 Robust scraping — handles HTML, PDF, JS-rendered pages, and Cloudflare-protected sites
- 📄 BM25 ranking — chunks and ranks evidence for relevance
- 🤖 Worker LLM — generates analysis with claim-level source attribution
- ✅ Checker LLM — validates logical consistency and flags fabricated facts
- 🛡️ Deterministic claim verifier — rule-based (non-LLM) check that each claim exists verbatim in sources
- 📜 Full query logging — every run is auditable as a
.jsonlfile
Query → Classification → Web Search → Scraping → Chunking + BM25 → Worker LLM → Checker LLM → Claim Verifier → Final Answer
- Query Classification — routes to finance, crypto, news, or general domain
- Web Search — SerpAPI fetches up to 8 URLs, filtered by whitelisted domains
- Scraping — multi-strategy scraper with caching, PDF support, and Yahoo Finance auto-extraction
- Chunking + BM25 — deduped chunks ranked by relevance; top 5 selected as evidence
- Worker LLM — produces final answer + claim list with evidence snippets and source URLs
- Checker LLM — ensures no fabricated facts, checks source-claim alignment
- Deterministic Verifier — non-LLM rule-based check; computes claim_precision and hallucination_rate
- Final Output — worker answer shown only if checker = VALID and claim_precision ≥ 0.9, else safe fallback with mismatch report
git clone cd submission/ pip install -r requirements.txt python scripts/run_subjective_rag.py "should I buy HDFC Bank stock now?"
Final answer is shown only when ALL conditions are met:
- Checker LLM returns VALID
- claim_precision ≥ 0.9
Otherwise, a safe fallback is returned with:
- Worker answer
- Verifier notes
- Claim-evidence mismatch report
- claim_precision
- hallucination_rate
- valid_claims / total_claims
- Real-time price feeds
- Sentiment scoring
- Multi-agent analysis
- Vector DB for historical queries
- Dashboard visualization
- Playbook templates per industry
Built by Vinayak