Skip to content

Architecture

Anjana Tiha edited this page Mar 28, 2026 · 2 revisions

Architecture

Design Principle

RAG Agent uses 4 classes and stateless modules. Classes own state and the operations that depend on that state. Modules own constants and stateless functions.

This constraint was chosen deliberately: it forces clear ownership boundaries, prevents hidden state leakage, and makes every component independently testable.


Class Responsibilities

Class Owns Key methods
DocumentLoader All ingestion: file scanning, URL fetching, chunker dispatch chunk_all_documents(), chunk_url(), chunk_url_recursive()
VectorStore ChromaDB, BM25, hybrid retrieval, reranking, response generation, conversation history build_or_load(), run_pipeline(), add_chunks()
Agent ReAct loop, all 6 tools run()
Benchmarker 7-metric evaluation, CSV export, run comparison run()

Module Responsibilities

Module Owns
config.py All constants — no functions
logger.py Stateless interaction logging
chunkers.py Stateless text-format chunkers (txt, md, csv, html)
binary_chunkers.py Stateless binary-format chunkers (pdf, docx, xlsx, xls, pptx)
url_crawl.py Stateless URL crawl and DuckDuckGo search
url_utils.py URL type detection, link extraction, topic filtering
reranker.py 7 type-aware LLM rerank prompt variants
metrics.py 7 stateless scoring functions
benchmark_report.py Stateless terminal report formatting
tool_benchmarks.py Agent tool benchmark suite
query_utils.py Stateless query helpers: classify_query, smart_top_n, build_instruction_prompt
ragas_eval.py RAGAS LLM-as-judge evaluation (optional — pip install -e ".[eval]")

Pipeline Flow

Documents / URLs
      │
      ▼
DocumentLoader
  scan_all_files()          ← recursive walk, type detection
  _dispatch_chunker()       ← routes each file to the right chunker
  chunk_url()               ← URL type detection + fetch + chunk
      │
      ▼ chunks (list of dicts)
VectorStore.build_or_load()
  _embed()                  ← BGE embeddings via Ollama
  ChromaDB.add()            ← persistent vector store
  BM25Okapi()               ← keyword index
      │
      ▼ index ready
VectorStore.prepare_pipeline(query)   ← classify → retrieve → rerank → return stream
  classify_query()          ← factual / comparison / summarise / general (query_utils)
  smart_top_n()             ← retrieval budget per query type (query_utils)
  _expand_query()           ← 2 LLM rewrites + original = 3 queries
  _hybrid_retrieve()        ← BM25 + dense fusion (alpha=0.5)
  _check_confidence()       ← skip LLM if best score < threshold
  _rerank()                 ← type-aware LLM scoring via reranker.py
  build_instruction_prompt() ← length-hinted anti-hallucination prompt (query_utils)
  ollama.chat(stream=True)  ← token stream returned to caller

VectorStore.finalize_pipeline(query, qtype, reranked, raw_response)
  _filter_hallucination()   ← truncate pivot phrases after no-info admissions
  conversation history      ← updated with filtered response
  logger.log_interaction()  ← interaction saved to rag_logs.json
      │
      ▼ answer + metadata

Key Design Decisions

Hybrid Search over Pure Dense Retrieval

BM25 + dense vector fusion achieves higher recall than either approach alone, particularly for structured documents (spreadsheets, CSVs) where exact keyword matching is more reliable than semantic similarity.

Type-Aware Reranking

A generic reranker prompt scores all chunk types the same way. Spreadsheet rows (Name=Alice | Role=Engineer) score poorly because they are not fluent prose. Seven type-specific prompts frame each format appropriately — the xlsx/csv prompt asks "does this spreadsheet row contain relevant information?" rather than treating it as a failed paragraph.

Confidence Gate

If no retrieved chunk clears the similarity threshold, the LLM is not called at all. This eliminates a class of hallucinations that occur when the model is forced to answer from training data because the context is empty.

Hallucination Filter

The model sometimes correctly admits it cannot find information ("there is no information about this") but then pivots to a hallucinated answer ("however, I can tell you that..."). The filter detects this pattern and truncates the response at the pivot phrase, preserving the admission and discarding the fabricated continuation.


Deployment Variants

Variant Embedding LLM Vector Store
Local Ollama (BGE) Ollama (LLaMA 3.2) ChromaDB PersistentClient
HF Space sentence-transformers (BGE) HF Inference API (LLaMA) ChromaDB EphemeralClient

The HF Space variant uses a cross-encoder for reranking instead of LLM prompts — cross-encoders score all candidate pairs in one batch call, which is significantly faster on serverless infrastructure than making one API call per candidate chunk.

Clone this wiki locally