Agentic RAG Agent

An intelligent Retrieval-Augmented Generation (RAG) agent built with LangGraph that implements an adaptive workflow for question-answering.

Features

All advanced features are enabled by default for optimal performance:

• Intelligent Retrieval Decision: Decides when to retrieve vs. respond directly
• Evidence-Gap Critic: Intelligent gap analysis and targeted retrieval (enabled by default)
• Query Decomposition: Map-reduce pattern for complex multi-hop queries (enabled by default)
• HyDE Query Expansion: Hypothetical document embeddings for better retrieval (enabled by default)
• Hybrid Retrieval: BM25 + Dense vector search for comprehensive coverage (enabled by default)
• Cross-Encoder Reranking: Precision reranking of retrieved documents (enabled by default)
• Answer Generation & Synthesis: Produces answers from relevant context with proper citations
• Citation Tracking: Full citation metadata and validation
• Structure-Aware Chunking: Intelligent chunking that preserves document structure
• Evaluation Framework: Comprehensive evaluation with RAGAS metrics, workflow metrics, and multi-hop metrics
• Metadata Filtering: Advanced retrieval filtering with MMR for diversity

Installation

pip install -e .

For evaluation features, install with optional dependencies:

pip install -e ".[evaluation]"

Quick Start

Basic Usage

from rag_agent import create_rag_agent

# Create agent from URLs
agent = create_rag_agent(
    urls=["https://example.com/doc1", "https://example.com/doc2"],
    model_name="gpt-4o"
)

# Query the agent
response = agent.query("What is the return policy?")
print(response)

Minimal Configuration

All advanced features are enabled by default, so the basic usage above already includes:

• HyDE query expansion
• Hybrid retrieval (BM25 + Dense)
• Cross-encoder reranking
• Evidence-gap critic
• Query decomposition for complex queries

Disabling Features (for simpler/faster mode)

from rag_agent import create_rag_agent

# Create a simpler agent with features disabled for faster responses
agent = create_rag_agent(
    urls=["https://example.com/docs"],
    model_name="gpt-4o",
    enable_query_decomposition=False,     # Disable map-reduce
    enable_hyde=False,                    # Disable query expansion
    enable_hybrid_retrieval=False,        # Use dense retrieval only
    enable_reranking=False,               # Skip reranking
)

response = agent.query("What is the return policy?")

With Persistent Vector Store (LanceDB)

By default, the agent uses an in-memory vector store. For production use or evaluations requiring persistence, use LanceDB:

from rag_agent import create_rag_agent

# Create agent with persistent LanceDB vector store
agent = create_rag_agent(
    urls=["https://example.com/docs"],
    model_name="gpt-4o",
    vectorstore_type="lancedb",
    vectorstore_path="./my_vectorstore",
    table_name="documents",
    mode="overwrite",  # or "append" to add documents
)

response = agent.query("What is the return policy?")

Reusing Existing Vector Store (useful for evaluations):

# First run: create and index
agent1 = create_rag_agent(
    documents=documents,
    model_name="gpt-4o",
    vectorstore_type="lancedb",
    vectorstore_path="./eval_vectorstore",
    mode="overwrite",
)

# Subsequent runs: reuse existing store (much faster)
agent2 = create_rag_agent(
    documents=[],  # Empty - will reuse existing
    model_name="gpt-4o",
    vectorstore_type="lancedb",
    vectorstore_path="./eval_vectorstore",
    reuse_existing=True,  # Reuse without re-indexing
)

For Evaluations with Isolation (recommended):

import time

# Use separate stores per experiment for isolation
experiment_id = f"eval_{int(time.time())}"
vectorstore_path = f"./eval_stores/{experiment_id}"

agent = create_rag_agent(
    documents=pool_docs,
    model_name="gpt-4o",
    vectorstore_type="lancedb",
    vectorstore_path=vectorstore_path,
    mode="overwrite",  # Fresh store for each experiment
    reuse_existing=False,  # Don't reuse across experiments
)

Evaluation Framework

Running Evaluations

from rag_agent import create_rag_agent
from rag_agent.evaluation import EvaluationPipeline, MIRAGELoader

# Create agent
agent = create_rag_agent(urls=["..."], model_name="gpt-4o")

# Load dataset
dataset = MIRAGELoader()
pipeline = EvaluationPipeline(agent=agent)

# Run evaluation
results = pipeline.run_evaluation(
    dataset=dataset,
    experiment_name="baseline",
    max_examples=50
)

print(results["metrics"])

Batch Evaluations

from rag_agent.evaluation import BatchEvaluator

evaluator = BatchEvaluator()
configs = [
    {"experiment_name": "full_features"},  # All features enabled (default)
    {"experiment_name": "no_hyde", "enable_hyde": False},  # Disable HyDE only
    {"experiment_name": "minimal", "enable_hyde": False, "enable_hybrid_retrieval": False},
]

results = evaluator.run_batch_evaluation(
    dataset=MIRAGELoader(),
    configs=configs,
    max_examples=30
)

Available Datasets

• MIRAGE: Metric-intensive benchmark for RAG evaluation (7,560 examples, 37,800-doc retrieval pool)
• Synthetic: Minimal baseline dataset for quick testing

Evaluation Metrics

• RAGAS Metrics: Faithfulness, Answer Relevancy, Context Precision, Context Recall
• Workflow Metrics: Retrieval decision accuracy, document grading precision, query rewriting effectiveness
• Multi-hop Metrics: Reasoning score, document chain quality, information integration

Project Structure

src/rag_agent/
├── agent.py              # Main RAGAgent class
├── graph.py              # LangGraph workflow
├── nodes.py              # Graph node implementations
├── evaluation/           # Evaluation framework
│   ├── datasets/         # Dataset loaders
│   ├── metrics/          # Metric calculators
│   ├── pipelines/        # Evaluation orchestrators
│   └── tracking/         # Experiment tracking
└── integration/          # rag3 feature integrations

Testing

# Run all tests
pytest tests/

# Run only unit tests
pytest -m "unit and not slow"

Local Quality Checks

Before committing, run local quality checks:

# Run all checks (linting, formatting, unit tests)
make check

# Or run the script directly
bash scripts/run_checks.sh

This runs:

• Ruff linting
• Black format checking
• Import sorting (isort)
• Fast unit tests (excluding slow tests)

Knowledge Graph Requirements

If you plan to use knowledge graph features (enable_knowledge_graph=True), you must install the spaCy English model:

python -m spacy download en_core_web_sm

Without this model, knowledge graph features will fail with a clear error message. The model is required for proper entity extraction and relation extraction.

Documentation

• Architecture: Complete system architecture with all features
• Evaluation Guide: Comprehensive evaluation framework usage guide
• Experiment Tracking: Experiment tracking workflow and best practices
• Contributing: Guidelines for contributors
• AGENTS.md: Detailed guidelines for AI coding assistants

License

MIT