Web RAG

A web-based Retrieval Augmented Generation (RAG) system that performs real-time web searches and generates vector embeddings for improved information retrieval.

Features

• Real-time web scraping and content extraction
• Vector embeddings storage in PostgreSQL
• Similarity search capabilities
• Integrated DuckDuckGo search
• Automatic text chunking and embedding generation

Setup

1. Set up your environment variables:

export POSTGRES_USER=your_user
export POSTGRES_PASSWORD=your_password
export POSTGRES_DB=your_database
export POSTGRES_HOST=localhost
export POSTGRES_PORT=5432
export PG_SET=true

2. Install dependencies:

pip install langchain-postgres langchain-core duckduckgo-search psycopg

3. Initialize the PostgreSQL database with the required schema. Also ensure that the pgvector extension is installed.

Usage

from web.vec import VectorStorage
from your_embeddings_model import Embeddings

# Initialize with your preferred embeddings model
embeddings = Embeddings()
vector_store = VectorStorage(embeddings)

# Search with automatic web retrieval
results = vector_store.search("your query", refresh=True)

# Search existing embeddings only with automatic web retrival if needed (default)
results = vector_store.search("your query")

# This can then be used as context for a RAG model
# Look at example.py for a good example of how to use this with Google's Gemini Model

Architecture

The system consists of three main components:

1. Web Scraper: Fetches and extracts content from web pages
2. Vector Storage: Manages embeddings in PostgreSQL
3. Search Interface: Provides similarity search capabilities

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

License

MIT License