RAG Browser

A hybrid web browser combining DuckDuckGo search with a RAG (Retrieval-Augmented Generation) ChatBot to intelligently query web page content.

Features

• Web Search: Google-like search interface powered by DuckDuckGo
• Page Selection: Intuitive selection of search results (up to 5 pages)
• Intelligent Scraping: Automatic extraction of relevant content from web pages
• RAG ChatBot: Chat with your indexed documents using LangChain/LangGraph
• Multi-LLM Support: Support for Ollama (local), OpenAI, and Anthropic
• Modern UI: Built with PySide6 for a sleek user experience
• Streaming Responses: Real-time streaming of LLM responses in chat

Prerequisites

• Python 3.10 or higher
• uv for dependency management
• (Optional) Ollama installed for local LLM usage
• (Optional) OpenAI or Anthropic API keys

Installation

1. Clone the Repository

git clone <repository-url>
cd rag-browser

2. Install uv (if not already installed)

Linux/macOS:

curl -LsSf https://astral.sh/uv/install.sh | sh

Windows:

powershell -c "irm https://astral.sh/uv/install.ps1 | iex"

3. Create Virtual Environment and Install Dependencies

uv venv
source .venv/bin/activate  # On Windows: .venv\Scripts\activate
uv pip install -e ".[dev]"

4. Configure API Keys (Optional)

If you plan to use OpenAI or Anthropic:

cp .env.example .env
# Edit .env and add your API keys

Add to .env:

OPENAI_API_KEY=your_openai_key_here
ANTHROPIC_API_KEY=your_anthropic_key_here

5. Install Pre-commit Hooks

pre-commit install

Usage

Launch the Application

python -m src.main

Configuration

On first launch, access the settings (⚙️ icon in the top right) to configure your LLM:

1. Choose your provider (Ollama, OpenAI, or Anthropic)
2. Configure provider-specific settings
3. Test the connection
4. Save

Workflow

1. Search: Enter your query and launch a DuckDuckGo search
2. Select: Check the relevant pages (maximum 5)
3. Index: Click "Continue to Chat" to scrape and index the pages
4. Chat: Ask questions about the indexed content

The chat interface supports:

• Real-time streaming responses
• Source citations with relevance scores
• Multi-turn conversations with history management
• Markdown-style formatting

Architecture

rag-browser/
├── src/
│   ├── ui/              # PySide6 user interface
│   │   ├── main_window.py       # Main application window
│   │   ├── home_view.py         # Home screen
│   │   ├── search_view.py       # Search interface
│   │   ├── chat_view.py         # Chat interface with streaming
│   │   ├── dialogs/             # Configuration dialogs
│   │   └── widgets/             # Reusable UI components
│   ├── search/          # Search and web scraping
│   │   ├── duckduckgo_client.py # DuckDuckGo API client
│   │   └── web_scraper.py       # Web content extraction
│   ├── services/        # Core services
│   │   ├── embeddings/          # Text embedding services
│   │   ├── llm/                 # LLM client implementations
│   │   └── vector_store/        # Vector storage (FAISS)
│   ├── indexer/         # Document processing and indexing
│   │   ├── document_processor.py # Text chunking
│   │   └── indexer.py           # Document indexing
│   ├── rag/             # RAG pipeline with LangGraph
│   │   ├── engine.py            # RAG orchestration
│   │   ├── chains/              # LangGraph chains
│   │   └── nodes/               # Retrieval, reranking, grading, generation
│   ├── utils/           # Utilities
│   │   ├── logger.py            # Logging configuration
│   │   └── exceptions.py        # Custom exceptions
│   ├── config.py        # Configuration management
│   └── main.py          # Application entry point
├── config/              # Configuration files
│   └── default_config.yaml      # Default settings
└── tests/               # Unit tests

Technology Stack

Frontend

• PySide6: Modern Qt6 bindings for Python
• Custom UI: ChatGPT-inspired interface with streaming support

Backend

• LangChain/LangGraph: RAG pipeline orchestration
• FAISS: Efficient vector similarity search
• Sentence Transformers: Local text embeddings (all-MiniLM-L6-v2)
• Trafilatura: Intelligent web content extraction

LLM Providers

• Ollama: Local LLM inference (llama3.2, mistral, etc.)
• OpenAI: GPT models (gpt-4o-mini, gpt-4o)
• Anthropic: Claude models (claude-sonnet-4)

RAG Pipeline

The system uses a LangGraph-based pipeline with four stages:

1. Retrieval: Fetch top-K similar chunks using FAISS
2. Reranking: Cross-encoder reranking for better relevance
3. Grading: Filter out irrelevant chunks
4. Generation: Streaming LLM response with source citations

Testing

# Run all tests
pytest

# With coverage report
pytest --cov=src --cov-report=html

# Specific test file
pytest tests/test_config.py

# View coverage report
open htmlcov/index.html  # macOS/Linux
start htmlcov/index.html  # Windows

Development

Code Formatting

# Format code
black src/ tests/

# Lint code
ruff check src/ tests/ --fix

Pre-commit Hooks

Pre-commit hooks run automatically before each commit:

• Code formatting with Black
• Linting with Ruff
• YAML, JSON, TOML validation
• Private key detection

Run manually:

pre-commit run --all-files

Configuration

Configuration is managed at two levels:

1. Default: config/default_config.yaml (version controlled)
2. User: ~/.rag-browser/user_config.yaml (generated by the app)

User settings override default settings. Key configuration options:

Search Settings

• max_search_results: Maximum search results to display (default: 20)
• max_selected_pages: Maximum pages to index (default: 5)
• timeout: Search timeout in seconds (default: 10)

LLM Settings

• provider: ollama, openai, or anthropic
• temperature: Response creativity (0.0-1.0, default: 0.7)
• max_tokens: Maximum response length (default: 2000)

RAG Settings

• chunk_size: Text chunk size in characters (default: 1000)
• chunk_overlap: Overlap between chunks (default: 200)
• top_k_retrieval: Number of chunks to retrieve (default: 10)
• top_k_rerank: Number of chunks after reranking (default: 5)
• relevance_threshold: Minimum relevance score (default: 0.2)

Embedding Settings

• provider: sentence-transformers (local)
• model: all-MiniLM-L6-v2 (384 dimensions)
• batch_size: Processing batch size (default: 32)

Project Status

Current version: 0.1.0

Completed Features

• ✅ Complete project structure
• ✅ YAML configuration with default/user merge
• ✅ Configuration module with 96% test coverage
• ✅ Logger and custom exceptions
• ✅ Comprehensive unit tests (11/11 passing)
• ✅ Pre-commit hooks
• ✅ Full documentation
• ✅ DuckDuckGo search interface
• ✅ Web scraping with Trafilatura
• ✅ Document processing and chunking
• ✅ FAISS vector store
• ✅ Local embeddings with Sentence Transformers
• ✅ LLM clients (Ollama, OpenAI, Anthropic)
• ✅ LangGraph RAG pipeline with streaming
• ✅ Chat UI with real-time streaming
• ✅ Source citations with relevance scores
• ✅ Multi-turn conversation management

Contributing

Contributions are welcome! Please follow these steps:

1. Fork the project
2. Create a feature branch (git checkout -b feature/amazing-feature)
3. Make your changes with pre-commit hooks active
4. Add tests for new functionality
5. Ensure all tests pass (pytest)
6. Commit your changes (git commit -m 'Add amazing feature')
7. Push to the branch (git push origin feature/amazing-feature)
8. Open a Pull Request

Coding Standards

• Follow PEP 8 (enforced by Ruff)
• Use Black for code formatting (line length: 100)
• Add type hints to all functions
• Write docstrings for all public functions and classes
• Maintain test coverage above 90%

Troubleshooting

Ollama Connection Issues

If you encounter connection errors with Ollama:

1. Ensure Ollama is running: ollama serve
2. Check the base URL in settings (default: http://localhost:11434)
3. Verify the model is available: ollama list

API Key Issues

• Store API keys in environment variables (.env file)
• Never commit API keys to version control
• User config file automatically filters sensitive data

Scraping Errors

Some websites may block scraping. This is normal. Select alternative sources if a page fails to load.

Performance

• First embedding generation may be slow (model download)
• Subsequent runs use cached model
• Consider using GPU acceleration for Ollama (CUDA/ROCm)

Acknowledgments

• LangChain for the RAG framework
• LangGraph for workflow orchestration
• DuckDuckGo for the search API
• Trafilatura for content extraction
• FAISS for vector search
• Sentence Transformers for embeddings
• PySide6 for the UI framework

License

MIT License - see the LICENSE file for details.

Contact

For questions or suggestions, please open an issue on GitHub.

---

Made with ❤️ using Python, LangChain, and PySide6