KnowBaseRAG – AI-Powered Web Intelligence & Research Assistant

An intelligent web research assistant that leverages AI to help users gather, process, and understand information from multiple web sources efficiently.

🎯 Motivation

The goal of this project was to understand how modern AI systems:

• Combine LLMs with external knowledge
• Scale beyond prompt-only approaches
• Reduce hallucinations using vector search

🌟 Features

• Web Content Ingestion: Automatically extract and process content from multiple URLs
• Semantic Search: Find relevant information using vector embeddings
• Question Answering: Get precise answers to your research questions
• Document Processing: Automatically chunk and index documents for efficient retrieval
• Modern Web Interface: Intuitive Streamlit-based UI for easy interaction

🏗️ Architecture

graph TD
    A[User] -->|Interacts with| B[Streamlit Frontend]
    B -->|Sends Requests| C[FastAPI Backend]
    C -->|Processes with| D[LangChain]
    D -->|Stores/Retrieves| E[ChromaDB]
    D -->|Uses| F[Embedding Models]
    C -->|Fetches Data| G[Web Sources]
    
    style A fill:#c0392b,stroke:#333,color:white,stroke-width:2px,font-weight:bold
    style B fill:#2980b9,stroke:#333,color:white,stroke-width:2px,font-weight:bold
    style C fill:#27ae60,stroke:#333,color:white,stroke-width:2px,font-weight:bold
    style D fill:#8e44ad,stroke:#333,color:white,stroke-width:2px,font-weight:bold
    style E fill:#16a085,stroke:#333,color:white,stroke-width:2px,font-weight:bold
    style F fill:#f39c12,stroke:#333,color:white,stroke-width:2px,font-weight:bold
    style G fill:#d35400,stroke:#333,color:white,stroke-width:2px,font-weight:bold

Loading

The system follows a modern microservices architecture with the following components:

1. Frontend: Streamlit-based web interface
2. Backend API: FastAPI server handling business logic
3. Vector Database: ChromaDB for efficient similarity search
4. AI Processing: LangChain for document processing and question answering
5. Web Scraping: BeautifulSoup4 for content extraction

📁 Project Structure

KnowBaseRAG/
├── backend/                   # FastAPI backend service
│   ├── app/
│   │   ├── main.py            # FastAPI entry point and routes
│   │   ├── models/            # Pydantic data models
│   │   ├── routes/            # API endpoints (modularized)
│   │   ├── services/          # Business logic
│   │   │   ├── llm_service.py         # LLM integration
│   │   │   ├── nlp_service.py         # Text cleaning and chunking
│   │   │   ├── scraping_service.py    # Web content extraction
│   │   │   └── vector_store_service.py# ChromaDB integration
│   │   └── utils/             # Helper functions
│   ├── chroma_db/             # Local vector database storage
│   └── requirements.txt       # Backend dependencies
├── frontend/                  # Streamlit web interface
│   └── streamlit_app.py       # Main UI application
├── data/                      # Sample data and evaluation sets
├── notebooks/                 # Jupyter notebooks for experimentation
├── LICENSE                    # MIT License
└── README.md                  # Project documentation

🛠️ Tech Stack

Backend

• Framework: FastAPI
• AI/ML: LangChain, Sentence Transformers
• Vector Database: ChromaDB
• Web Scraping: BeautifulSoup4, Requests
• NLP: NLTK

Frontend

• Framework: Streamlit
• UI Components: Custom Streamlit components

🚀 Getting Started

Prerequisites

• Python 3.8+
• pip (Python package manager)

Installation

1. Clone the repository:

git clone https://github.com/yourusername/ai-web-intelligence-assistant.git
cd ai-web-intelligence-assistant

2. Set up the backend:

cd backend
python -m venv .venv
.venv\Scripts\activate  # On Windows
pip install -r requirements.txt

3. Set up the frontend:

cd ../frontend
pip install -r requirements.txt

Running the Application

1. Start the backend server:

cd backend
uvicorn app.main:app --reload

2. In a new terminal, start the frontend:

cd frontend
streamlit run streamlit_app.py

3. Open your browser and navigate to http://localhost:8501

📚 Usage

1. Ingest Knowledge:

   • Enter website URLs in the text area
   • Click "Ingest URLs" to process the content

2. Ask Questions:

   • Type your research question
   • Click "Ask" to get AI-powered answers

🧩 Retrieval-Augmented Generation (RAG)

This project uses a Retrieval-Augmented Generation (RAG) pipeline:

• Documents are embedded and stored in a vector database
• Relevant context is retrieved at query time
• The LLM generates answers strictly based on retrieved context

This prevents hallucinations and ensures source-grounded answers.

⚠️ Limitations

• Answers are limited to ingested sources only
• Large models may require higher system memory
• No authentication or multi-user support (yet)

🤝 Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

🙏 Acknowledgments

• LangChain for the amazing AI orchestration framework
• ChromaDB for the lightweight vector database
• Streamlit for the intuitive web interface