Dave-GPT — Local RAG Study Assistant

A fully local Retrieval-Augmented Generation (RAG) chatbot that lets you ask questions about your course materials. No API keys. No cloud. Everything runs on your machine.

---

Architecture

┌─────────────────────────────────────────────────────────────────┐
│                        app.py  (Streamlit UI)                   │
│                                                                 │
│  ┌─────────────┐   ┌──────────────────┐   ┌─────────────────┐  │
│  │  Section 1  │   │    Section 2     │   │   Section 3     │  │
│  │  Upload     │   │  Build Knowledge │   │   Ask Dave      │  │
│  │  Documents  │   │     Base         │   │  (Chat UI)      │  │
│  └──────┬──────┘   └────────┬─────────┘   └────────┬────────┘  │
└─────────│──────────────────│──────────────────────│────────────┘
          │                  │                       │
          ▼                  ▼                       ▼
┌─────────────────┐  ┌──────────────────┐  ┌─────────────────────┐
│   extractors/   │  │   rag/chunker.py  │  │    retrieve_context │
│                 │  │                  │  │  (FAISS search)      │
│  pdf_extractor  │  │  create_chunks() │  │                     │
│  audio_extractor│  │  (LangChain text │  │  embed query        │
│  video_extractor│  │   splitter)      │  │  → top-k chunks     │
│  notebook_extrc │  └────────┬─────────┘  └────────┬────────────┘
│  text_extractor │           │                      │
│  web_extractor  │           ▼                      ▼
│  epub_extractor │  ┌──────────────────┐  ┌─────────────────────┐
└─────────────────┘  │  rag/embedder.py │  │    generate_answer  │
                     │                  │  │                     │
                     │  all-MiniLM-L6-v2│  │  GPU:  Mistral-7B   │
                     │  (384-dim vecs)  │  │  CPU:  TinyLlama-1B │
                     │  FAISS IndexFlat │  │  (HuggingFace       │
                     │  → .pkl saved    │  │   Transformers)     │
                     └──────────────────┘  └─────────────────────┘
                                │
                     ┌──────────▼──────────┐
                     │  dave_gpt_          │
                     │  knowledge_base.pkl │
                     │  (chunks + index)   │
                     └─────────────────────┘

How it works end-to-end

Step	What happens
1. Upload	Files (PDF, audio, video, notebooks, text, HTML, EPUB) are saved to documents/
2. Extract	Each file is routed to its extractor — audio/video uses faster-whisper for transcription
3. Chunk	Text is split into overlapping segments (default 512 chars, 50 overlap)
4. Embed	all-MiniLM-L6-v2 converts every chunk to a 384-dim vector
5. Index	FAISS IndexFlatL2 stores all vectors in dave_gpt_knowledge_base.pkl
6. Query	Your question is embedded → top-k nearest chunks are retrieved
7. Generate	Retrieved chunks + question are fed to the LLM; answer is streamed back

Model selection (automatic)

Hardware	LLM loaded	VRAM needed
CUDA GPU	mistralai/Mistral-7B-Instruct-v0.2 (4-bit NF4)	~5 GB
CPU only	TinyLlama/TinyLlama-1.1B-Chat-v1.0	~1 GB RAM

---

Project Structure

part-time gig/
├── app.py                        # Streamlit UI — main entry point
├── build_knowledge_base.py       # CLI alternative to build the index
├── dave_gpt_knowledge_base.pkl   # Built FAISS index + chunks (generated)
│
├── extractors/                   # File-type extractors
│   ├── __init__.py               # Registry: extension → extractor function
│   ├── pdf_extractor.py          # PDF via pymupdf4llm
│   ├── audio_extractor.py        # MP3/WAV/M4A via faster-whisper
│   ├── video_extractor.py        # MP4/MKV/WEBM (extracts audio track)
│   ├── notebook_extractor.py     # .ipynb via nbformat
│   ├── text_extractor.py         # TXT/MD plain text
│   ├── web_extractor.py          # HTML/URLs via trafilatura
│   └── epub_extractor.py         # EPUB via ebooklib
│
├── rag/                          # RAG pipeline
│   ├── chunker.py                # Text splitting (LangChain)
│   └── embedder.py               # Embedding + FAISS index build/load
│
├── evaluation/                   # Optional RAGAS evaluation
│   └── ragas_eval.py
│
├── documents/                    # Drop your course materials here
│   ├── Chapter 1/
│   └── Chapter 2/
│
├── requirements.txt              # All dependencies
└── README.md

---

Setup

1. Clone the repo

git clone https://github.com/dehiska/LocalLLM.git
cd LocalLLM

2. Create a virtual environment (recommended)

python -m venv venv

# Windows
venv\Scripts\activate

# Mac/Linux
source venv/bin/activate

3. Install Python dependencies

pip install -r requirements.txt

GPU users: If you have an NVIDIA GPU and want Mistral-7B, install the CUDA-enabled version of PyTorch first:

pip install torch --index-url https://download.pytorch.org/whl/cu121

Then re-run pip install -r requirements.txt.

4. Install ffmpeg (required for audio/video extraction)

Windows:

winget install ffmpeg

Mac:

brew install ffmpeg

Linux:

sudo apt install ffmpeg

Verify it works:

ffmpeg -version

---

Running the App

streamlit run app.py

Opens at http://localhost:8501 in your browser.

---

Usage

Option A — Streamlit UI (recommended)

1. Upload documents — drag and drop PDFs, audio, video, notebooks, or text files (max 5 at a time), or paste a URL
2. Build knowledge base — click "Build Knowledge Base"; progress bars show extraction and embedding
3. Ask Dave — type a question in the chat box; Dave answers using only your uploaded materials

Option B — CLI (build index only)

# Process the default documents/ folder
python build_knowledge_base.py

# Custom input folder
python build_knowledge_base.py --input my_docs/

# Custom output file
python build_knowledge_base.py --output brain.pkl

# Smaller chunks (better for short Q&A)
python build_knowledge_base.py --chunk-size 256 --chunk-overlap 25

---

Sidebar Settings

Setting	Default	Description
Context chunks (k)	5	How many chunks are retrieved per question
Chunk size	512 chars	Size of each text chunk when building the index
Chunk overlap	50 chars	Overlap between consecutive chunks
Show source citations	On	Display which files each answer came from
Show retrieved context	Off	Show the raw chunks fed to the LLM
Show RAGAS evaluation	Off	Score answer quality (requires OpenAI key or Ollama)

---

Supported File Types

Format	Extensions	Extractor used
PDF	.pdf	pymupdf4llm
Audio	.mp3 .wav .m4a	faster-whisper (transcription)
Video	.mp4 .mkv .webm	faster-whisper (audio track → transcription)
Jupyter Notebook	.ipynb	nbformat
Text / Markdown	.txt .md	plain text read
Web / HTML	.html .htm or URL	trafilatura
EPUB	.epub	ebooklib

---

Requirements

• Python 3.10+
• ffmpeg (system install, see Setup above)
• ~1 GB RAM minimum (CPU/TinyLlama mode)
• ~6 GB VRAM recommended (GPU/Mistral-7B mode)