📄 DOCWISE Pipeline

A Retrieval-Augmented Generation (RAG) application that lets you ingest documents and ask natural language questions about their content. Built with durable workflow orchestration, local vector search, and LLM-powered answers.

---

🏗️ Architecture

┌──────────────┐       ┌───────────────┐       ┌──────────────┐
│  Streamlit   │──────▶│   Inngest     │──────▶│   FastAPI    │
│  Frontend    │ event │  Dev Server   │ invoke│   Workers    │
└──────────────┘       └───────────────┘       └──────┬───────┘
                                                      │
                              ┌────────────────────────┼────────────────────────┐
                              │                        │                        │
                       ┌──────▼──────┐          ┌──────▼──────┐         ┌──────▼──────┐
                       │  PDF Reader │          │  Embedding  │         │    Groq     │
                       │  (LlamaIdx) │          │ (MiniLM-L6) │         │  LLM (API)  │
                       └─────────────┘          └──────┬──────┘         └─────────────┘
                                                       │
                                                ┌──────▼──────┐
                                                │   Qdrant    │
                                                │  Vector DB  │
                                                └─────────────┘

How It Works

1. Ingest — Upload a document → text is extracted, chunked (800 chars, 200 overlap), embedded with all-MiniLM-L6-v2, and stored in Qdrant.
2. Query — Ask a question → question is embedded, top-k similar chunks are retrieved from Qdrant, and a Groq-hosted Llama 3.1 model generates an answer grounded in those chunks.
3. Orchestration — Inngest handles durable execution with automatic retries, throttling, and rate limiting.

---

📁 Project Structure

RAG_PROJECT/
├── main.py              # FastAPI app + Inngest function definitions (ingest & query)
├── data_loader.py       # File reading, text chunking, and embedding logic
├── vector_db.py         # Qdrant client wrapper (upsert & similarity search)
├── custom_types.py      # Pydantic models for type-safe data passing between steps
├── streamlit_app.py     # Streamlit UI for uploading PDFs and asking questions
├── pyproject.toml       # Project metadata and dependencies
├── .env                 # Environment variables (API keys — not committed)
└── uploads/             # Uploaded Files stored here

---

🚀 Getting Started

Prerequisites

• Python 3.14+
• uv (package manager)
• Qdrant running locally on port 6333
• Inngest Dev Server
• A Groq API key

1. Clone & Install

git clone https://github.com/Alwazf99/RAG_PROJECT.git
cd RAG_PROJECT
uv sync

2. Set Up Environment Variables

Create a .env file in the project root:

GROQ_API_KEY=<your-groq-api-key>

3. Start Qdrant

docker run -p 6333:6333 qdrant/qdrant

4. Start Inngest Dev Server

npx inngest-cli@latest dev

5. Start the FastAPI Worker

uv run uvicorn main:app --reload

6. Start the Streamlit UI

uv run streamlit run streamlit_app.py

---

🖥️ Usage

Ingest a File

1. Open the Streamlit app (default: http://localhost:8501)
2. Upload a PDF file
3. The ingestion pipeline runs automatically (chunking → embedding → storage)

Ask a Question

1. Type your question in the text input
2. Choose how many chunks to retrieve (top-k)
3. Click Ask — the answer appears with source references

---

⚙️ Key Design Decisions

Decision	Rationale
Inngest for orchestration	Durable steps with automatic retries; throttle/rate-limit prevent overloading
Local embeddings (MiniLM)	Fast, free, no API calls needed for vectorization
Groq LLM	Extremely fast inference for Llama 3.1 via OpenAI-compatible API
Qdrant	Lightweight vector DB that runs locally with no setup overhead
Deterministic UUIDs	Re-ingesting the same PDF overwrites vectors instead of duplicating

---

🛡️ Rate Limiting & Throttling

• Throttle: Max 2 ingestion runs per minute globally
• Rate Limit: Same PDF (source_id) can only be re-ingested once every 4 hours

---

📦 Dependencies

Package	Purpose
fastapi + uvicorn	HTTP server exposing Inngest webhook
inngest	Durable workflow orchestration
llama-index-core + llama-index-readers-file	PDF parsing and text splitting
sentence-transformers	Local text embedding
qdrant-client	Vector database client
openai	OpenAI-compatible client (used for Groq)
streamlit	Web UI
python-dotenv	Environment variable loading

---

📝 License

MIT

---

🤝 Contributing

1. Fork the repo
2. Create a feature branch (git checkout -b feature/amazing-feature)
3. Commit your changes (git commit -m 'Add amazing feature')
4. Push to the branch (git push origin feature/amazing-feature)
5. Open a Pull Request