Ask anything about any library — in plain English. DocuMentor crawls documentation from any URL, indexes it using a hybrid RAG pipeline, and answers developer questions with the right function, when to use it, when to avoid it, and working code.
- Overview
- Key Features
- Architecture
- Tech Stack
- Project Structure
- Prerequisites
- Installation
- Environment Variables
- Running the Application
- API Reference
- How It Works
- Troubleshooting
- Contributing
- License
DocuMentor solves a real problem for developers learning new libraries: official documentation is written for experienced users, organized by API reference rather than use case, and requires you to already know what you're looking for.
Paste any documentation URL. DocuMentor crawls the entire site, parses and indexes every function into a structured knowledge base, and lets you ask questions in plain English. The system classifies your intent (function search, error fix, or concept explanation), retrieves the most relevant documentation chunks using hybrid BM25 + semantic search, reranks them with a cross-encoder, and generates a grounded answer using a local LLM — with source citations and zero hallucination of function names.
Target users: Computer science students, bootcamp learners, self-taught developers, and anyone onboarding to an unfamiliar library.
- Bring-your-own documentation — works with any library that has a public documentation URL, not just popular ones baked into training data
- Three intelligent response modes:
- Function recommendation — describes a goal, gets ranked function candidates with trade-offs and code snippets
- Anti-pattern detection — every answer includes a structured "avoid when" section
- Error fix mode — paste a traceback, get root cause + fix + working code
- Hybrid retrieval pipeline — BM25 keyword search + dense embeddings merged via Reciprocal Rank Fusion (RRF), followed by a cross-encoder reranker
- Hallucination-proof by design — the LLM is strictly grounded to retrieved documentation chunks only
- Version caching — already-indexed libraries are served from cache for 7 days; no redundant re-crawling
- Fully local LLM — runs on your machine via Ollama with no API costs
- Real-time indexing progress — frontend polls and displays crawl → parse → index stages with a progress bar
The system follows a 7-stage pipeline from URL to answer: URL Input → Scrape (httpx + BeautifulSoup / Cloudflare Browser Rendering) → Clean + Structure (strip noise, extract function boundaries) → Chunk (by function boundary, not token count) → Embed (sentence-transformers all-MiniLM-L6-v2 → ChromaDB) → Hybrid Retrieval (BM25 + semantic search → RRF merge) → Rerank (cross-encoder/ms-marco-MiniLM-L-6-v2) → LLM Answer (Ollama qwen2.5-coder:7b, strict JSON output)
Each scraped function is stored as a typed JSON object:
{
"type": "function",
"name": "pandas.read_csv",
"library": "pandas",
"version": "2.1.0",
"params": ["filepath_or_buffer", "sep", "dtype"],
"use_when": ["loading tabular data from disk"],
"avoid_when": ["file > available RAM — use chunksize or Dask"],
"example": "df = pd.read_csv('data.csv', dtype={'id': str})",
"related": ["pandas.read_parquet", "pandas.read_json"],
"notes": "Use engine='pyarrow' for 3–5× faster parse",
"source_url": "https://pandas.pydata.org/docs/..."
}| Layer | Technology |
|---|---|
| Backend framework | FastAPI + Uvicorn |
| Scraping (static) | httpx + BeautifulSoup4 |
| Scraping (JS-heavy) | Cloudflare Browser Rendering REST API |
| Embeddings | sentence-transformers all-MiniLM-L6-v2 |
| Vector store | ChromaDB (persistent, local) |
| Keyword search | rank-bm25 |
| Reranker | sentence-transformers cross-encoder/ms-marco-MiniLM-L-6-v2 |
| Local LLM | Ollama (qwen2.5-coder:7b) |
| Frontend | React 18 + TypeScript + Vite |
| Styling | Tailwind CSS + shadcn/ui |
| Data validation | Pydantic v2 |
DOCUMENTOR/
├── BACKEND/
│ ├── app.py # FastAPI application, route definitions
│ ├── pipeline.py # RAG orchestration (intent → retrieve → rerank → LLM)
│ ├── retriever.py # Hybrid BM25 + semantic search with RRF fusion
│ ├── reranker.py # Cross-encoder reranking
│ ├── llm.py # Ollama integration and JSON answer generation
│ ├── intent_classifier.py # Rule-based + LLM intent classification
│ ├── prompts.py # Prompt templates per intent type
│ ├── parser.py # HTML → plain text → structured function extraction
│ ├── vector_store.py # ChromaDB + BM25 storage, caching, embeddings
│ ├── utils.py # Crawlers (static + Cloudflare)
│ ├── model.py # Pydantic request/response models
│ ├── schemas/
│ │ └── function.py # FunctionSchema definition
│ ├── data/ # Persisted ChromaDB + per-library manifests (gitignored locally)
│ ├── requirements.txt
│ ├── .env # Environment variables (not committed)
│ ├── test_crawl.py # Cloudflare crawl integration test
│ └── test_parser.py # Parser validation script
│
└── FRONTEND/
├── src/
│ ├── pages/
│ │ ├── Index.tsx # Home: doc URL input, quick-start pills, feature previews
│ │ ├── ChatPage.tsx # Chat UI after indexing
│ │ ├── QueryPage.tsx # Explore / demo route
│ │ └── NotFound.tsx
│ ├── components/ # Shared UI (shadcn/ui)
│ ├── hooks/
│ └── lib/
├── .env # VITE_API_BASE_URL
├── package.json
├── tailwind.config.ts
└── vite.config.ts
Ensure the following are installed before proceeding:
- Python 3.11+
- Node.js 18+
- Ollama — install from ollama.ai
Pull the LLM model before starting:
ollama pull qwen2.5-coder:7bgit clone https://github.com/your-username/documentor.git
cd documentorcd BACKEND
# Create and activate a virtual environment
python3 -m venv .venv
source .venv/bin/activate # Windows: .venv\Scripts\activate
# Install dependencies
pip install -r requirements.txtcd FRONTEND
npm install# Ollama (local LLM)
OLLAMA_MODEL=qwen2.5-coder:7b
OLLAMA_URL=http://localhost:11434/api/generate
# Cloudflare Browser Rendering (optional — only needed for JS-heavy doc sites)
CF_ACCOUNT_ID=your_cloudflare_account_id
CF_API_TOKEN=your_cloudflare_api_tokenOptional — data directory override (set in the shell or add to .env if your tooling loads it into the environment):
| Variable | Purpose |
|---|---|
DOCUMENTOR_DATA_DIR |
Absolute path to a writable folder for data/ contents. Default: BACKEND/data. Use this in Docker, read-only installs, or when the default path is not writable. ChromaDB stores SQLite under <DOCUMENTOR_DATA_DIR>/chroma. |
The Cloudflare credentials are optional. The system automatically falls back to the static scraper (
httpx+BeautifulSoup) for most documentation sites. Cloudflare is only used for JS-rendered sites like React, Vue, or Next.js docs.
VITE_API_BASE_URL=http://localhost:8000Quick setup: Copy BACKEND/.env.example to BACKEND/.env and FRONTEND/.env.example to FRONTEND/.env, then edit values. The example files list every variable the app recognizes (including optional DOCUMENTOR_DATA_DIR and Cloudflare fields).
ollama servecd BACKEND
source .venv/bin/activate
uvicorn app:app --reload --port 8000cd FRONTEND
npm run devThe application will be available at http://localhost:8080 (see FRONTEND/vite.config.ts).
| Path | Description |
|---|---|
/ |
Home — paste a documentation URL or pick a quick-start library (Pandas, FastAPI, Three.js, Scikit-learn, SQLAlchemy). Optional ?prefill=<encoded-doc-url> fills the input (used by Switch library in chat). What you can ask cards open a modal; click outside or press Escape to close. |
/chat?url=...&ready=1 |
Chat — ask questions against indexed docs. The sidebar lists recent sessions (from localStorage keys starting with documentor_history); click a row to restore that library’s thread. Switch library (or the active doc pill) sends you home with the current URL prefilled. |
/explore |
Explore — additional demo / query page in the app shell. |
Initiates a crawl, parse, and index job for a documentation URL. Returns immediately; runs in the background.
Request body:
{ "url": "https://fastapi.tiangolo.com/reference/apirouter/" }Response:
{
"status": "started",
"pages_indexed": 0,
"library_name": "fastapi.tiangolo.com",
"message": "Crawl started. Poll /api/crawl/status for progress."
}Polls the current indexing status for a given URL.
Response states: crawling → parsing → indexing → done | error
{
"status": "done",
"pages": 42,
"functions": 187
}Runs the full RAG pipeline against the indexed documentation.
Request body:
{
"content": "How do I add a route to a router?",
"source_url": "https://fastapi.tiangolo.com/reference/apirouter/",
"use_reranker": true
}Response:
{
"status": "success",
"intent": "concept_explain",
"processed_content": "To add a route to a router in FastAPI...",
"recommended_functions": ["include_router"],
"use_when": ["When you want to add a route to a router in FastAPI"],
"avoid_when": ["When you have a simple API with few routes"],
"code_snippet": "from fastapi import APIRouter...",
"source_url": "https://fastapi.tiangolo.com/reference/apirouter/",
"confidence": 1.0,
"fixes": [],
"retrieved_chunks": [...]
}Returns all parsed function objects for a completed crawl job.
Before retrieval, every query is classified into one of three intents:
| Intent | Trigger | Output format |
|---|---|---|
function_search |
Default — action-oriented queries | Ranked functions + use/avoid when + code |
error_fix |
Traceback detected or capitalized ErrorName | Root cause + numbered fixes + working code |
concept_explain |
"how", "why", "what", "difference", etc. | Plain explanation + example |
Ambiguous queries are resolved by the local LLM before retrieval.
Both BM25 (keyword) and dense embedding (semantic) searches run in parallel. Results are merged using Reciprocal Rank Fusion: score = Σ 1 / (k + rank_i) where k = 60
The top-5 fused results are then re-scored by the cross-encoder reranker, and the top-3 are passed to the LLM.
The LLM receives only the retrieved documentation chunks — no general knowledge is used. The system prompt enforces strict grounding:
"Answer ONLY using the provided documentation chunks. If the answer is not in the chunks, say so. Do not use general knowledge."
The model is instructed to return a fixed JSON schema, ensuring the frontend always receives structured, renderable output.
ChromaDB persists metadata in SQLite under BACKEND/data/chroma (or under DOCUMENTOR_DATA_DIR/chroma if set). Try:
- Ensure the backend process can create and write to that directory (not a read-only mount).
- Set
DOCUMENTOR_DATA_DIRto a folder you own, then restart the API. - Stop the server, delete the
chromafolder inside your data directory (not the whole repo), and re-run indexing so Chroma can recreate a clean database. - Avoid running multiple API workers against the same Chroma path; use a single Uvicorn worker for local dev, or give each worker its own data directory.
Confirm ollama serve is running and that OLLAMA_URL / OLLAMA_MODEL in BACKEND/.env match your setup (ollama list).
Set VITE_API_BASE_URL in FRONTEND/.env to the full origin of the FastAPI server (for example http://localhost:8000).
Run these scripts to verify individual pipeline stages:
# Test the Cloudflare crawl connection
cd BACKEND
python test_crawl.py
# Test the parser against a live documentation page
python test_parser.py
# Output saved to: test_parser_output.jsonContributions are welcome. To contribute:
- Fork the repository
- Create a feature branch:
git checkout -b feature/your-feature - Commit your changes:
git commit -m 'Add your feature' - Push to the branch:
git push origin feature/your-feature - Open a pull request
Please ensure your changes do not break the existing parser or retrieval pipeline before submitting.
- Sandboxed Python snippet executor (verify code runs before showing)
- VS Code extension
- Multi-library comparison mode
- Auto re-index on new library version detection
- Conversation history persistence
- GitHub Issues + Stack Overflow ingestion as additional context
This project is licensed under the MIT License. See LICENSE for details.
Built by AKSHAT RAI as part of an intermediate-level RAG systems project.
"Students waste hours scanning documentation for a function they can't name, to solve a problem they can only describe in plain English." — This project exists to fix that.