Build Your Own Private AI Expert That Answers From YOUR Documents
No cloud. No subscriptions. Your data stays on your machine.
RAG (Retrieval Augmented Generation) lets your AI answer questions from YOUR documents—not just generic internet knowledge.
Instead of the AI making things up, it:
- Searches your documents for relevant information
- Retrieves the most relevant chunks
- Generates an answer grounded in your actual data
Result: No hallucinations. Real answers. From your files.
By the end of this guide, you'll have:
- ✅ AI that reads and understands your documents
- ✅ Ask questions in plain English, get accurate answers
- ✅ Upload PDFs, TXT, DOCX, MD files
- ✅ 100% local and private
- ✅ Works offline
| Requirement | Minimum | Recommended |
|---|---|---|
| OS | Windows 10/11, macOS 12+, Ubuntu 22.04+ | Windows 11, macOS 14+ |
| RAM | 8GB | 16GB+ |
| Storage | 10GB free | 20GB+ free |
| GPU | Not required | NVIDIA GPU (faster) |
| Software | Purpose |
|---|---|
| Ollama | Runs local LLM models |
| AnythingLLM | RAG platform with UI |
Windows (PowerShell):
# Check Windows version
winver
# Check available RAM
systeminfo | findstr "Total Physical Memory"
# Check if Ollama is installed (if you have it)
ollama --versionLinux/macOS:
# Check RAM
free -h
# Check Ollama
ollama --versionSkip this section if you already have Ollama running.
Ollama is a local LLM runner. It downloads and runs AI models directly on your machine—no cloud required.
-
Download Ollama:
- Go to ollama.ai/download
- Click "Download for Windows"
- Run the installer
-
Verify Installation:
ollama --version -
Pull a Model:
# Pull Llama 3 (8B) - Good balance of speed and quality ollama pull llama3 # Or pull Mistral (7B) - Fast and capable ollama pull mistral
-
Test the Model:
ollama run llama3 "Hello, how are you?"
# One-line install
curl -fsSL https://ollama.ai/install.sh | sh
# Verify
ollama --version
# Pull model
ollama pull llama3
# Test
ollama run llama3 "Hello!"# Download from ollama.ai or use Homebrew
brew install ollama
# Start Ollama service
ollama serve
# Pull model (new terminal)
ollama pull llama3| Model | Size | Best For |
|---|---|---|
llama3 |
4.7GB | General purpose, good quality |
mistral |
4.1GB | Fast, good for Q&A |
phi3 |
2.3GB | Lightweight, quick responses |
llama3:70b |
40GB | Best quality (needs 64GB+ RAM) |
Pull a model:
ollama pull llama3# Check if Ollama API is accessible
curl http://localhost:11434/api/tagsExpected: JSON response with your models listed.
AnythingLLM is an all-in-one RAG platform. It handles:
- Document upload and processing
- Chunking and embeddings
- Vector storage
- Chat interface
All in one simple application.
-
Download AnythingLLM:
- Go to anythingllm.com/download
- Click "Download for Windows"
- Run the
.exeinstaller
-
Launch AnythingLLM:
- Find it in Start Menu
- Or double-click desktop shortcut
# Download AppImage
wget https://anythingllm.com/download/linux
# Make executable
chmod +x AnythingLLM-*.AppImage
# Run
./AnythingLLM-*.AppImage# Download from anythingllm.com
# Or use Homebrew
brew install --cask anythingllm# Pull and run
docker pull mintplexlabs/anythingllm
docker run -d -p 3001:3001 \
--name anythingllm \
-v anythingllm_storage:/app/server/storage \
mintplexlabs/anythingllmAccess at: http://localhost:3001
When AnythingLLM opens for the first time:
- LLM Provider: Select "Ollama"
- Ollama URL:
http://localhost:11434(default) - Model: Select your pulled model (e.g.,
llama3) - Embedding Model: Select "AnythingLLM Embedder" (built-in) or "Ollama"
- Vector Database: Select "LanceDB" (built-in, no setup needed)
Click "Save and Continue"
RAG doesn't feed your entire document to the AI. That would be:
- Too slow
- Too expensive (tokens)
- Often impossible (context limits)
Instead, RAG chunks your document into smaller pieces.
Original Document (10 pages)
↓
[Chunk 1] - Paragraph about topic A
[Chunk 2] - Paragraph about topic B
[Chunk 3] - Paragraph about topic C
[Chunk 4] - Table with data
[Chunk 5] - Summary section
↓
Each chunk = 500-1000 characters
Step 1: Create a Workspace in AnythingLLM
- Click "New Workspace"
- Name it:
test-rag
Step 2: Upload a Document
- Click "Upload" or drag-drop a PDF/TXT file
- Watch the processing status
Step 3: View the Chunks
- Click on the uploaded document
- Select "View Chunks" or check document settings
- You'll see your document split into numbered chunks
| Setting | Default | Description |
|---|---|---|
| Chunk Size | 1000 | Characters per chunk |
| Chunk Overlap | 200 | Overlap between chunks (context continuity) |
Where to adjust:
- Workspace Settings → Document Processing
- Or: Settings → Embedding Preferences
| Chunk Size | Pros | Cons |
|---|---|---|
| Small (500) | More precise retrieval | May lose context |
| Medium (1000) | Balanced | Good default |
| Large (2000) | More context | Less precise |
Chunks are text. But computers can't search text efficiently for meaning.
Solution: Convert text to numbers (vectors).
This is called an embedding.
"The weather is sunny today"
↓
Embedding Model
↓
[0.23, -0.45, 0.67, 0.12, -0.89, ...]
↓
512-1536 numbers representing MEANING
Key insight: Similar text = Similar numbers
"The weather is sunny" → [0.23, -0.45, 0.67, ...]
"It's a bright day" → [0.21, -0.44, 0.65, ...] ← Very close!
"I love pizza" → [0.89, 0.12, -0.34, ...] ← Very different
Step 1: Upload a document (if not already done)
Step 2: Watch the embedding process
- AnythingLLM shows "Embedding..." status
- Each chunk is converted to a vector
- Progress bar shows completion
Step 3: Check embedding model
- Settings → Embedding Preferences
- See which model is creating embeddings
| Model | Speed | Quality | Setup |
|---|---|---|---|
| AnythingLLM Embedder | Fast | Good | Built-in (default) |
| Ollama (nomic-embed-text) | Medium | Better | Pull model |
| OpenAI (ada-002) | Fast | Excellent | Needs API key |
To use Ollama for embeddings:
# Pull embedding model
ollama pull nomic-embed-textThen select "Ollama" as embedder in AnythingLLM settings.
In AnythingLLM:
- Go to your workspace
- Click on uploaded document
- Check status: "Embedded" ✓
Embeddings need to be stored somewhere for fast searching.
That's what a Vector Database does.
Document Uploaded
↓
Chunked into pieces
↓
Each chunk embedded (numbers)
↓
Vectors stored in Vector DB
↓
Ready for similarity search!
Your Question: "What's the refund policy?"
↓
Convert to embedding
↓
[0.34, -0.56, 0.78, 0.23, ...]
↓
Search Vector DB for similar vectors
↓
Found: Chunk 7, Chunk 12, Chunk 3
(most similar to your question)
↓
Return these chunks to the LLM
| Database | Setup | Best For |
|---|---|---|
| LanceDB | Built-in | Beginners (default) |
| ChromaDB | Docker | More control |
| Pinecone | Cloud | Production |
| Qdrant | Docker | Advanced |
Recommendation: Stick with LanceDB (built-in). No setup needed.
Step 1: After uploading documents, vectors are automatically stored
Step 2: Check vector count
- Workspace Settings → Documents
- Shows: "X vectors stored"
Step 3: Test similarity search
- Ask a question in chat
- AnythingLLM retrieves relevant chunks
- Watch "Sources" section show which chunks were used
┌─────────────────────────────────────────────────────────┐
│ YOUR QUESTION │
│ "What's the return policy?" │
└─────────────────────────────────────────────────────────┘
↓
┌─────────────────────────────────────────────────────────┐
│ EMBED THE QUESTION │
│ Convert to vector [0.34, -0.56, ...] │
└─────────────────────────────────────────────────────────┘
↓
┌─────────────────────────────────────────────────────────┐
│ SEARCH VECTOR DATABASE │
│ Find chunks with similar embeddings │
└─────────────────────────────────────────────────────────┘
↓
┌─────────────────────────────────────────────────────────┐
│ RETRIEVE TOP CHUNKS │
│ Chunk 7: "Returns accepted within 30 days..." │
│ Chunk 12: "Refunds processed in 5-7 business..." │
└─────────────────────────────────────────────────────────┘
↓
┌─────────────────────────────────────────────────────────┐
│ BUILD PROMPT │
│ "Based on the following context, answer the question" │
│ Context: [Chunk 7] [Chunk 12] │
│ Question: "What's the return policy?" │
└─────────────────────────────────────────────────────────┘
↓
┌─────────────────────────────────────────────────────────┐
│ LLM GENERATES │
│ "The return policy allows returns within 30 days. │
│ Refunds are processed in 5-7 business days..." │
└─────────────────────────────────────────────────────────┘
↓
┌─────────────────────────────────────────────────────────┐
│ ANSWER WITH SOURCES │
│ Answer displayed + source chunks shown │
└─────────────────────────────────────────────────────────┘
Step 1: Open your workspace with documents
Step 2: Ask a question in the chat
What does this document say about [specific topic]?
Step 3: Watch the process
- Query is processed
- Chunks are retrieved
- Answer is generated
Step 4: Check the sources
- Click "Sources" or "Citations"
- See exactly which chunks were used
- Verify the answer is grounded in your data
| Do This | Not This |
|---|---|
| "What does the policy say about refunds?" | "Tell me about refunds" |
| "Summarize section 3 of the document" | "Summarize everything" |
| "What are the three main points about X?" | "What's in the document?" |
| "Compare X and Y from the document" | "Compare stuff" |
More specific = Better retrieval = Better answers
| Setting | Default | Description |
|---|---|---|
| Top K | 4 | Number of chunks to retrieve |
| Score Threshold | 0.7 | Minimum similarity score |
Where to adjust:
- Workspace Settings → Chat Settings
- Increase Top K for broader context
- Increase threshold for stricter matching
Setup:
- Create workspace:
meeting-notes - Upload meeting transcripts (TXT, MD, DOCX)
- Start chatting
Example Queries:
What were the action items from last week's meeting?
Who is responsible for the marketing campaign?
What decisions were made about the Q4 budget?
Summarize all discussions about Project Alpha
Setup:
- Create workspace:
company-policies - Upload HR docs, policy PDFs, handbooks
- Start chatting
Example Queries:
What is the remote work policy?
How many vacation days do employees get?
What's the process for expense reimbursement?
What are the rules about using company equipment?
Setup:
- Create workspace:
study-materials - Upload textbook PDFs, lecture notes
- Start chatting
Example Queries:
Explain the concept of [topic] from chapter 5
What are the key formulas in this section?
Create a summary of the main points
Quiz me on the content from pages 50-60
Setup:
- Create workspace:
tech-docs - Upload API docs, README files, wikis
- Start chatting
Example Queries:
How do I authenticate with the API?
What parameters does the /users endpoint accept?
Show me an example of creating a new resource
What are the rate limits?
Setup:
- Create workspace:
research-project - Upload multiple related documents
- Cross-reference between them
Example Queries:
Compare the findings from document A and document B
What do all three papers say about [topic]?
Find contradictions between these sources
Synthesize the main arguments across all documents
| Tip | Why |
|---|---|
| One workspace per topic | Keeps retrieval focused |
| Clean document names | Easier to track sources |
| Update regularly | Add new docs as they come |
| Test with known answers | Verify RAG is working correctly |
Symptom: AnythingLLM can't connect to Ollama
Fix:
# Check if Ollama is running
curl http://localhost:11434/api/tags
# If not, start Ollama
ollama serveWindows: Make sure Ollama is running in system tray
Symptom: Model dropdown is empty
Fix:
# Pull a model
ollama pull llama3
# Verify
ollama listSymptom: Error when uploading PDF
Fixes:
- Check file size (some limits apply)
- Try converting PDF to TXT first
- Check AnythingLLM logs for specific error
Symptom: Answers don't match document content
Fixes:
- Increase Top K (retrieve more chunks)
- Lower threshold (match more loosely)
- Re-chunk with smaller chunk size
- Be more specific in your questions
Symptom: Takes too long to respond
Fixes:
- Use smaller model (
phi3instead ofllama3) - Reduce Top K setting
- Use SSD for storage
- Close other applications
Symptom: Crash or freeze during processing
Fixes:
- Use smaller model
- Process fewer documents at once
- Increase virtual memory (Windows)
- Close other applications
AnythingLLM Desktop:
- Help → View Logs
- Or:
%APPDATA%/AnythingLLM/logs(Windows) - Or:
~/.config/AnythingLLM/logs(Linux)
Docker:
docker logs anythingllmAdd context-specific instructions:
You are an expert assistant for [Company Name].
Always cite the specific document and section when answering.
If you're not sure, say "I couldn't find this in the documents."
Format answers with bullet points when listing items.
Where: Workspace Settings → System Prompt
Use Ollama for better embeddings:
# Pull dedicated embedding model
ollama pull nomic-embed-text
# Or mxbai for better quality
ollama pull mxbai-embed-largeConfigure in AnythingLLM:
- Settings → Embedding Preferences
- Select "Ollama"
- Choose embedding model
| Document Type | Recommended Chunk Size |
|---|---|
| Technical docs | 500-800 |
| Policies/Legal | 800-1000 |
| Books/Narratives | 1000-1500 |
| Meeting notes | 400-600 |
workspace: company-hr
└── policies.pdf
└── handbook.pdf
└── benefits.pdf
workspace: project-alpha
└── requirements.pdf
└── meeting-notes/
└── tech-specs.pdf
workspace: personal-notes
└── journal.md
└── ideas.txt
Keep workspaces focused for better retrieval accuracy.
AnythingLLM has an API for programmatic access:
# Chat endpoint
curl -X POST http://localhost:3001/api/v1/workspace/{slug}/chat \
-H "Content-Type: application/json" \
-d '{"message": "What is in my documents?"}'Get API key: Settings → API Keys
| Task | Command |
|---|---|
| Check Ollama | ollama --version |
| List models | ollama list |
| Pull model | ollama pull llama3 |
| Run model | ollama run llama3 |
| Start Ollama | ollama serve |
| Pull embedding model | ollama pull nomic-embed-text |
| Action | Where |
|---|---|
| New Workspace | + button in sidebar |
| Upload Document | Drag-drop or Upload button |
| View Chunks | Click document → View Chunks |
| Adjust Settings | Gear icon → Workspace Settings |
| Check Sources | After answer → "Sources" link |
Document → Chunks → Embeddings → Vector DB
↓
Question → Embed → Search → Retrieve → LLM → Answer
| Resource | Link |
|---|---|
| Ollama | ollama.ai |
| AnythingLLM | anythingllm.com |
| AnythingLLM Docs | docs.anythingllm.com |
| Ollama Models | ollama.ai/library |
| LangChain (Advanced) | langchain.com |
| ChromaDB (Advanced) | trychroma.com |
| Aspect | Regular Chat | RAG |
|---|---|---|
| Knowledge | Training data only | Your documents |
| Accuracy | May hallucinate | Grounded in sources |
| Up-to-date | Cutoff date | As current as your docs |
| Privacy | Data sent to cloud | 100% local |
| Customization | Generic | Domain-specific |
Built with 🧠 for Local AI Enthusiasts
Last Updated: March 2026