🚀 GitHub to Qdrant Vector Processing Pipeline

High-performance document processing pipeline that transforms GitHub repositories containing markdown, PDFs, and 150+ text file types into searchable vector databases for AI applications. Now with multi-repository batch processing, multiple embedding providers, and state-of-the-art deduplication algorithms.

🌟 Overview

This project automatically processes GitHub repositories containing documentation (markdown, PDFs, code, and text files) and creates optimized vector embeddings for Retrieval-Augmented Generation (RAG), semantic search, and AI chat applications. It supports multiple embedding providers including cloud-based APIs and local models, featuring cutting-edge deduplication algorithms and intelligent PDF extraction.

✨ Key Features

🔄 Multi-Repository Processing: Process multiple repos sequentially with one command
🤖 Multi-Provider Support: Azure OpenAI, Mistral AI & Sentence Transformers
📑 PDF Processing: PyMuPDF (60x faster), PyPDFLoader, and Mistral OCR API
⚡ 5-15x Faster Processing: Vectorized duplicate detection algorithms
🎯 Smart Deduplication: Two-stage content hash + semantic similarity
📊 Real-time Progress: Detailed processing reports with summary statistics
🛡️ Production Ready: Error handling, rate limiting, retry logic
🎛️ Highly Configurable: YAML configs with environment variable support
📚 150+ File Types: Process code, docs, configs, PDFs, and more
📄 Individual File Processing: Maintain document boundaries for better search (v0.3.3)
🔒 Deterministic IDs: Consistent vector IDs across runs prevent duplicates (v0.3.3)
💾 Embedding Cache: LRU cache reduces API calls by 20-30% (v0.3.2)
🎨 Semantic Chunking: Context-aware text splitting for better retrieval (v0.3.1)
📈 Quality Scoring: Rank chunks by information density and relevance (v0.3.2)
🔧 Configurable Payloads: Choose content fields for compatibility (v0.3.2)
🧩 Hybrid Retrieval: Optional dense + sparse BM25 search with Qdrant Query API (v0.5)
🗜️ TurboQuant Ready: Optional Qdrant 1.18+ TurboQuant collection setup (v0.5)

🎯 Perfect For

AI Chatbots - Create knowledge bases from documentation
Semantic Search - Enable intelligent document discovery
RAG Applications - Augment LLMs with domain-specific knowledge
Technical Documentation - Process markdown, PDFs, and code with specialized embeddings
Content Processing - Handles 150+ file types including PDFs, HTML, TXT, code files

🏃‍♂️ Quick Start

1. Clone and Setup

git clone <your-repo-url>
cd github-qdrant-sync
python3 -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate
pip install -r requirements.txt

2. Configure API Keys

Set up your API keys for your chosen embedding provider (see Configuration Guide below)

3. Configure

Copy the example config and add your API keys:

cp config.yaml.example config.yaml
# Edit config.yaml with your settings

# Set API keys in .env file for security
cp .env.example .env
# Edit .env with your API keys

4. Run

python github_to_qdrant.py config.yaml

🛠️ Installation & Dependencies

System Requirements

Python 3.10+
4GB+ RAM (for large repositories)
Internet connection (for API calls)

Dependencies

pip install -r requirements.txt

Core Dependencies:

langchain - Document processing
qdrant-client>=1.18.0 - Vector database client with TurboQuant support
openai - Azure OpenAI embeddings
numpy - Vectorized operations
mistralai - Mistral AI embeddings
sentence-transformers - Local embedding models (optional)

Virtual Environment (Recommended)

python3 -m venv venv
source venv/bin/activate  # Linux/Mac
# or
venv\Scripts\activate     # Windows

pip install -r requirements.txt

⚙️ Configuration Guide

🔧 Basic Configuration

The project uses YAML configuration files with environment variable support. Start with config.yaml.example:

# Embedding provider selection
embedding_provider: azure_openai  # or mistral_ai, sentence_transformers

github:
  repository_url: https://github.com/your-org/your-repo.git
  branch: main
  token: ${GITHUB_TOKEN}  # For private repos (from .env file)

qdrant:
  url: ${QDRANT_URL}  # e.g., https://your-cluster.qdrant.io
  api_key: ${QDRANT_API_KEY}
  collection_name: your-collection
  vector_size: 3072  # Must match embedding model
  upload_batch_size: 64

🤖 Embedding Providers

Azure OpenAI

embedding_provider: azure_openai

azure_openai:
  api_key: ${AZURE_OPENAI_API_KEY}
  endpoint: ${AZURE_OPENAI_ENDPOINT}
  deployment_name: text-embedding-3-large
  api_version: "2024-02-01"

qdrant:
  vector_size: 3072  # for text-embedding-3-large

Mistral AI

embedding_provider: mistral_ai

mistral_ai:
  api_key: ${MISTRAL_API_KEY}
  model: codestral-embed  # or mistral-embed
  output_dimension: 3072

qdrant:
  vector_size: 3072

Sentence Transformers (Local)

embedding_provider: sentence_transformers

sentence_transformers:
  model: intfloat/multilingual-e5-large
  vector_size: 1024

qdrant:
  vector_size: 1024
  vector_name: intfloat/multilingual-e5-large  # Optional: for MCP compatibility

Sentence Transformers Benefits:

✅ No API Keys Required - Runs locally
✅ No Rate Limits - Process any amount of data
✅ Privacy - Data never leaves your machine
✅ Cost Effective - No per-token charges
✅ Offline Capable - Works without internet

🎛️ Performance Tuning

processing:
  chunk_size: 1000              # Characters per chunk
  chunk_overlap: 200            # Overlap for context
  chunking_strategy: semantic   # 'semantic' or 'recursive' (v0.3.1+)
  embedding_batch_size: 50      # Optimized batch size
  batch_delay_seconds: 1        # Required for Azure OpenAI
  deduplication_enabled: true   # Enable smart deduplication
  similarity_threshold: 0.95    # Duplicate detection threshold
  file_mode: all_text          # Process all text files (or markdown_only)

# New in v0.3.2: Configurable payload fields
payload:
  metadata_structure: nested     # nested or flat
  content_fields:
    - content      # For n8n compatibility
    - page_content # For LangChain
    - document     # For MCP compatibility
    - text         # Alternative field name
  preview_length: 200           # Preview snippet length
  minimal_mode: false           # Reduce payload size by 50%

🧩 Qdrant 1.18+ Options

TurboQuant is opt-in and requires Qdrant server or Cloud 1.18+ plus qdrant-client>=1.18.0. Start with bits4 on a test collection before enabling it for production data. Sparse BM25 vectors use Qdrant Document inference; set qdrant.cloud_inference: true for Qdrant Cloud/server inference or install qdrant-client[fastembed] for local inference.

qdrant:
  vector_name: dense  # Required when sparse_vector.enabled is true
  cloud_inference: false

  quantization:
    enabled: false
    method: turbo
    bits: bits4       # bits4, bits2, bits1_5, or bits1
    always_ram: true
    apply_to_existing_collections: false

  sparse_vector:
    enabled: false
    name: sparse
    model: qdrant/bm25

Other recent Qdrant features are mainly operational wins for this project: 1.18 adds memory monitoring, per-collection metrics, strict-mode guardrails, audit-log querying, and in-place named-vector schema changes; 1.17 improves write-load search latency and observability; 1.16 improves filtered search and disk-efficient storage. Hybrid retrieval and TurboQuant are the pieces wired into this repo because they directly affect ingestion and RAG quality.

🚀 Usage Examples

Basic Usage

# Process with default config
python github_to_qdrant.py config.yaml

# Process different repository
python github_to_qdrant.py config.yaml --repo-url https://github.com/other/repo.git

🔄 Multi-Repository Processing (New!)

Process multiple repositories sequentially with a single command:

# Process multiple repositories from a list file
python github_to_qdrant.py config.yaml --repo-list repositories.yaml

Repository List File Format (repositories.yaml):

repositories:
  # Basic repository
  - url: https://github.com/langchain-ai/langchain.git
    collection_name: langchain-docs

  # Repository with specific branch
  - url: https://github.com/openai/openai-python.git
    branch: main
    collection_name: openai-python-docs

  # Private repository using SSH
  - url: git@github.com:myorg/private-repo.git
    branch: develop
    collection_name: private-docs

  # Multiple versions of the same project
  - url: https://github.com/facebook/react.git
    branch: main
    collection_name: react-latest

  - url: https://github.com/facebook/react.git
    branch: 18.x
    collection_name: react-v18

Features:

✅ Sequential processing with progress tracking
✅ Individual collection names per repository
✅ Continues processing if one repository fails
✅ Comprehensive summary report at the end
✅ All global settings from config.yaml apply

Example Output:

============================================================
Processing repository 2/5
Repository: https://github.com/openai/openai-python.git
Branch: main
Collection: openai-python-docs
============================================================
[... processing output ...]

============================================================
MULTI-REPOSITORY PROCESSING SUMMARY
============================================================
Total repositories: 5
✅ Successful: 4
❌ Failed: 1

Details:
------------------------------------------------------------
✅ langchain → langchain-docs
   Files: 234, Chunks: 1,234
   Time: 45.2s
✅ openai-python → openai-python-docs
   Files: 89, Chunks: 567
   Time: 23.1s
❌ private-repo → Failed
   Error: Authentication error
✅ react → react-latest
   Files: 456, Chunks: 2,345
   Time: 89.3s
✅ react → react-v18
   Files: 423, Chunks: 2,123
   Time: 82.7s
------------------------------------------------------------

Totals:
   Files processed: 1,202
   Chunks created: 6,269
   Processing time: 240.3s (4m 0s)

Embedding Cache Performance:
   💾 Total hits: 1,543
   💾 Total misses: 4,726
   💾 Hit rate: 24.6%
   💾 Cache size: 500/500
============================================================

🔍 RAG Retrieval & Querying

Query your vector database using the included retrieval CLI:

Basic Query:

python rag_retrieval.py config.yaml --query "How do I configure authentication?"

With Filters:

# In config.yaml:
retrieval:
  mode: dense
  top_k: 10
  fetch_k: 40  # Retrieves more candidates for grouping
  max_chunks_per_file: 3  # Caps results per file
  filters:
    repository: my-repo-name  # Optional filtering

Hybrid Dense + Sparse Retrieval (Qdrant 1.18+):

qdrant:
  vector_name: dense
  cloud_inference: true  # Or install qdrant-client[fastembed] for local sparse inference
  sparse_vector:
    enabled: true
    name: sparse
    model: qdrant/bm25

retrieval:
  mode: hybrid
  fusion: rrf
  top_k: 10
  fetch_k: 40

Hybrid mode stores a named dense vector and a sparse BM25 vector per chunk, then uses Qdrant Query API prefetches with reciprocal-rank fusion. Existing unnamed-vector collections should be recreated or migrated intentionally before enabling hybrid mode.

JSON Output (for programmatic use):

python rag_retrieval.py config.yaml --query "setup guide" --format json

Verbose Logging:

python rag_retrieval.py config.yaml --query "api reference" --verbose

Features:

✅ Smart Grouping: Caps results per file for better context diversity
✅ Parent Window Expansion: Retrieve surrounding context around matched chunks
✅ Multiple Output Formats: Human-readable text or machine-readable JSON
✅ Flexible Filtering: Filter by repository, file type, or any metadata field
✅ Marker Exclusion: Internal incremental-sync markers are hidden by default
✅ Hybrid Search: Optional dense+sparse retrieval for better keyword recall
✅ Timing Information: Debug mode shows embedding, search, and grouping times
✅ Robust Error Handling: Clear error messages and suggestions

Parent Window Context:

# Retrieve surrounding chunks for expanded context
python rag_retrieval.py config.yaml --query "installation" --with-parent-window

Example Output:

INFO: Querying collection: my-docs
INFO: Query: How do I configure authentication?
INFO: Returning 10 results

#1 score=0.8234 file=docs/authentication.md
preview: Configure authentication using OAuth2 or API keys...

#2 score=0.7891 file=guides/setup.md
preview: Authentication setup requires the following steps...

#3 score=0.7654 file=api/reference.md
preview: API authentication methods include bearer tokens...

Configuration Options:

retrieval:
  mode: dense                 # dense or hybrid
  fusion: rrf                 # hybrid only: rrf or dbsf
  top_k: 10                  # Final number of results to return
  fetch_k: 40                # Candidates to fetch before grouping (should be 3-4x top_k)
  max_chunks_per_file: 3     # Maximum results per file
  parent_window: 2           # Chunks before/after for context expansion
  filters:                   # Optional metadata filters
    repository: my-repo
    source_type: markdown

CLI Arguments:

--query: Your search query (required)
--limit: Override config's top_k
--format: Output format (text or json)
--with-parent-window: Enable context expansion
--verbose / -v: Enable debug logging
--quiet / -q: Suppress info messages

Use Cases:

🤖 AI Chatbots: Retrieve relevant context for LLM responses
🔍 Semantic Search: Find similar content across documentation
📚 Documentation Q&A: Answer questions from your knowledge base
🧪 Testing Retrieval Quality: Evaluate embedding and chunking strategies

Multiple Configurations

# Different repository configurations
python github_to_qdrant.py config_technical.yaml

# Multi-language documentation
python github_to_qdrant.py config_multilang.yaml

# Large documentation projects
python github_to_qdrant.py config_enterprise.yaml

📑 PDF Processing

The pipeline includes state-of-the-art PDF processing with three modes:

1. Local Mode (Offline, Fast)

pdf_processing:
  enabled: true
  mode: local  # Uses PyMuPDF (60x faster) with PyPDFLoader fallback

2. Cloud Mode (Mistral OCR API)

pdf_processing:
  enabled: true
  mode: cloud  # Best quality, handles scanned PDFs, $0.001/page
  cloud:
    max_pages_per_doc: 100  # Cost control

3. Hybrid Mode (Smart Selection)

pdf_processing:
  enabled: true
  mode: hybrid  # Local first, cloud for complex/scanned PDFs
  hybrid:
    force_cloud_patterns:  # Always use OCR for these
      - "*scan*.pdf"
      - "*ocr*.pdf"

Advanced Examples

Process specific branch:

github:
  repository_url: https://github.com/your-org/your-repo.git
  branch: develop

Switch to Mistral AI:

embedding_provider: mistral_ai

mistral_ai:
  api_key: ${MISTRAL_API_KEY}
  model: codestral-embed
  output_dimension: 3072

Use Local Models:

embedding_provider: sentence_transformers

sentence_transformers:
  model: intfloat/multilingual-e5-large
  vector_size: 1024

📊 Embedding Models Comparison

Provider	Model	Dimensions	Best For	Context
Azure OpenAI	text-embedding-ada-002	1536	General text	2,048 tokens
Azure OpenAI	text-embedding-3-small	1536	Efficient processing	8,191 tokens
Azure OpenAI	text-embedding-3-large	3072	Best quality	8,191 tokens
Mistral AI	mistral-embed	1024	General text	8,000 tokens
Mistral AI	codestral-embed	3072	Technical docs	8,000 tokens
Sentence Transformers	all-MiniLM-L6-v2	384	Lightweight/Fast	256 tokens
Sentence Transformers	multilingual-e5-large	1024	Multilingual	512 tokens

💡 Recommendations

Technical Documentation: Use codestral-embed (Mistral AI)
General Documentation: Use text-embedding-3-large (Azure OpenAI)
Cost-Effective: Use text-embedding-3-small (Azure OpenAI)
Code Repositories: Use codestral-embed (Mistral AI)
Privacy/Offline: Use multilingual-e5-large (Sentence Transformers)
Fast/Lightweight: Use all-MiniLM-L6-v2 (Sentence Transformers)
Multilingual Content: Use multilingual-e5-large (Sentence Transformers)
No API Costs: Use any Sentence Transformers model

🚄 Performance & Optimization

⚡ Deduplication Performance

This project features cutting-edge deduplication that's 5-15x faster than traditional methods:

Traditional Approach (Slow)

O(n²) complexity: Each chunk compared to ALL previous chunks
Individual similarity calculations
No progress reporting
Hours for large repositories

Our Optimized Approach (Fast)

✅ Content hash pre-filtering: Instant exact duplicate removal
✅ Vectorized similarity: Batch NumPy operations
✅ Progress reporting: Real-time feedback
✅ Memory optimization: Batched processing
✅ Smart thresholding: Configurable similarity detection

📈 Processing Speed

Repository Size	Traditional	Optimized	Speedup
Small (100 files)	5 minutes	1 minute	5x
Medium (500 files)	45 minutes	5 minutes	9x
Large (1000+ files)	3+ hours	15 minutes	12x+

⚙️ Rate Limiting

Azure OpenAI:

Batch size: 50 chunks
Delay: 1 second between batches
Auto-retry with exponential backoff

Mistral AI:

Batch size: 50 chunks
Delay: 1 second between batches
Shorter retry delays

Sentence Transformers:

No rate limits (local processing)
Batch size: 50 chunks (for memory management)
Processing speed depends on hardware (CPU/GPU)

🧠 Memory Usage

Batched processing: Prevents memory overflow
Streaming embeddings: Process chunks incrementally
Automatic cleanup: Temporary files removed

Local Models (Sentence Transformers):

Model loaded once, reused for all chunks
Additional VRAM usage for GPU acceleration
Faster processing with dedicated GPU

📦 Data Structure & Metadata

🏗️ Payload Structure (Updated v0.3.2)

This project uses a configurable payload structure for maximum compatibility with n8n, LangChain, MCP, and other frameworks. Each document chunk is stored with the following structure:

⚡ Qdrant Payload Indexes (Optional)

For large collections, enabling payload indexes improves performance for filtered queries (e.g. “only PDFs”, “only a specific repository”, “only a specific file path”).

Nested metadata (payload.metadata_structure: nested): index fields use paths like metadata.repository.
Flat metadata (payload.metadata_structure: flat): index fields use paths like repository.

Configure in qdrant.payload_indexes in config.yaml (see config.yaml.example).

🔁 Incremental Sync (track_file_changes) and Shared Collections

When processing.track_file_changes: true, the pipeline computes a SHA-256 file_hash per file and uses deterministic identifiers to safely support either:

One repo per collection, or
Multiple repos/branches sharing a collection.

Key metadata fields:

repo_id: SHA-256 of repo_url@branch
file_id: SHA-256 of repo_id:file_path
file_upload_id: SHA-256 of file_id:file_hash

This prevents accidental cross-repo deletes when different repos contain the same file_path, and it enables auto-repair of partial uploads by writing a per-file marker only after upload succeeds.

{
  // Configurable content fields (choose which to include via config)
  "content": "Full document text content here...",      // n8n
  "page_content": "Full document text content here...",  // LangChain
  "document": "Full document text content here...",      // MCP
  "text": "Full document text content here...",          // Alternative

  // Flattened metadata (v0.3.2) - no nesting for better performance
  "doc_id": "repo-name_file.md_123",
  "chunk_id": 123,
  "source": "path/to/file.md",
  "source_type": "markdown",  // pdf, code, config, etc.
  "repository": "your-repo-name",
  "branch": "main",
  "preview": "First 200 characters of content...",
  "chunk_size": 850,
  "token_count": 213,          // NEW in v0.3.2
  "quality_score": 0.87,       // NEW in v0.3.2 (0-1 scale)
  "timestamp": 1705147200,     // Unix timestamp (smaller)
  "content_hash": "abc123de",
  "extraction_method": "default",

  // PDF-specific fields (when applicable)
  "page_number": 5,
  "total_pages": 42
}

📋 Field Descriptions

Content Fields (configurable via payload.content_fields):

content - Full text for n8n compatibility
page_content - Full text for LangChain compatibility
document - Full text for MCP server compatibility
text - Alternative field name some systems use

Metadata Fields (flattened in v0.3.2):

doc_id - Unique document identifier (repo_file_chunk)
chunk_id - Sequential chunk number
source - Path to source file
source_type - File type (pdf, markdown, code, config, etc.)
preview - Configurable preview snippet (default 200 chars)
token_count - Token count for LLM context management (v0.3.2)
quality_score - Content quality score 0-1 (v0.3.2)
timestamp - Unix timestamp (more compact than ISO)
content_hash - MD5 hash for duplicate detection
extraction_method - How content was extracted

✅ Benefits

🔧 Standard Compatible - Works seamlessly with LangChain, n8n, and most Qdrant clients
🤖 MCP Server Ready - Includes document field for Qdrant MCP server compatibility
💾 Efficient Storage - Clean separation between standard and compatibility fields
🔍 Easy Filtering - Structured metadata enables precise search and filtering
📊 Debug Friendly - Preview field allows quick content inspection without full retrieval

📁 Project Structure

github-qdrant-sync/
├── github_to_qdrant.py      # 🌟 Main processing script
├── pdf_processor.py         # 📑 Advanced PDF processing module
├── config.yaml.example      # 📝 Configuration template with docs
├── config.yaml              # 🔧 Your configuration (gitignored)
├── repositories.yaml.example # 📋 Multi-repo list template
├── repositories.yaml        # 📋 Your repository list (gitignored)
├── .env.example             # 🔐 Environment variables template
├── .env                     # 🔑 Your API keys (gitignored)
├── requirements.txt         # 📦 Python dependencies
├── .gitignore              # 🚫 Git exclusions
├── README.md               # 📖 This documentation
├── CLAUDE.md               # 🤖 AI assistant context (gitignored)
├── venv/                   # 🐍 Virtual environment (gitignored)
└── markdown/               # 📄 Generated markdown output (gitignored)
    ├── repo-name/
    │   ├── __combined_markdown.md
    │   ├── folder1.md
    │   └── folder2.md
    └── ...

📄 Configuration Files

config.yaml.example - Template with inline documentation and examples
config.yaml - Your custom configuration (gitignored)
repositories.yaml.example - Template for multi-repository processing
repositories.yaml - Your repository list (gitignored)
.env.example - Template for environment variables
.env - Your API keys and sensitive data (gitignored)

Why YAML?

✅ Cleaner syntax - More readable than JSON ✅ Comments support - Inline documentation ✅ Environment variables - Secure API key management via ${VAR_NAME} syntax ✅ Multi-line strings - Better for long text values ✅ Default values - Support for ${VAR:-default} pattern

🔧 Troubleshooting

Common Issues

🚫 "ModuleNotFoundError: No module named 'mistralai'"

# Reinstall requirements to ensure all dependencies are available
pip install -r requirements.txt

🔑 "Authentication Error"

Check API keys in config file
Verify endpoint URLs
Ensure API keys have proper permissions

⏱️ "Rate Limit Exceeded"

Increase batch_delay_seconds in config
Reduce embedding_batch_size
Check API quotas

🧠 "Out of Memory"

Reduce chunk_size in config
Increase batch_delay_seconds
Process smaller repositories

🔗 "Connection Failed"

Check internet connection
Verify Qdrant URL and API key
Test with smaller batch size

🐛 Debug Mode

Enable detailed logging:

logging:
  level: DEBUG
  format: "%(asctime)s - %(name)s - %(levelname)s - %(message)s"

📞 Getting Help

Check the logs - Enable DEBUG logging
Verify configuration - Use config.yaml.example
Test connections - Run with minimal config
Check dependencies - Reinstall requirements
API quotas - Verify account limits

🔒 Security & Best Practices

🔐 API Key Management

❌ Never commit API keys to Git!

✅ Safe practices:

# Use .env file (recommended)
cp .env.example .env
# Edit .env with your API keys:
# AZURE_OPENAI_API_KEY=your-key
# MISTRAL_API_KEY=your-key
# QDRANT_API_KEY=your-key
# GITHUB_TOKEN=your-token

# Then use environment variables in config.yaml:
# api_key: ${AZURE_OPENAI_API_KEY}

# Or use separate config files (gitignored)
cp config.yaml.example config.local.yaml
# Edit config.local.yaml with real keys
python github_to_qdrant.py config.local.yaml

🛡️ Production Deployment

Use environment variables for secrets
Enable rate limiting
Monitor API usage and costs
Set up proper logging
Use dedicated service accounts

🎯 Example Workflows

Workflow 1: Technical Documentation Site

# 1. Configure for technical content in config.yaml
embedding_provider: mistral_ai
mistral_ai:
  model: codestral-embed
  output_dimension: 3072

# 2. Process repository
python github_to_qdrant.py config.yaml

Workflow 2: Multi-Language Documentation

# Process different language versions
python github_to_qdrant.py config_english.yaml  # English docs
python github_to_qdrant.py config_german.yaml   # German docs
python github_to_qdrant.py config_french.yaml   # French docs

# Or use multi-repo processing with single config
python github_to_qdrant.py config.yaml --repo-list multilang_repos.yaml

Workflow 3: Continuous Integration

#!/bin/bash
# Update vector database when docs change
git pull origin main
python github_to_qdrant.py config.yaml
echo "Vector database updated successfully"

📊 Performance Metrics

Real-World Results

Large Documentation Repository (1,200+ files):

Processing time: 12 minutes (vs 3+ hours traditional)
Duplicates removed: 1,847 chunks (23% of total)
Final chunks: 6,234 unique vectors
Accuracy: 99.8% (manual validation)

Performance breakdown:

Repository cloning: 30 seconds
Markdown processing: 2 minutes
Embedding generation: 6 minutes
Deduplication: 3 minutes
Upload to Qdrant: 1 minute

📝 Changelog

v0.5.0 (2026-05-26) - Qdrant 1.18 & RAG Quality Modernization

✨ TurboQuant Support: Optional Qdrant 1.18+ TurboQuant collection config
✨ Hybrid Retrieval: Optional dense + sparse BM25 ingestion and RRF querying
🔧 Shared Qdrant Config: Ingestion and retrieval now share connection parsing
🐛 Metadata Config Fix: payload.metadata_structure is canonical, with legacy fallback
🐛 Exclude Pattern Fixes: Glob-aware excludes for paths like *.pyc
🧪 Unit Tests: Added pytest coverage for config, payloads, Qdrant setup, and retrieval

v0.4.1 (2025-12-19) - Robustness & Reliability Improvements

🐛 Retrieval Fixes: Fixed fetch_k defaults, collection existence checks, empty result guidance, and debug logging
✨ CLI Polish: Added JSON output, verbose/quiet logging, timing details, and clearer help text
🔧 Pipeline Reliability: Added orphaned marker cleanup, config validation, and payload index mismatch warnings
📚 Documentation: Expanded retrieval docs and updated example configs

v0.4.0 (2025-12-17) - Retrieval, Indexing & Robust Incremental Sync

✨ Retrieval CLI: Added rag_retrieval.py for querying Qdrant collections
✨ Grouping by File: Optionally cap/interleave results per file for better context diversity
✨ Token-aware Chunking: Optional token_recursive chunking strategy using tiktoken
✨ Qdrant Payload Indexes: Optional payload index management for faster filtered queries
🔁 Robust Incremental Sync: Dedup-safe skipping via per-file markers written after successful upload
🛡️ Shared Collection Safety: Multi-repo/branch safe scoping with repo_id / file_id / file_upload_id
🐛 Track File Changes Fixes: Safer deletes + legacy fallback handling for older collections

v0.3.4 (2025-11-12) - Metadata & Embedding Config Standardization

✨ Standardized embedding config: Use model consistently across providers
🐛 Metadata field fixes: Improved repository/name mapping and attribution fields
📌 Embedding tracking: Persist embedding_provider and embedding_model in metadata

v0.3.3 (2025-09-16) - Individual File Processing & Deduplication

✨ Individual File Processing: Process files separately for better context preservation
✨ Deterministic ID Generation: Fixed duplicate vector creation on repeated runs
🐛 Fixed Duplicate Vectors: Resolved issue where each run added exactly 1 duplicate vector
🐛 Fixed combine_documents Setting: Properly respects combine_documents: false configuration
🚀 Clean Interrupt Handling: Improved Ctrl+C handling without verbose tracebacks
🔧 Type Safety: Fixed type annotations for better IDE support
📊 Better Search Quality: Individual file processing maintains document boundaries

v0.3.2 (2025-09-14) - Performance & Compatibility

✨ Embedding Cache: LRU cache reduces API calls by 20-30%
✨ Configurable Payloads: Choose content fields for n8n/LangChain/MCP compatibility
✨ Quality Scoring: Rank chunks by information density (0-1 scale)
✨ Token Counting: Track tokens for LLM context management
🚀 Flattened Metadata: Improved filtering performance
🚀 CI/CD Optimization: 10x faster (30s vs 3min)
🐛 Fixed formatting and import issues

v0.3.1 (2025-09-14) - Semantic Intelligence

✨ Semantic Chunking: Context-aware text splitting for 30% better retrieval
✨ Mistral Vision API: Extract text/content from images in PDFs
🚀 Improved PDF processing with image extraction
🐛 Fixed PDF processing edge cases

v0.3.0 (2025-09-12) - Multi-Repository Support

✨ Multi-Repository Processing: Process multiple repos with one command
✨ Repository Lists: YAML configuration for batch processing
📊 Comprehensive summary reports with statistics
🚀 Shared resources across repositories

v0.2.0 (2025-09-10) - Enhanced Processing

✨ PDF Processing: Three modes (local, cloud, hybrid)
✨ Mistral OCR API: Cloud-based PDF extraction
✨ 150+ File Types: Extended file type support
🚀 5-15x Faster: Vectorized deduplication

v0.1.0 (2025-09-08) - Initial Release

🎉 Initial public release
✨ Azure OpenAI and Mistral AI support
✨ Basic markdown and text processing
✨ Qdrant integration

📜 License

This project is licensed under the MIT License - see the LICENSE file for details.

🙏 Acknowledgments

Qdrant - High-performance vector database
Azure OpenAI - Advanced embedding models
Mistral AI - Specialized code embeddings
LangChain - Document processing framework

Made with ❤️ for the AI community

Transform your documentation into intelligent, searchable knowledge bases.

Name		Name	Last commit message	Last commit date
.github/workflows		.github/workflows
tests		tests
.env.example		.env.example
.gitignore		.gitignore
LICENSE		LICENSE
README.md		README.md
config.payload_index_smoke.yaml.example		config.payload_index_smoke.yaml.example
config.yaml.example		config.yaml.example
github_to_qdrant.py		github_to_qdrant.py
pdf_processor.py		pdf_processor.py
rag_retrieval.py		rag_retrieval.py
repositories.shared_collection.smoke.yaml.example		repositories.shared_collection.smoke.yaml.example
repositories.yaml.example		repositories.yaml.example
requirements.txt		requirements.txt
ruff.toml		ruff.toml
run_sync.sh		run_sync.sh

Latest commit

History

Folders and files

Repository files navigation