DocFoundry

A comprehensive Documentation Intelligence System that transforms scattered documentation into an intelligent, searchable knowledge base. DocFoundry discovers, ingests, normalizes, indexes, and serves documentation from multiple sources including websites, repositories, feeds, and local files.

Key Value Propositions:

• 🎯 Unified Knowledge Access: Centralize documentation from multiple sources into a single searchable interface
• 🧠 Intelligent Search: Combine full-text search with semantic understanding for precise results
• 🔌 Developer-First: Seamless integration with VS Code, Chrome, and development workflows
• 📊 Analytics-Driven: Comprehensive observability and search analytics for continuous improvement
• 🚀 Production-Ready: Scalable architecture supporting both SQLite and PostgreSQL backends

🚀 Features

Core Capabilities

• 🔍 Intelligent Search: Full-text search with FTS5, semantic search with embeddings, and hybrid search modes
• 📚 Multi-Source Ingestion: HTML crawling, Markdown processing, feed ingestion, and repository scanning
• 🎯 Smart Chunking: Advanced document chunking with context preservation and metadata extraction
• 📊 Analytics & Monitoring: Comprehensive observability with OpenTelemetry, performance metrics, and search analytics
• 🔌 Developer Integrations: VS Code extension, Chrome extension, and MCP server support
• 📈 Data Lineage Tracking: Complete audit trail of document processing with version tracking and change detection
• ⚡ Incremental Processing: Efficient content updates with hash-based change detection and selective reprocessing
• 🔒 Enhanced Security: Rate limiting, input validation, CORS protection, and security headers
• 🎯 Performance Gates: Automated performance monitoring with configurable thresholds and alerts

Architecture Components

• 📖 Documentation Site: MkDocs-powered human-readable documentation (mkdocs.yml, docs/)
• ⚙️ Source Registry: Configurable crawl rules and source definitions (sources/*.yaml)
• 🔄 Processing Pipelines: Modular ingestion pipelines for various content types (pipelines/)
• 🗃️ Indexing Engine: Flexible indexing with SQLite FTS5 or PostgreSQL + pgvector (indexer/)
• 🌐 RAG API: FastAPI-based REST API with search, document retrieval, and capture endpoints (server/)
• 🛠️ Development Tools: VS Code extension for in-editor search and Chrome extension for content capture
• 📋 Automation: Comprehensive Makefile with development and deployment tasks
• 🔄 Shared Services: Data models, incremental processing, and lineage tracking (services/shared/)
• 📊 Lineage API: RESTful endpoints for data lineage management and audit trails (services/api/)
• 🛠️ Migration Tools: Database migration helpers and data consistency utilities (scripts/)

Database Support

• SQLite + FTS5: Lightweight local development with full-text search
• PostgreSQL + pgvector: Production-ready with vector similarity search and advanced analytics
• Seamless Migration: Built-in migration tools and unified adapter interface

🚀 Quick Start

System Requirements

Minimum Requirements:

• Python 3.11 or higher
• 4GB RAM (8GB recommended for production)
• 2GB available disk space
• Git for version control

Optional Dependencies:

• Docker and Docker Compose for containerized deployment
• PostgreSQL 14+ with pgvector extension for production deployments
• Redis for caching (future enhancement)
• Node.js 18+ for browser extension development

Installation

# Clone the repository
git clone https://github.com/AvaPrime/docfoundry.git
cd docfoundry

# Create and activate virtual environment
python -m venv .venv
source .venv/bin/activate  # On Windows: .venv\Scripts\activate

# Install dependencies
pip install -r requirements.txt

# Build the initial index from the docs/ folder
python indexer/build_index.py

# Start the API server
uvicorn server.rag_api:app --reload --port 8001

# Optional: Preview documentation site
mkdocs serve -a 127.0.0.1:8000

Using Make Commands

DocFoundry includes a comprehensive Makefile for common tasks:

# Show all available commands
make help

# Complete setup with development dependencies
make setup

# Run the full workflow: crawl, index, and start API
make all

# Start the API server
make api

# Build documentation
make docs

# Run tests
make test

📋 Usage Workflow

1. Configure Sources

Create or edit source configuration files in sources/:

# sources/example.yaml
id: example-docs
name: Example Documentation
base_url: https://example.com/docs
start_urls:
  - https://example.com/docs/getting-started
allow_patterns:
  - "/docs/.*"
deny_patterns:
  - "/docs/internal/.*"
max_depth: 3
delay: 1.0

2. Ingest Content

# Crawl HTML content to Markdown
python pipelines/html_ingest.py sources/example.yaml

# Process feeds (RSS/Atom)
python pipelines/feed_ingest.py sources/blog.yaml

# Ingest repository content
python pipelines/repo_ingest.py sources/github-repo.yaml

3. Build Search Index

# Rebuild the search index
python indexer/build_index.py

# Or use make command
make index

4. Search and Query

REST API

# Basic search
curl -X POST http://localhost:8001/search \
  -H 'Content-Type: application/json' \
  -d '{"q": "authentication setup", "limit": 10}'

# Advanced search with filters
curl -X POST http://localhost:8001/search \
  -H 'Content-Type: application/json' \
  -d '{
    "q": "API configuration",
    "limit": 5,
    "source_filter": "example-docs",
    "search_type": "hybrid"
  }'

# Get document content
curl "http://localhost:8001/doc?path=docs/vendors/example/setup.md"

# Capture web page
curl -X POST http://localhost:8001/capture \
  -H 'Content-Type: application/json' \
  -d '{"url": "https://example.com/article", "title": "Important Article"}'

# Data Lineage API
# Get lineage summary
curl "http://localhost:8001/lineage/summary"

# Find documents needing reprocessing
curl "http://localhost:8001/lineage/reprocessing-candidates?chunker_version=2.0"

# Get document lineage history
curl "http://localhost:8001/lineage/document/123"

# Trigger reprocessing
curl -X POST "http://localhost:8001/lineage/reprocess/123" \
  -H 'Content-Type: application/json' \
  -d '{"force": false}'

# Performance monitoring
curl "http://localhost:8001/performance/health"

Development Tools

• VS Code Extension: Use Ctrl+Shift+P → "DocFoundry: Search Docs" for in-editor search
• Chrome Extension: Click the toolbar icon to save pages to docs/research/
• MCP Server: Connect compatible MCP clients for agent-based document access

🔧 Configuration

Environment Variables

# Database Configuration
DATABASE_URL=sqlite:///./docfoundry.db  # or postgresql://...
DATABASE_TYPE=sqlite  # or postgres

# API Configuration
API_HOST=0.0.0.0
API_PORT=8001
API_LOG_LEVEL=info

# Search Configuration
SEARCH_LIMIT_DEFAULT=10
SEARCH_LIMIT_MAX=100

# Crawling Configuration
CRAWL_DELAY_DEFAULT=1.0
CRAWL_MAX_DEPTH=5
CRAWL_RESPECT_ROBOTS=true

# Observability
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317
OTEL_SERVICE_NAME=docfoundry
ENABLE_METRICS=true

# Security Configuration
RATE_LIMIT_REQUESTS=100
RATE_LIMIT_WINDOW=60
CORS_ORIGINS=http://localhost:3000,http://localhost:8000
SECURE_HEADERS=true

# Performance Gates
PERFORMANCE_THRESHOLD_RESPONSE_TIME=2000
PERFORMANCE_THRESHOLD_ERROR_RATE=0.05
PERFORMANCE_MONITORING_ENABLED=true

# Lineage Tracking
LINEAGE_RETENTION_DAYS=90
INCREMENTAL_PROCESSING=true
CONTENT_HASH_ALGORITHM=sha256

Database Setup

SQLite (Default)

# No additional setup required
# Database file created automatically

PostgreSQL with pgvector

# Install PostgreSQL and pgvector extension
# Create database
createdb docfoundry

# Set environment variable
export DATABASE_URL="postgresql://user:password@localhost/docfoundry"
export DATABASE_TYPE="postgres"

# Run migrations
python -c "from server.postgres_adapter import PostgresAdapter; PostgresAdapter().create_tables()"

🚀 Deployment

Docker Deployment

# Build the image
docker build -t docfoundry .

# Run with SQLite
docker run -p 8001:8001 -v $(pwd)/data:/app/data docfoundry

# Run with PostgreSQL
docker run -p 8001:8001 \
  -e DATABASE_URL="postgresql://user:pass@host/db" \
  -e DATABASE_TYPE="postgres" \
  docfoundry

Production Considerations

• Reverse Proxy: Use nginx or similar for SSL termination and load balancing
• Process Management: Use systemd, supervisor, or container orchestration
• Monitoring: Enable OpenTelemetry for observability
• Backup: Regular database backups, especially for PostgreSQL
• Security: Configure proper authentication and rate limiting

🛣️ Roadmap

Current Features ✅

• Multi-source content ingestion (HTML, feeds, repositories)
• Hybrid search with semantic and keyword matching
• RESTful API with comprehensive endpoints
• VS Code and Chrome browser extensions
• SQLite and PostgreSQL support with pgvector
• OpenTelemetry observability integration
• MCP server protocol support
• Data lineage tracking with complete audit trails
• Incremental processing with hash-based change detection
• Enhanced security with rate limiting and input validation
• Performance monitoring with automated gates and alerts
• Production-ready deployment with Docker Compose
• Database migrations with Alembic integration

Planned Enhancements 🚧

• Advanced Workflow Orchestration: Temporal integration for complex pipelines
• Enhanced AI Integration: LLM-powered content summarization and tagging
• Enterprise Features: RBAC, advanced audit logging, and compliance tools
• Performance Optimizations: Distributed indexing and Redis caching layers
• Content Intelligence: Automatic content categorization and relationship mapping
• API Enhancements: GraphQL support and webhook integrations
• Advanced Analytics: Search behavior analysis and content recommendation engine
• Multi-tenant Support: Organization-based data isolation and access control

🤝 Contributing

We welcome contributions from the community! DocFoundry is built with collaboration in mind, and we appreciate all forms of contribution including code, documentation, bug reports, and feature suggestions.

Getting Started

Before Contributing:

1. Read our Code of Conduct
2. Check existing issues and discussions
3. Review our project roadmap to understand current priorities

Development Environment Setup

# 1. Fork and clone the repository
git clone https://github.com/YOUR_USERNAME/docfoundry.git
cd docfoundry

# 2. Set up development environment
make setup  # Installs dev dependencies and pre-commit hooks

# 3. Verify installation
make test   # Run test suite
make lint   # Check code quality

# 4. Start development server
make dev    # Starts API with auto-reload

Development Workflow

Code Quality Standards:

• Python Style: Follow PEP 8, enforced by black and flake8
• Type Safety: Add type hints for all new functions and classes
• Documentation: Include comprehensive docstrings following Google style
• Testing: Maintain >90% test coverage for new code
• Security: Follow OWASP guidelines, no hardcoded secrets

Commit Guidelines:

• Use conventional commits: feat:, fix:, docs:, test:, refactor:
• Keep commits atomic and well-described
• Reference issues in commit messages: fixes #123

Types of Contributions

🐛 Bug Reports

• Use the bug report template
• Include reproduction steps, expected vs actual behavior
• Provide system information and logs when relevant

✨ Feature Requests

• Use the feature request template
• Explain the use case and expected benefits
• Consider implementation complexity and maintenance burden

📝 Documentation

• Improve existing documentation clarity and accuracy
• Add examples and tutorials for common use cases
• Translate documentation to other languages

🔧 Code Contributions

• Start with "good first issue" labeled items
• Discuss major changes in issues before implementation
• Follow the pull request template requirements

Pull Request Process

1. Create Feature Branch

   git checkout -b feature/descriptive-name

2. Implement Changes

   • Write code following our standards
   • Add comprehensive tests
   • Update documentation as needed

3. Quality Checks

   make test     # Run full test suite
   make lint     # Check code quality
   make docs     # Verify documentation builds

4. Submit Pull Request

   • Use the provided PR template
   • Link related issues
   • Request review from maintainers

5. Review Process

   • Address reviewer feedback promptly
   • Keep PR scope focused and manageable
   • Ensure CI/CD checks pass

Recognition

Contributors are recognized in:

• CHANGELOG.md for each release
• GitHub contributors page
• Special mentions for significant contributions

For questions about contributing, join our GitHub Discussions or reach out to the maintainers.

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

🔧 Troubleshooting

Common Issues

Installation Problems

# Python version issues
python --version  # Should be 3.11+

# Virtual environment activation
# Windows
.venv\Scripts\activate
# macOS/Linux
source .venv/bin/activate

# Dependency conflicts
pip install --upgrade pip
pip install -r requirements.txt --force-reinstall

Database Issues

# SQLite permissions
chmod 664 docfoundry.db
chmod 775 $(dirname docfoundry.db)

# PostgreSQL connection
psql $DATABASE_URL -c "SELECT version();"

# Reset database
rm docfoundry.db  # SQLite only
python indexer/build_index.py

API Server Problems

# Port already in use
lsof -i :8001  # Find process using port
kill -9 <PID>  # Kill process

# Check server logs
uvicorn server.rag_api:app --log-level debug

# Test API health
curl http://localhost:8001/health

Search Issues

# Rebuild search index
python indexer/build_index.py --force

# Check index statistics
curl http://localhost:8001/stats

# Verify source configuration
python -c "from sources.loader import load_sources; print(load_sources())"

Performance Optimization

For Large Datasets:

• Use PostgreSQL with pgvector for production
• Enable database connection pooling
• Configure appropriate chunk sizes in source configs
• Monitor memory usage during indexing

Search Performance:

• Use specific source filters when possible
• Limit result counts for broad queries
• Consider semantic search for conceptual queries
• Use full-text search for exact term matching

Getting Help

If you encounter issues not covered here:

1. Check the FAQ section
2. Search existing issues
3. Create a new issue with detailed information
4. Join our community discussions for real-time help

🆘 Support & Community

Documentation & Resources

• 📚 Full Documentation: DocFoundry Docs (coming soon)
• 🎯 Quick Start Guide: See Quick Start section above
• 🔧 API Reference: Available at /docs endpoint when server is running
• 📋 Examples: Check the examples/ directory for usage patterns

Community Support

• 💬 GitHub Discussions: Join conversations
  • Q&A for usage questions
  • Feature discussions and feedback
  • Show and tell your implementations
• 🐛 Issue Tracker: Report bugs
• 📧 Email Support: support@docfoundry.dev (for security issues)

Response Times

• Community Support: Best effort, typically within 24-48 hours
• Bug Reports: Acknowledged within 48 hours, fix timeline depends on severity
• Security Issues: Acknowledged within 24 hours, patches prioritized

Contributing to Support

Help others by:

• Answering questions in discussions
• Improving documentation
• Sharing usage examples and tutorials
• Reporting and helping fix bugs

🙏 Acknowledgments

• Built with FastAPI for the REST API
• Powered by pgvector for vector similarity search
• Uses OpenTelemetry for observability
• Inspired by the need for intelligent documentation management

---

DocFoundry - Transform your documentation into an intelligent, searchable knowledge base. 🚀