HelixQuery - YouTube Video Search Engine

A distributed, fault-tolerant semantic search engine for YouTube videos built with modern microservices architecture.

🚀 Quick Start

Prerequisites

• Docker & Docker Compose
• Python 3.11+
• Node.js 18+
• Rust 1.70+

Development Setup

1. Clone and setup the project:

   git clone <repository-url>
   cd HelixQuery
   ./setup.sh

2. Start development environment:

   ./scripts/dev-start.sh

3. Configure your environment:

   • Copy .env.example to .env
   • Add your YouTube API key
   • Update other configuration as needed

4. Access services:

   • API Documentation: http://localhost:8000/docs
   • MinIO Console: http://localhost:9001 (minioadmin/minioadmin)
   • Grafana: http://localhost:3001 (admin/admin)
   • Prometheus: http://localhost:9090

🏗️ Architecture

HelixQuery consists of six main microservices:

• Genesis Crawler: YouTube data collection and processing
• Pulse Indexer: Full-text search with Rust/Tantivy
• Vector Engine: Semantic search with sentence transformers
• Lexicon Service: NLP and query enhancement
• API Gateway: Request routing and result aggregation
• Web Interface: Modern SvelteKit frontend

🧪 Testing

# Run all tests
pytest

# Run integration tests
pytest tests/integration/

# Run performance tests
k6 run tests/performance/load-test.js

🚢 Deployment

Local Development

docker-compose -f docker-compose.dev.yml up -d

Production

# Deploy to AWS ECS
./scripts/deploy-aws.sh

# Deploy to Kubernetes
kubectl apply -f k8s/

📊 Monitoring

• Metrics: Prometheus + Grafana
• Logging: Structured JSON logging
• Tracing: OpenTelemetry integration
• Health Checks: Kubernetes-ready endpoints

🔒 Security

• Automated dependency scanning
• Container vulnerability scanning
• License compliance checking
• Secrets detection
• Infrastructure security scanning

📚 Documentation

• Implementation Guide
• Deployment Strategy
• API Documentation

🤝 Contributing

1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Run tests and linting
5. Submit a pull request

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.