A production-grade implementation for deploying and benchmarking distributed SQL query engines (Presto/Trino, ClickHouse, Spark) on Kubernetes with comprehensive fault tolerance, monitoring, and performance optimization.
- Deploy and benchmark distributed SQL query engines on Kubernetes clusters
- Simulate analytical workloads of 10TB+ scale for benchmarking query performance
- Ensure fault-tolerant, reproducible deployments with Infrastructure-as-Code and CI/CD
- Optimize for scalability, latency reduction, and reliability under failure conditions
�
‚ Kubernetes Cluster ‚
¤
‚ � � � ‚
‚ ‚ Presto ‚ ‚ ClickHouse ‚ ‚ Spark ‚ ‚
‚ ‚ Cluster ‚ ‚ Cluster ‚ ‚ Operator ‚ ‚
‚ ˜ ˜ ˜ ‚
¤
‚ � � � ‚
‚ ‚ Kafka ‚ ‚ MinIO/S3 ‚ ‚ Monitoring ‚ ‚
‚ ‚ (Ingestion) ‚ ‚ (Storage) ‚ ‚ Stack ‚ ‚
‚ ˜ ˜ ˜ ‚
¤
‚ � � � ‚
‚ ‚ Prometheus ‚ ‚ Grafana ‚ ‚ ELK/EFK ‚ ‚
‚ ‚ (Metrics) ‚ ‚ (Dashboards) ‚ ‚ (Logs) ‚ ‚
‚ ˜ ˜ ˜ ‚
˜
- Terraform >= 1.0
- kubectl >= 1.24
- helm >= 3.8
- Docker
- AWS CLI / Google Cloud SDK
-
Clone and Setup
git clone https://github.com/suhasramanand/distributed-query-engine-reliability.git cd distributed-query-engine-reliability -
Configure Cloud Provider
# For AWS export AWS_ACCESS_KEY_ID="your-access-key" export AWS_SECRET_ACCESS_KEY="your-secret-key" export AWS_REGION="us-west-2" # For GCP export GOOGLE_APPLICATION_CREDENTIALS="path/to/service-account.json" export GOOGLE_PROJECT="your-project-id"
-
Deploy Infrastructure
cd terraform terraform init terraform plan terraform apply -
Deploy Applications
cd ../helm ./deploy-all.sh -
Run Benchmarks
cd ../benchmarks ./run-tpch-benchmark.sh
terraform/ # Infrastructure as Code
‚ modules/
‚ ‚ eks/ # EKS cluster configuration
‚ ‚ networking/ # VPC, subnets, security groups
‚ ‚ storage/ # S3/GCS bucket configuration
‚ ‚ monitoring/ # CloudWatch/Stackdriver setup
‚ environments/
‚ dev/
‚ prod/
helm/ # Helm charts
‚ presto/ # Presto/Trino cluster
‚ clickhouse/ # ClickHouse cluster
‚ spark-operator/ # Spark operator
‚ kafka/ # Kafka for data ingestion
‚ minio/ # MinIO object storage
‚ monitoring/ # Prometheus, Grafana, ELK
benchmarks/ # Performance testing
‚ tpch/ # TPC-H benchmark scripts
‚ tpcds/ # TPC-DS benchmark scripts
‚ data-generator/ # Synthetic data generation
‚ results/ # Benchmark results
fault-tests/ # Chaos engineering
‚ chaos-mesh/ # Chaos Mesh experiments
‚ litmus/ # Litmus chaos experiments
‚ recovery-tests/ # Recovery time testing
.github/workflows/ # CI/CD pipelines
‚ terraform.yml # Infrastructure deployment
‚ helm-deploy.yml # Application deployment
‚ benchmarks.yml # Automated benchmarking
docs/ # Documentation
architecture.md # Detailed architecture
deployment.md # Deployment guide
troubleshooting.md # Troubleshooting guide
- Query Latency: 40% reduction through optimization
- Recovery Time: 35% improvement in failover procedures
- Throughput: Support 10TB+ analytical workloads
- Availability: 99.9% uptime with fault tolerance
Each query engine is optimized for:
- Presto/Trino: Memory management, connector optimization
- ClickHouse: Merge tree settings, compression ratios
- Spark: Executor memory, shuffle optimization
- Horizontal Pod Autoscaler (HPA) for query engines
- Vertical Pod Autoscaler (VPA) for resource optimization
- Cluster autoscaling for node pools
- Query performance metrics
- Resource utilization
- Error rates and SLAs
- Throughput and latency trends
- Query timeout alerts
- Resource exhaustion warnings
- SLA violation notifications
- Cluster health status
- TPC-H queries (1GB to 10TB scale)
- TPC-DS workload simulation
- Custom analytical queries
- Pod restart scenarios
- Node failure simulation
- Network latency injection
- Storage failure testing
- Fork the repository
- Create a feature branch
- Make your changes
- Add tests and documentation
- Submit a pull request
MIT License - see LICENSE file for details