Pulling changes from BYK WIP to RAG-17

[Thirunayan22]

This pull request introduces a comprehensive centralized logging and monitoring stack for the project, leveraging Grafana, Loki, and supporting services. The changes include new configuration files, Docker Compose service definitions, and documentation to enable real-time log aggregation, visualization, and alerting for deployment and model orchestration processes. The stack is designed to be easily integrated with Python services and supports advanced log filtering and monitoring via Grafana dashboards.

Logging and Monitoring Stack Setup

• Added docker-compose.yml with service definitions for Loki, Grafana, Qdrant, Langfuse (worker/web), Clickhouse, Minio, Redis, and Postgres, including health checks, persistent volumes, and network configuration for seamless orchestration.
• Introduced .env file with environment variables for all services, including credentials, endpoints, and configuration options for S3, Redis, Clickhouse, Grafana, and Langfuse.

Grafana and Loki Configuration

• Added grafana-configs/loki-config.yaml for Loki server setup (filesystem storage, schema, cache, and analytics opt-out), and grafana-configs/grafana-datasources.yaml to auto-provision Loki as the default Grafana datasource. [1] [2]
• Created grafana-configs/grafana-dashboards.yaml to organize dashboards under the "RAG Module" folder for easier navigation.

Monitoring Dashboard

• Added grafana-configs/grafana-dashboard-deployment.json with a ready-to-use Grafana dashboard for the RAG Module Orchestrator, featuring log count graphs by level/service and a real-time log viewer with filtering by service and log level.

Documentation

• Provided a detailed grafana-configs/README.md explaining the architecture, setup, usage, integration instructions for Python services, troubleshooting tips, and best practices for log monitoring and retention.