Red Cell: Cluster Configuration & Deployment

This repository contains scripts and configurations used at Ognitio to automate the deployment process of our clusters. As we run Apache Mesos and a few of his frameworks for scheduling, we plan to continuously share publicly related templates and scripts through this directory.

Status

This project is undergoing heavy refactoring, hardening and testing and isn't even considered alpha.

Technology stack

We mainly rely on Ansible to configure our VMs.

The following list contains some of the projects that composes our stack:

• Consensus

  • Apache ZooKeeper: Distributed systems coordination

• Resource management

  • Apache Mesos

• Scheduling (i.e. Mesos frameworks)

  • Marathon: Long running services
  • Aurora: Long-running services and cron jobs
  • Chronos: Cron jobs
  • Kafka: Apache Kafka on Apache Mesos
  • Elasticsearch: Elasticsearch on Apache Mesos

• Logging and metrics

  • Transport
    • Telegraf: Plugin-driven server agent for reporting metrics
    • Fluentd: Open source data collector
  • Storage
    • InfluxDB: Distributed time series database
    • Elasticsearch: Distributed, RESTful Search Engine
  • Visualization
    • Kibana
    • Grafana: InfluxDB metrics on dashboard

• Container registry

  • Distribution: Docker toolset to pack, ship, store, and deliver content

• Service discovery and load balancing

  • HAProxy: TCP/HTTP Load Balancer
  • Mesos DNS: DNS-based service discovery for Mesos

• Application isolation

  • Weave: Network of Docker containers

Further work

• Enable SSL within Mesos and frameworks
• Authorization and authentication
• Secret files distribution
• Finer cloud providers deployment support (GCE, AWS, ...)
• Generate Ansible inventory file