Vision

Description

Vision is a stack for monitoring and logging. It provides :

• Elasticsearch web interface : http://xxx:9200
• Kibana web interface : http://xxx:9393
• Grafana web interface : http://xxx:9999/
• InfluxDB web interface : http://xxx:8083
• Heka dashboard : http://xxx:4352

Installation

• Download and install a release :

    $ curl https://github.com/nlamirault/vision/releases/download/x.y.z/vision-x.y.z-linux.tar.gz
    $ tar zxvf vision-x.y.z
    $ cd vision-x.y.z
    $ ./init.sh
  

• Start services using Docker Compose:

    $ docker-compose up -d
  

• Start services using SystemD:

    $ cp -r vision-*.service /lib/systemd/system/
  

Usage

Monitoring servers : Elasticsearch/Kibana/Beats

• Install Topbeat and Packetbeat

• Launch Elasticsearch and Kibana services

• Loading the templates into Elasticsearch

    $ curl -XPUT 'http://localhost:9200/_template/packetbeat' \
        -d@beats/topbeat.template.json
  
    $ curl -XPUT 'http://localhost:9200/_template/packetbeat' \
        -d@beats/packetbeat.template.json
  

• Running topbeat metrics :

    $ topbeat -c beats/topbeat.yml
  

• Running packetbeat metrics :

    $ packetbeat -c beats/packetbeat.yml
  

• Testing the installation:

    $ curl -XGET 'http://localhost:9200/topbeat-*/_search?pretty'
    $ curl -XGET 'http://localhost:9200/packetbeat-*/_search?pretty'
  

• Loading Kibana dashboards:

    $ curl -L -O http://download.elastic.co/beats/dashboards/beats-dashboards-1.0.0.tar.gz
    $ tar zxvf beats-dashboards-1.0.0.tar.gz
    $ cd beats-dashboards-1.0.0
    $ ./load.sh
  

• Then open the Kibana website (http://localhost:9393), then select Topbeat index, and open Topbeat dashboard. Do same with Packetbeat index and dashboard.

Monitoring servers : Telegraf/InfluxDB/Grafana

• Install Telegraf

• Launch InfluxDB and Grafana services

• Running telegraf metrics :

    $ telegraf -config telegraf/telegraf.conf
  

• Then open the Grafana dashboard (http://localhost:9191) and import the Vision Telegraf dashboard from (grafana/grafana-telegraf.json)

• You could explore metrics into the InfluxDB UI on http://localhost:8083 with the query :

    SHOW MEASUREMENTS
  

Monitoring servers : Prometheus/Grafana

• Install Prometheus and Prometheus node exporter

• Launch services :

    $ ./prometheus -config.file=prometheus/vision.yml
    $ ./node_exporter
  

• Check Prometheus installation on : http://localhost:9090 and http://localhost:9090/consoles/node.html

• hen open the Grafana dashboard (http://localhost:9191) and import the Vision Prometheus dashboard from (grafana/grafana-prometheus.json)

Log analysis (Elasticsearch/Filebeat/Kibana)

• Install Filebeat

• Launch Elasticsearch and Kibana services

• Loading the Index Template into Elasticsearch

    $ curl -XPUT 'http://localhost:9200/_template/packetbeat' \
        -d@beats/filebeat.template.json
  

• Running filebeat metrics :

    $ filebeat -c beats/filebeat.yml
  

SystemD

You could use services files to launch Vision monitoring tools using SystemD.

$ sudo cp systemd/*.service /lib/systemd/system/
$ sudo systemctl enable vision-telegraf
$ sudo systemctl enable vision-topbeat
$ sudo systemctl enable vision-packetbeat

Development

• Creates a virtual machine called vision-dev for the development environment :

    $ ./docker-machine create -d virtualbox vision-dev
    $ eval "$(./docker-machine env vision-dev)"
  

• Check vision machine runnning :

    $ ./docker-machine ls
  

• Launch vision :

    $ ./docker-compose up
  

• Open your browser and navigate to the IP address associated with the vision virtual machine :

    $ ./docker-machine ip
  

• To see which environment variables are available to the web service, run:

    $ ./docker-compose run web env
  

Deployment

With our app running locally, we can now push this exact same environment to a cloud hosting provider with Docker Machine

Set your credentials in your environment :

$ source XXXXXXX.sh

Deploy a new instance :

$ docker-machine -D create -d digitalocean \
    --digitalocean-access-token $DIGITALOCEAN_TOKEN \
    vision-prod

Now we have two Machines running, one locally and one on Digital Ocean:

$ docker-machine ls
NAME            ACTIVE     DRIVER         STATE     URL
vision-dev      *          virtualbox     Running   tcp://w.x.y.z:2376
vision-prod                digitalocean   Running   tcp://a.b.c.d:2376

Set vision-prod as the active machine and load the Docker environment :

$ ./docker-machine active vision-prod
$ eval "$(./docker-machine env vision-prod)"

Finally, let's build the application in the Cloud :

$ ./docker-compose build
$ ./docker-compose up -d

Support

Feel free to ask question or make suggestions in our [Issue Tracker][].

License

See LICENSE for the complete license.

Changelog

A changelog is available.

Contact

Nicolas Lamirault nicolas.lamirault@gmail.com