Skip to content

Data collection and visualisation in Kibana for startups

Notifications You must be signed in to change notification settings

julienracine/gerboise

 
 

Repository files navigation

Gerboise: data collection and visualisation in Kibana for Pôle Emploi Startups

Gerboise is a project that allows app administrators to collect and visualise significant logs from their servers. It was designed to be really easy to setup; however, some familiarity with Kibana and Elasticsearch is required.

Kibana screenshot

There are two parts of Gerboise that need to be configured and launched. First, there is the log storage server that runs Elasticsearch and Kibana. Nginx with basic authentication runs in front of Elasticsearch. It is quite possible to run many Elasticsearch/Kibana instances for many startups on just one server: ports should not conflict. Then, log collection must be run on the application servers.

All services on both servers are configured to run inside docker containers. Thus, docker and docker-compose install need to be installed on all servers.

Install

Log storage and visualisation

Install system requirements (htpasswd and python):

apt install apache2-utils python

Generate configuration files for your application:

./configure.py myapp

The application ID will be used to generate ports for all services. Thus, it needs to be unique to this application.

The configuration files will then be located in apps/myapp. You can now run Nginx, Elasticsearch and Kibana with:

cd apps/myapp
docker-compose up

You should be able to open Kibana at http://localhost:56YOURAPPID. By default, Kibana comes with no dashboard or visualisation. You can import a few pre-configured dashboards and visualisation in Management 🠦 Saved Objects 🠦 Import 🠦 Select kibana.json. (NOTE: this might not work because you don't have the same index IDs, but it's worth trying)

Log collection

Copy the apps/myapp/docker-compose-fluentbit.yml generated by the configure.py myapp command to your application servers. This is a docker-compose file that collects CPU stats, memory usage and nginx access logs and sends them to your Elasticsearch instance. To start collecting logs, just run:

ES_PASSWORD=mypassword docker-compose -f docker-compose-fluentbit.yml

TODO

  • Let's encrypt certificates for nginx
  • run curator periodically to clean ES indices

About

Data collection and visualisation in Kibana for startups

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • Python 100.0%