ELK stack

[cytopia]

ELK stack

Goal

Be able to use ELK stack (Elastic Search, Logstash and Kibana)

DESCRIPTION

This PR adds configurable Elastic Search, Logstash and Kibana container (known as ELK stack) to the Devilbox.

• Refs: Submitting Elasticsearch Docker Example #360

[science695]

Sorry for not reviewing this earlier.

this looks great, however: Logstash requires that you put config files in a folder. This describes what data sources logstash should import into elastic. I would recommend keeping the data volume, but adding in a host mounted directory for this folder.

From my example in #360

-e "PATH_CONFIG=/etc/logstash/conf.d"
-v /etc/logstash/:/etc/logstash \

These config files also often list the path to a metadata file (i.e.: last updated timestamp or last id), in my setup I have it as /etc/logstash/meta/table_last_update.

[cytopia]

@science695 I must have missed that part. Do you have an example of a logstash configuration file that I can actually try out?

[science695]

Logstash is quite flexible, with many many different configuration types, from sql to logfiles to elastic beats ....

Here is Elastic's documentation on how to import apache access logs into an elastic index (very simple, almost too simple):

https://www.elastic.co/guide/en/logstash/current/config-examples.html#_processing_apache_logs

The file listed there would go in /etc/logstash/conf.d/apache-logs.conf

But the idea for this would be that conf.d folder is exposed to the host so we could put our own configs in, possibly also the /etc/logstash/meta/ folder if you are using the jdbc (SQL) input plugin.
( https://www.elastic.co/guide/en/logstash/current/plugins-inputs-jdbc.html#_usage )