Infrastructure Elasticsearch Logstash Kibana

Cheminement de l'information

• lecture du log
• aiguillage vers HAProxy
• répartition de charge
• logstash shipper :
  • interception du message + écriture dans Redis
• logstash indexer :
  • lecture depuis Redis + typage du message
  • aiguillage vers HAProxy
• répartition de charge
• écriture sur le cluster ES
• recherche du client
• répartition de charge
• demande prise en charge par Kibana
• recherche exécutée sur le cluster Elasticsearch

Fichier de configuration rsyslog

• https://github.com/opendocnet/elk/blob/master/rsyslog.conf

Fichier de configuration HAProxy

• https://github.com/opendocnet/elk/blob/master/haproxy.conf

Supervision HAProxy

Nagios plugins :

• sessions : https://github.com/opendocnet/elk/blob/master/check_haproxy_sessions
• backend : https://github.com/opendocnet/elk/blob/master/check_haproxy_backend

Fichiers de configuration logstash

• logstash shipper : https://github.com/opendocnet/elk/blob/master/logstash_shipper.conf
• logstash indexer : https://github.com/opendocnet/elk/blob/master/logstash_indexer.conf

Fichiers démarrage et arrêt des logstash

• logstash shipper : https://github.com/opendocnet/elk/blob/master/logstash-shipper
• logstash indexer : https://github.com/opendocnet/elk/blob/master/logstash-indexer

Supervision de l'API logstash

• Logstash shipper API :
• nagios plugin : https://github.com/opendocnet/elk/blob/master/check_events_logstash_shipper
• url : http://X.X.X.X:9620/_node/stats/pipeline

"pipeline": {"events": {"duration_in_millis": 88141432,"in": 29675905,"filtered": 29675905,"out": 29675905},

• Logstash indexer API :
• nagios plugin : https://github.com/opendocnet/elk/blob/master/check_events_logstash_indexer
• url : http://X.X.X.X:9621/_node/stats/pipeline

"pipeline": {"events": {"duration_in_millis": 2084712,"in": 1244671,"filtered": 1244671,"out": 1244671},"plugins": {"inputs": [],"filters": [{"id": "6cb0de0bebbaaf981d8017a9e0a3222e42545d94-2","events": {"duration_in_millis": 293125,"in": 1244671,"out": 1244671},"matches": 1244552,"failures": 119,"patterns_per_field": {"message": 1},"name": "grok"}],

Fichier de configuration Redis

• https://github.com/opendocnet/elk/blob/master/redis.conf

Fichier de configuration Elasticsearch

• https://github.com/opendocnet/elk/blob/master/elasticsearch.conf

Supervision Elasticsearch

• https://github.com/opendocnet/elk/blob/master/check_elasticsearch_error
• https://github.com/opendocnet/elk/blob/master/check_elasticsearch_health
• https://github.com/opendocnet/elk/blob/master/check_elasticsearch_search_per_second
• https://github.com/opendocnet/elk/blob/master/check_elasticsearch_search_slowlog
• https://github.com/opendocnet/elk/blob/master/check_elasticsearch_stats
• https://github.com/opendocnet/elk/blob/master/check_elasticsearch_thread_pool