README.md

Distributed Logging using external Elastic stack

Openshift comes with its Opensource EFK stack that is versions behind the new EFK stack and has less features. There is no security and alerting features. The Opensource version of Elastic does not have security and alerting built into it for free, but the OpenDistro version from Amazon has all of that. Follow the below steps to provision an external OpenDistro Elasticsearch and Kibana cluster.

Design

Review design (design/Centralized logging) on draw.io

High-level design

Network design

Prerequisite - Environment configuration

Docker log settings

https://docs.docker.com/config/containers/logging/configure/

• Set this in /etc/docker/daemon.json in all Openshift nodes to restrict log file count and size

    {
        "log-driver": "json-file",
        "log-opts": {
            "max-size": "10m",
            "max-file": "1"
        }
    }
  

• Restart docker service

• Repeat the above steps on all compute nodes

  • On master and infra, work on 1 node at a time.

Fluentd log settings

This is optional. The default is 10 fluentd log files each 1MB. Set the number of files to 5 each with max size 1MB.
oc set env ds/logging-fluentd LOGGING_FILE_AGE=5 LOGGING_FILE_SIZE=1024000"

Provision

• Got through these readme's to provision, configure and run Elasticsearch and Kibana
  • Elasticsearch
  • Kibana
• After Elasticsearch is up, configure Fluentd in Openshift
  • Fluentd

Data retention

Indices have to be deleted periodically based on the business retention policy and also to maintain optimum storage capacity needs and improve search speed. Curator is the Elasticsearch maintenance tool. It supports deleting indices out of many other actions.

• Curator configuration

OpenDistro Elasticsearch and Kibana Reference

• Reference
• Configuration
• Change admin password
• SSL configuration
• Discussion forum

Readme's

• Elasticsearch
• Fluentd
• Kibana
• Curator