Skip to content

sadsfae/ansible-elk

Folders and files

NameName
Last commit message
Last commit date

Latest commit

Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 

Repository files navigation

ansible-elk

Ansible Playbook for setting up the ELK/EFK Stack and Filebeat client on remote hosts

ELK

GA

What does it do?

  • Automated deployment of a full 6.x series ELK or EFK stack (Elasticsearch, Logstash/Fluentd, Kibana)
    • 5.6 and 2.4 ELK versions are maintained as branches and master branch will be 6.x currently.
    • Uses Nginx as a reverse proxy for Kibana, or optionally Apache via apache_reverse_proxy: true
    • Generates SSL certificates for Filebeat or Logstash-forwarder
    • Adds either iptables or firewalld rules if firewall is active
    • Tunes Elasticsearch heapsize to half your memory, to a max of 32G
    • Deploys ELK clients using SSL and Filebeat for Logstash (Default)
    • Deploys rsyslog if Fluentd is chosen over Logstash, picks up the same set of OpenStack-related logs in /var/log/*
    • All service ports can be modified in install/group_vars/all.yml
    • Optionally install curator
    • Optionally install Elastic X-Pack Suite
    • This is also available on Ansible Galaxy

Requirements

  • RHEL7 or CentOS7 server/client with no modifications
  • RHEL7/CentOS7, Rocky or Fedora for ELK clients using Filebeat
  • ELK/EFK server with at least 8G of memory (you can try with less but 5.x series is quite demanding - try 2.4 series if you have scarce resources).
  • You may want to modify vm.swappiness as ELK/EFK is demanding and swapping kills the responsiveness.
    • I am leaving this up to your judgement.
echo "vm.swappiness=10" >> /etc/sysctl.conf
sysctl -p

Notes

  • Current ELK version is 6.x but you can checkout the 5.6 or 2.4 branch if you want that series
  • I will update this playbook for major ELK versions going forward as time allows.
  • Sets the nginx htpasswd to admin/admin initially
  • nginx ports default to 80/8080 for Kibana and SSL cert retrieval (configurable)
  • Uses OpenJDK for Java
  • It's fairly quick, takes around 3minutes on a test VM
  • Fluentd can be substituted for the default Logstash
    • Set logging_backend: fluentd in group_vars/all.yml
  • Install curator by setting install_curator_tool: true in install/group_vars/all.yml
  • Install Elastic X-Pack Suite for Elasticsearch, LogStash or Kibana via:
    • install_elasticsearch_xpack: true
    • install_kibana_xpack: true
    • install_logstash_xpack: true
    • Note: Deploying X-Pack will wrap your ES with additional authentication and security, Kibana for example will have it's own credentials now - the default is username: elastic and password: changeme

ELK/EFK Server Instructions

  • Clone repo and setup your hosts file
git clone https://github.com/sadsfae/ansible-elk
cd ansible-elk
sed -i 's/host-01/elkserver/' hosts
sed -i 's/host-02/elkclient/' hosts
  • If you're using a non-root user for Ansible, e.g. AWS EC2 likes to use ec2-user then set the follow below, default is root.
ansible_system_user: ec2-user
  • Run the playbook
ansible-playbook -i hosts install/elk.yml

Create your Kibana Index Pattern

  • Next you'll login to your Kibana instance and create a Kibana index pattern.

ELK

  • Note: Sample data can be useful, you can try it later however.

ELK

ELK

ELK

ELK

  • At this point you can setup your client(s) to start sending data via Filebeat/SSL

ELK Client Instructions

  • Run the client playbook against the generated elk_server variable
ansible-playbook -i hosts install/elk-client.yml --extra-vars 'elk_server=X.X.X.X'
  • Once this completes return to your ELK and you'll see log results come in from ELK/EFK clients via filebeat

ELK

5.6 ELK/EFK (Deprecated)

  • The 5.6 series of ELK/EFK is also available, to use this just use the 5.6 branch
git clone https://github.com/sadsfae/ansible-elk
cd ansible-elk
git checkout 5.6

2.4 ELK/EFK (Deprecated)

  • The 2.4 series of ELK/EFK is also available, to use this just use the 2.4 branch
git clone https://github.com/sadsfae/ansible-elk
cd ansible-elk
git checkout 2.4
  • You can view a deployment video here:

Ansible Elk

File Hierarchy

.
β”œβ”€β”€ hosts
β”œβ”€β”€ install
β”‚Β Β  β”œβ”€β”€ elk_client.yml
β”‚Β Β  β”œβ”€β”€ elk.yml
β”‚Β Β  β”œβ”€β”€ group_vars
β”‚Β Β  β”‚Β Β  └── all.yml
β”‚Β Β  └── roles
β”‚Β Β      β”œβ”€β”€ apache
β”‚Β Β      β”‚Β Β  β”œβ”€β”€ tasks
β”‚Β Β      β”‚Β Β  β”‚Β Β  └── main.yml
β”‚Β Β      β”‚Β Β  └── templates
β”‚Β Β      β”‚Β Β      β”œβ”€β”€ 8080vhost.conf.j2
β”‚Β Β      β”‚Β Β      └── kibana.conf.j2
β”‚Β Β      β”œβ”€β”€ curator
β”‚Β Β      β”‚Β Β  β”œβ”€β”€ files
β”‚Β Β      β”‚Β Β  β”‚Β Β  └── curator.repo
β”‚Β Β      β”‚Β Β  β”œβ”€β”€ tasks
β”‚Β Β      β”‚Β Β  β”‚Β Β  └── main.yml
β”‚Β Β      β”‚Β Β  └── templates
β”‚Β Β      β”‚Β Β      β”œβ”€β”€ curator-action.yml.j2
β”‚Β Β      β”‚Β Β      └── curator-config.yml.j2
β”‚Β Β      β”œβ”€β”€ elasticsearch
β”‚Β Β      β”‚Β Β  β”œβ”€β”€ files
β”‚Β Β      β”‚Β Β  β”‚Β Β  β”œβ”€β”€ elasticsearch.in.sh
β”‚Β Β      β”‚Β Β  β”‚Β Β  └── elasticsearch.repo
β”‚Β Β      β”‚Β Β  β”œβ”€β”€ tasks
β”‚Β Β      β”‚Β Β  β”‚Β Β  └── main.yml
β”‚Β Β      β”‚Β Β  └── templates
β”‚Β Β      β”‚Β Β      └── elasticsearch.yml.j2
β”‚Β Β      β”œβ”€β”€ elk_client
β”‚Β Β      β”‚Β Β  β”œβ”€β”€ files
β”‚Β Β      β”‚Β Β  β”‚Β Β  └── elk.repo
β”‚Β Β      β”‚Β Β  └── tasks
β”‚Β Β      β”‚Β Β      └── main.yml
β”‚Β Β      β”œβ”€β”€ filebeat
β”‚Β Β      β”‚Β Β  β”œβ”€β”€ meta
β”‚Β Β      β”‚Β Β  β”‚Β Β  └── main.yml
β”‚Β Β      β”‚Β Β  β”œβ”€β”€ tasks
β”‚Β Β      β”‚Β Β  β”‚Β Β  └── main.yml
β”‚Β Β      β”‚Β Β  └── templates
β”‚Β Β      β”‚Β Β      β”œβ”€β”€ filebeat.yml.j2
β”‚Β Β      β”‚Β Β      └── rsyslog-openstack.conf.j2
β”‚Β Β      β”œβ”€β”€ firewall
β”‚Β Β      β”‚Β Β  β”œβ”€β”€ handlers
β”‚Β Β      β”‚Β Β  β”‚Β Β  └── main.yml
β”‚Β Β      β”‚Β Β  └── tasks
β”‚Β Β      β”‚Β Β      └── main.yml
β”‚Β Β      β”œβ”€β”€ fluentd
β”‚Β Β      β”‚Β Β  β”œβ”€β”€ files
β”‚Β Β      β”‚Β Β  β”‚Β Β  β”œβ”€β”€ filebeat-index-template.json
β”‚Β Β      β”‚Β Β  β”‚Β Β  └── fluentd.repo
β”‚Β Β      β”‚Β Β  β”œβ”€β”€ tasks
β”‚Β Β      β”‚Β Β  β”‚Β Β  └── main.yml
β”‚Β Β      β”‚Β Β  └── templates
β”‚Β Β      β”‚Β Β      β”œβ”€β”€ openssl_extras.cnf.j2
β”‚Β Β      β”‚Β Β      └── td-agent.conf.j2
β”‚Β Β      β”œβ”€β”€ heartbeat
β”‚Β Β      β”‚Β Β  β”œβ”€β”€ meta
β”‚Β Β      β”‚Β Β  β”‚Β Β  └── main.yml
β”‚Β Β      β”‚Β Β  β”œβ”€β”€ tasks
β”‚Β Β      β”‚Β Β  β”‚Β Β  └── main.yml
β”‚Β Β      β”‚Β Β  └── templates
β”‚Β Β      β”‚Β Β      └── heartbeat.yml.j2
β”‚Β Β      β”œβ”€β”€ instructions
β”‚Β Β      β”‚Β Β  └── tasks
β”‚Β Β      β”‚Β Β      └── main.yml
β”‚Β Β      β”œβ”€β”€ kibana
β”‚Β Β      β”‚Β Β  β”œβ”€β”€ files
β”‚Β Β      β”‚Β Β  β”‚Β Β  └── kibana.repo
β”‚Β Β      β”‚Β Β  β”œβ”€β”€ tasks
β”‚Β Β      β”‚Β Β  β”‚Β Β  └── main.yml
β”‚Β Β      β”‚Β Β  └── templates
β”‚Β Β      β”‚Β Β      └── kibana.yml.j2
β”‚Β Β      β”œβ”€β”€ logstash
β”‚Β Β      β”‚Β Β  β”œβ”€β”€ files
β”‚Β Β      β”‚Β Β  β”‚Β Β  β”œβ”€β”€ filebeat-index-template.json
β”‚Β Β      β”‚Β Β  β”‚Β Β  └── logstash.repo
β”‚Β Β      β”‚Β Β  β”œβ”€β”€ tasks
β”‚Β Β      β”‚Β Β  β”‚Β Β  └── main.yml
β”‚Β Β      β”‚Β Β  └── templates
β”‚Β Β      β”‚Β Β      β”œβ”€β”€ 02-beats-input.conf.j2
β”‚Β Β      β”‚Β Β      β”œβ”€β”€ logstash.conf.j2
β”‚Β Β      β”‚Β Β      └── openssl_extras.cnf.j2
β”‚Β Β      β”œβ”€β”€ metricbeat
β”‚Β Β      β”‚Β Β  β”œβ”€β”€ meta
β”‚Β Β      β”‚Β Β  β”‚Β Β  └── main.yml
β”‚Β Β      β”‚Β Β  β”œβ”€β”€ tasks
β”‚Β Β      β”‚Β Β  β”‚Β Β  └── main.yml
β”‚Β Β      β”‚Β Β  └── templates
β”‚Β Β      β”‚Β Β      └── metricbeat.yml.j2
β”‚Β Β      β”œβ”€β”€ nginx
β”‚Β Β      β”‚Β Β  β”œβ”€β”€ tasks
β”‚Β Β      β”‚Β Β  β”‚Β Β  └── main.yml
β”‚Β Β      β”‚Β Β  └── templates
β”‚Β Β      β”‚Β Β      β”œβ”€β”€ kibana.conf.j2
β”‚Β Β      β”‚Β Β      └── nginx.conf.j2
β”‚Β Β      β”œβ”€β”€ packetbeat
β”‚Β Β      β”‚Β Β  β”œβ”€β”€ meta
β”‚Β Β      β”‚Β Β  β”‚Β Β  └── main.yml
β”‚Β Β      β”‚Β Β  β”œβ”€β”€ tasks
β”‚Β Β      β”‚Β Β  β”‚Β Β  └── main.yml
β”‚Β Β      β”‚Β Β  └── templates
β”‚Β Β      β”‚Β Β      └── packetbeat.yml.j2
β”‚Β Β      └── xpack
β”‚Β Β          └── tasks
β”‚Β Β              └── main.yml
└── meta
    └── main.yml

56 directories, 52 files