Follow instructions to create demo ELK stack in your environment.
For easier setup, I prepared
Vagrantfile (description of virtual machine) and
Ansible playbook (complete recipe for installing ELK stack on machine). You
will need Vagrant and
Ansible version 1.7.1. installed on your machine.
Then launch this two commands:
$ vagrant up # you will be asked for root password due to folder sharing $ ansible-playbook elk.yml
When they finish, you will have prepared Vagrant ELK machine.
Default Logstash configuration watching just
(further variants are described bellow in Modifications
If your initial situation is different (for example, you want to run playbook against your own server), take look at Configuration section below.
After succesfull installation, open your web browser and put there
http://192.168.99.200 URL. Enter credentials (
demo, if you not changed
Once you log in, you will see settings page asking you for selecting
index pattern. Select
@timestamp value in Time-field name field and click
on Create button.
If you now click on Discover item menu, you will see your first message in Kibana. Congratulation!
If you want run Ansible playbook agains different target than Vagrant box, you must setup SSH access and update several variables.
hosts.ini file is stored IP address of your target machine. Update it
according to your situation (on both lines).
remote_user-- username for SSH access
private_key_file-- path to your private SSH key
logger_kibana_auth_password-- credentials for HTTP Basic Authorization to Kibana web interface
For further details, see official Ansible documentation please.
During my talk I tweak configuration of Logstash several times to gradually present key features.
You will find those modification scripts in
/configs directory inside VirtualBox
machine. Just log in with
vagrant ssh command, go into
/configs folder and run as
user this scripts:
/etc/logstash/conf.d/logstash.conffile. From now it will consume not only
/etc/log/syslogfile, but also
/var/log/nginx/kibana.log. To be precise, script also change things around user/group acccess to Kibana log file.
/etc/logstash/conf.d/logstash.confnow contains rules for parsing Nginx log. After you run this script, you must go into Settings section in Kibana and update index (click on orange buttons with "recycle" arrows). Then you will see rich set of Nginx attributes on Discovery section.
Whole new Nginx configuration will be added into machine. It will serve simple static web site. Logstash configuration will be modified too, we need to observe new access log and make small modification in filter section.
Then you must run script
03_traffic.sh. It will invoke
traffic.pyscript, which will emulate traffic on our small static site. See documentation in this file for further information.
Last modification which wasn't covered in talk. It add new rules for Logstash, so data from Nginx will contain new attribute
extensionrepresenting requested file extension. I want demonstrate some subaggregation visualisations with this.