You know, for visualization of Flume Data utilizing Elasticsearch and Kibana.
Ruby Shell Python
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Failed to load latest commit information.

Flume Deployment Using Elasticsearch and Kibana3 in Vagrant

This is an example configuration of how to utilize apache flume with Elasticsearch as a sink, and then visualize its data using Kibana.

To run the configuration:

git clone
cd Flume-Deployment
vagrant up es worker

What it does

Deploys Elasticsearch and Kibana to Deploys Flume-NG, apache, and a custom app for generating random http responses to

How to add data from the worker to ES

I've put together a test script will make 10000 requests against this will generate 10000 random http responses that are logged in the apache log which is used as an exec source in flume. The entries are then transfer to in logstash format so kibana can interpret them visually.

To run the script:

vagrant ssh worker
python /vagrant/

How to visualize the data that is contained in ES with Kibana

Port 80 on the ES machine is port forwarded to 8080 on the host so Kibana can be reached from there.

Start with an unconfigured dashboard and then add the panels and queries. In particular for this case the histogram is a rather nice way to view the data.


Also it's a good idea to set a time filter. Top right hand corner.

Why did you do this

There are not that many resources available on how to properly deploy and setup flume with an es sink that can be used to visualize data.

What comes next

Replace the exec source of tailing the apache error log with any other source, and you now have a way to query data that is being generated by any source using Kibana.

Video Tutorial