This is some sample code of how to do centralised logging. So you can get something as awesome as this: http://demo.kibana.org/
- Logback logstash json encoder: write json format to your logs so you can handle multiline logging with ease, e.g. stack traces.
- Logstash forwarder: Lightweight client to forward logs, so you don't have to run the Logstash client in another jvm.
- Logstash: Logstash to receive your logs and dump them in elasticsearch.
- ElasticSearch: Store all the things.
- Kibana: Visualization. Query all the things.
- Apache reverse proxy: Kibana needs direct access to elasticsearch, so you want to redirect that through a reverse proxy for security.
This assumes you're working on Ubuntu servers. Multiple senders and a single receiver. Receiver will run Kibana too.
wget -O - http://packages.elasticsearch.org/GPG-KEY-elasticsearch | sudo apt-key add
sudo vim /etc/apt/sources.list.d/elasticsearch.list
deb http://packages.elasticsearch.org/logstash/1.3/debian stable main
deb http://packages.elasticsearch.org/elasticsearch/0.90/debian stable main
sudo apt-get update
sudo apt-get install elasticsearch logstash
Generate SSL certs with:
$ openssl req -x509 -batch -days 365000 -nodes -newkey rsa:2048 -keyout lumberjack.key -out lumberjack.crt
Drop 'logstash-receiver.conf' and your generated SSL .crt and .key files in ~/lumberjack
Drop 'upstart/logstash-receiver.conf' in /etc/init/. Change the 'deploy' user to the user you're using.
sudo service elasticsearch start
sudo service logstash-receiver start
Now you're ready to receive data.
- Use the provided logback.xml in your Java/Clojure apps. See clojure example logback integration for details.
- Dependency for Maven
<dependency> <groupId>net.logstash.logback</groupId> <artifactId>logstash-logback-encoder</artifactId> <version>2.4</version> </dependency>
- Dependency for Lein
- This will write your logs out in json format.
- Build and install Logstash forwarder. I recommend keeping a .deb on hand.
- Create ~/lumberjack
- Copy the same SSL certs you generated for the receiver and 'lumberjack.conf' into this directory.
- Adjust the paths and receiving server in 'lumberjack.conf'
- Drop 'upstart/lumberjack.conf' in /etc/init/. Change the 'deploy' user to the user you're using.
- sudo service lumberjack start
- Your logs should be sent to the receiver now.
I've included a basic apache config as 'apache.conf' which you can drop in /etc/apache2/sites-available and enable. This uses SSL and HTTP basic auth on everything. /elasticsearch path reverse proxies to elasticsearch running on port 9200. And /kibana loads Kibana from /var/www/kibana.