Log transport that just works with zero effort.
JavaScript Erlang Perl Ruby
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
apps/zerolog
rel
.gitignore
LICENSE
Makefile
README.org
rebar
rebar.config
start.sh

README.org

Zerolog - Log transport that just works with zero effort.

Zerolog is a distributed, decentralized log transport originally developed as a replacement for Scribe.

Below, you will find the “quick start” directions for setting up and using Zerolog. For more information, browse the following files:

  • README: this file
  • apps/zerolog/src/ the source tree for Zerolog
  • rel/zerolog/etc/app.config: Zerolog configuration file

Design

For obvious reasons Zerolog uses a client-server architecture where clients sends the logs as Protocol Buffers over ZeroMQ or as HTTP POST to a REST service to one or many Zerolog servers. Zerolog can be setup in a cluster to avoid single point of failures. Also the ZeroMQ transport layer will keep track of send messages and even buffer to memory and then to disk if the Zerolog server can’t be reached.

Zerolog uses a clustered caching database (Riak) to buffer incoming log messages and when reaching a set threshold it will push the whole set to disco as one file.

Motivation

After struggling with Scribe to get it to compile and work with Hadoop for log transportation we felt that enough is enough. Sure these are complex issues but done right they should be easy enough to install and use. After looking into Chukwa, Flume and others we felt that we could accomplish basically the same but with 1/10th the code.

Around the same time we finally gave up on Hadoop and moved to Disco and ironically none of the above solutions work with Disco so rolling our own made all the more sense.

Quick Start

This section assumes that you have copy of the Zerolog source tree. If not just

$ git clone git://github.com/kivra/zerolog.git

To get started, you need to:

  1. Build Zerolog
  2. Configure Zerolog for your log analysis backend
  3. Start the Zerolog server
  4. Connect a client and store log data

Building Zerolog

Assuming you have a working Erlang installation, building Zerolog should be as simple as:

$ cd $ZEROLOG
$ make rel

Configuring Zerolog

Everything needed to configure Zerolog is in the rel/zerolog/etc/app.config file.

[
    {zerolog, [
        {enabled_backends, [zerolog_tty, zerolog_disco]},
        {enabled_receivers, [zerolog_rest, zerolog_zeromq]},
        {zerolog_tty, []},
        {zerolog_disco, [
                            % 64MB (64x1024x1024)
                            {threshold, 67108864},

                            % Which nodes in the cluster 
                            %{nodes, ['zerolog@127.0.0.1', 'zerolog@an.ot.her.ip']},

                            {master, "http://localhost:8989"},
                            {prefix, "0b"},
                            {tag, "data:zerolog"}
                        ]},
        {zerolog_zeromq, [{addr, "tcp://*:2121"}]},
        {zerolog_rest, [
                        {ip, "127.0.0.1"},
                        {port, 2122}
        ]}
    ]},

%% Nothing needs to be edited below.
%% ===================================================================
[...]
].

Zerolog is configured to use one or many backends, they are enabled when put under enabled_backends and configured in each of their entries. Zerolog can also use different receivers for incoming log messages, they are enabled when put under enabled_receivers.

Starting Zerolog

Once you have successfully built Zerolog, you can start the server with the following commands:

$ cd $ZEROLOG/rel/zerolog
$ bin/zerolog start

Connecting a client to Zerolog

Now that you have a functional server, let’s try storing some logs in it.

Zeromq example:

$ cd $ZEROLOG
$ erl -pa deps/*/ebin -pa apps/zerolog/ebin
Erlang R15B (erts-5.9) [source] [64-bit] [smp:2:2] [async-threads:0] [hipe] [kernel-poll:false]

Eshell V5.9  (abort with ^G)
1> zerolog_client:run(1).

zerolog_client is a small client test library with one function run/1 which takes one argument number of time to send, so zerolog_client:run(10) would send 10 messages.

REST example:

$ curl -X POST -d "testdata" http://127.0.0.1:2122/zerolog
ok

Setting up a Zerolog cluster:

Zerolog is designed and meant to be run in a cluster. When run in a cluster you can configure your client to send to any of the configured Zerolog servers. They will (when using the Disco backend) keep a clustered database in sync where all the log messages get stored until the specified threshold has been reached after which it pushes everything in to Disco and clears the pushed log messages from the database.

Setting up a Zerolog cluster is meant to be dirt simple and robust. When running in a cluster make sure you have correct ip adresses and erlang cookies set to be able to for the cluster to talk to each other. Editing the file rel/zerolog/etc/vm.args and change the parameters -name zerolog@127.0.0.1 and -setcookie zerolog to match the ip of your machineand secret cookie used to secure your erlang nodes.

With default settings in your app.config you can start up the first zerolog server in the cluster. Then you can go on to configure and start up the remaining machines in the Zerolog cluster. When starting up the first machine you should see something like:

=INFO REPORT==== 5-Jan-2012::14:02:51 ===
Starting Zerolog

=INFO REPORT==== 5-Jan-2012::14:02:51 ===
'zerolog@10.0.0.1' elected

On the next machine open app.config and edit the section below to match your cluster. Also, don’t forget to set the correct values in vm.args as well.

[
        {zerolog_disco, [
                            % Which nodes in the cluster 
                            {nodes, ['zerolog@10.0.0.1', 'zerolog@10.0.0.2']},
[...]
].

The nodes specified under {nodes, []} are the other nodes in the cluster. When you startup the second node you should something like:

=INFO REPORT==== 5-Jan-2012::14:02:51 ===
Starting Zerolog

=INFO REPORT==== 5-Jan-2012::14:02:51 ===
'zerolog@10.0.0.2' surrendered to 'zerolog@10.0.0.1'.

=INFO REPORT==== 5-Jan-2012::13:47:26 ===
Sent join request to Riak node: 'zerolog@10.0.0.1'

you can now execute riak_core_status:ringready(). which should print all the nodes in the database cluster.

Client libraries:

As of this writing these are the ready made options: