A lightweight replacement for logstash indexer in Go
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Failed to load latest commit information.
route Modified to allow multiple instances of same file type. May 25, 2016
LICENSE Update license for PacketZoom Apr 7, 2016
README.md Fix typo in run instructions: "go build" instead of "go get" Apr 8, 2016
main.go Make Filebeat a first class input and put parser go into filebeat dir Apr 7, 2016


LogZoom - A Lumberjack => Logstash indexer in Go

LogZoom is a lightweight, Lumberjack-compliant log indexer based off the fine work of Hailo's Logslam. It accepts the Lumberjack v2 protocol, which is currently supported by [Elastic's Filebeat] (https://github.com/elastic/beats).

It was written with the intention of being a smaller, efficient, and more reliable replacement for logstash and Fluentd.

What does LogZoom do?

Like Logstash, LogZoom receives JSON data from Filebeat via the Lumberjack v2 protocol and inserts the data into different outputs. For example, let's say your application generated a JSON line for every event:

{"@timestamp":"2016-03-31T22:23:14+0000", "url": "http://www.google.com"}
{"@timestamp":"2016-03-31T22:25:14+0000", "url": "http://www.bing.com"}
{"@timestamp":"2016-03-31T22:26:14+0000", "url": "http://www.yahoo.com"}

As the diagram shows, you can then run a single process of LogZoom to receive this data and insert to Elasticsearch, S3, etc:

![LogZoom Basic Diagram](images/logzoom.png)

Unlike Logstash, however, LogZoom does not attempt to manipulate data in any shape or form. JSON data that arrives from Filebeat is directly sent to outputs as-is.

Many users commonly use Logstash by adding a grok filter, "currently the best way in logstash to parse crappy unstructured log data." LogZoom currently does NOT support this use case; it is designed for software applications that generate structured data directly.

For example, if you are trying to use Kibana, a frontend to Elasticsearch, you may need the @timestamp field, which Logstash typically inserts for you. With LogZoom, your application must generate this field in each JSON log line. The advantages of using this approach:

  1. LogZoom doesn't have to decode the JSON, insert a new field, and encode the JSON again. Logstash and Fluentd spend a fair amount of CPU time doing this.

  2. The application explicitly defines the semantics of @timestamp. When we used Logstash, we were confused that each record was stamped when the entry was received by the central Logstash process, not when it was generated by the client. This caused great confusion, as we would often see large gaps in data when the data was just marked with the wrong timestamp.

Supported IO


  • Filebeat (Lumberjack V2 Protocol)
  • Redis Message Queue


  • Redis Message Queue
  • TCP Streaming
  • WebSocket Streaming
  • Elasticsearch
  • S3

Getting Started

1. Create config

Create a YAML config file specifying the desired input and outputs. An example config can be found in examples/example.config.yml:

    ssl_crt: /etc/filebeat/filebeat.crt
    ssl_key: /etc/filebeat/filebeat.key
    host: :7201
    host: :7202
      - http://localhost:9200

2. Run the server

$ go build
$ $GOPATH/bin/logzoom -config=examples/example.config.yml
2016/04/07 20:22:50 Starting server
2016/04/07 20:22:50 Starting buffer
2016/04/07 20:22:50 Starting input filebeat
2016/04/07 20:22:50 Starting output tcp
2016/04/07 20:22:50 Starting output websocket
2016/04/07 20:22:50 Starting output elasticsearch
2016/04/07 20:22:50 Setting HTTP timeout to 1m0s
2016/04/07 20:22:50 Setting GZIP enabled: false
2016/04/07 20:22:50 Connected to Elasticsearch

Streaming logs via TCP

nc localhost 7201

Streaming logs via WebSocket

Connect to http://localhost:7202 in a browser.
A list of known sources will be displayed.

Elasticsearch support

Note that currently only Elasticsearch 1.x is supported. If you need 2.x support, I think it is just a matter of updating LogZoom to use Olliver Eilhard's 3.x client.