logtalez is a minimal command line client (and API) for retrieving log streams from the rsyslog logging daemon over zeromq.
Go

README.md

logtalez Build Status Doc Status

logtalez

Problem Statement

We want to tail logs from remote servers as conveniently as if they were local, in a safe and secure manner.

Solution

logtalez - a library and command line client for subscribing to log streams from rsyslog using the omczmq output plugin..

  • Create dynamic topics using rsyslog's parsing and template features.
  • Subscribe to topics to receive the logs you want.
  • Publisher side filtering keeps bandwidth usage low.
  • Brokerless design keeps operation simple.
  • Ephemeral streaming keeps things light weight.
  • CurveZMQ authentication and encryption keeps things secure.

Installation

Dependencies

libsodium

Version: 1.0.11 (or newer)

Sodium is a "new, easy-to-use software library for encryption, decryption, signatures, password hashing and more". ZeroMQ uses sodium for the basis of the CurveZMQ security protocol.

ZeroMQ

Version: 4.2.0 (or newer)

ZeroMQ is an embeddable ZMTP protocol library.

CZMQ

Version: 4.0.1 (or newer)

CZMQ is a high-level C binding for ZeroMQ. It provides an API for various services on top of ZeroMQ such as authentication, actors, service discovery, etc.

GoCZMQ

GoCZMQ is a Go interface to the CZMQ API.

Rsyslog

Version: 8.9.0 or newer

Rsyslog is the "rocket fast system for log processing". You will need to use the "--enable-omczmq" configure flag to build zeromq + curve support.

Generating Certificates

logtalez uses CURVE security certificates generated by the zcert API. They are stored in ZPL format. Logtalez includes a simple cert generation tool (curvecertgen) for convenience.

To generate a public / private key pair:

$ ./curvecertgen bogus_cert
Name: Brian
Email: bogus@whatever.com
Organization: Bogus Org
Version: 1

The above would generate a bogus_cert and bogus_cert_secret file.

Configuring Your Rsyslog Server

The following rsyslog configuration snippet consists of:

  • A template that dynamically sets a "topic" on a message consisting of hostname.syslogtag + an "@cee" cookie and JSON message payload
  • A rule snippet that attempts to parse a syslog message as JSON, then outputs it over a zeromq publish socket using the template
module(load="mmjsonparse")
module(load="omczmq")

template(name="pubsub_host_tag" type="list") {
  property(name="hostname")
  constant(value=".")
  property(name="syslogtag" position.from="1" position.to="32")
  constant(value="@cee:")
  constant(value="{")
  constant(value="\"@timestamp\":\"")
  property(name="timereported" dateFormat="rfc3339" format="json")
  constant(value="\",\"host\":\"")
  property(name="hostname")
  constant(value="\",\"severity\":\"")
  property(name="syslogseverity-text")
  constant(value="\",\"facility\":\"")
  property(name="syslogfacility-text")
  constant(value="\",\"syslogtag\":\"")
  property(name="syslogtag" format="json")
  constant(value="\",")
  property(name="$!all-json" position.from="2")
} 

ruleset(name="zmq_pubsub_out") {
  action(
    name="zmq_pubsub"
    template="pubsub_host_tag"
    type="omczmq"
    endpoints="tcp://*:24444"
    socktype="PUB"
    authtype="CURVESERVER"
    clientcertpath="/etc/curve.d/"
    servercertpath="/etc/curve.d/my_server_cert"
  )
}

action(type="mmjsonprase")
if $parsesuccess == "OK" then {
  call zmq_pubsub_out
} 

Usage

    import "github.com/digitalocean/logtalez"

    func main() {
        endpoints := []string{"tcp://127.0.0.1:24444,tcp://example.com:24444"}
        topics = []string{"host1.nginx","host2.nginx","host3.nginx"}

        serverCert := "/home/me/.curve/server_public_cert"
        clientCert := "/home/me/.curve/client_public_cert"

        lt, err := logtalez.New(endpoints, topics, serverCert, clientCert)
        if err != nil {
            panic(err)
        }

        buf := make([]byte, 65560)

        for {
            n, err := lt.Read(buf)
            if err != nil && err != io.EOF {
                panic(err)
            }
            fmt.Println(string(buf[:n]))
        }
    }

Tools That Work Well with Logtalez

  • jq JSON processor
  • humanlog "Logs for humans to read."
  • Anything that can read stdout!

GoDoc

godoc

License

This project uses the MPL v2 license, see LICENSE