Bro plugin that provides a ZeroMQ log writer.
Switch branches/tags
Nothing to show
Clone or download
JustinAzoff Merge pull request #1 from dopheide-esnet/topic/dopheide/2.7-btest-fix
Remove version from show-plugin btest in prep for 2.7
Latest commit b40c240 Nov 16, 2018

README

Bro ZeroMQ Log Writer Plugin
============================

A Bro log writer that sends logging output using ZeroMQ.

This plugin uses the ZeroMQ "publish-subscribe" pattern, where Bro is
the publisher and a user-provided program is the subscriber.

Bro publishes each log record as a ZeroMQ multi-part message containing
two parts:  the first part contains the Bro log path (e.g. "conn"), and the
second part contains the log record in JSON format.  The first message part
is needed so that the subscriber knows which type of log record is
being received, and to allow subscribing to logs based on the Bro log path.


Prerequisites
-------------

This Bro plugin should be compatible with any recent release of ZeroMQ
and Bro.

Before attempting to install this plugin, make sure you have installed
ZeroMQ (http://zeromq.org) and Bro.  You will also need to obtain
the Bro source and build Bro (the source and installed versions of Bro must be
the same).


Installing
----------

This plugin can be installed as a Bro package using the Bro Package Manager
(see http://bro-package-manager.readthedocs.io).

Alternatively, to build this plugin from source, use the following commands
(replace BRO_SRC_DIR with the top-level directory of the Bro source code):

    # ./configure --bro-dist=BRO_SRC_DIR
    # make
    # sudo make install

If the configure script fails to find ZeroMQ, then use the "--with-zmq"
option to specify the prefix directory of the ZeroMQ installation.

In order to verify that the plugin was installed correctly, run the
command "bro -N" and look for the ZeroMQ log writer in the output.


Configuration
-------------

In order to enable logging to ZeroMQ, you must ensure that the plugin helper
script gets loaded and you must specify a subscriber endpoint where logs will
be sent.

Here is an example that you can add to your local.bro (note that if you
installed this plugin with the Bro package manager and if you're
using "@load packages" then you must omit the "@load" directive in all
examples below).  Be sure to change the example hostname and TCP port number
to the correct values for your subscriber endpoint:

    @load NCSA/ZeroMQWriter
    redef LogZeroMQ::endpoint = "tcp://localhost:12345";

After running "broctl deploy", Bro will write all logs to the specified
endpoint using ZeroMQ.  Note that all logs will also be written to disk,
unless you've specifically configured Bro otherwise.

If you don't want to send all logs, then you can specify which ones to send.
For example:

    @load NCSA/ZeroMQWriter
    redef LogZeroMQ::endpoint = "tcp://localhost:12345";
    redef LogZeroMQ::send_logs += { HTTP::LOG, Files::LOG };

After running "broctl deploy", Bro will write the "HTTP" and "Files"
logs to the specified endpoint using ZeroMQ.

Instead of specifying which logs to send via ZeroMQ, one could instead specify
which logs to not send (all others will be sent).  To do this, just replace
"send_logs" with "excluded_log_ids":

    @load NCSA/ZeroMQWriter
    redef LogZeroMQ::endpoint = "tcp://localhost:12345";
    redef LogZeroMQ::excluded_log_ids += { Conn::LOG, DNS::LOG };


ZeroMQ Endpoints
----------------

A ZeroMQ endpoint always has the form "transport://address", where "transport"
is a transport protocol that ZeroMQ recognizes (such as "tcp" or "ipc"),
and "address" is a transport-specific address.  For the "tcp" transport
protocol, an address consists of a hostname or IP address followed by a colon,
and then a TCP port number.  For the "ipc" transport, an address is the
pathname to a UNIX domain socket.

Here are some examples of valid endpoints:  "tcp://localhost:1234",
"tcp://10.1.2.3:4444", "ipc:///var/tmp/mysocket".


Logging to Multiple Subscribers
-------------------------------

If you want to send logs to more than one subscriber, then you will need to
add log filters using the "add_filter" function, being careful to specify
a value for the subscriber endpoint in the "config" field of each log filter.
In the following example, a new filter is added to the DNS and HTTP log
streams:

    @load NCSA/ZeroMQWriter
    redef LogZeroMQ::endpoint = "tcp://localhost:12345";
    redef LogZeroMQ::send_logs += { HTTP::LOG, DNS::LOG, Files::LOG };

    event bro_init() &priority=-10
        {
        local remote_filter_dns: Log::Filter = [
            $name = "remote-zmq",
            $writer = Log::WRITER_ZEROMQ,
            $interv = 0 sec,
            $config = table(["endpoint"] = "tcp://10.1.2.3:44000")
        ];

        local remote_filter_http: Log::Filter = [
            $name = "remote-zmq",
            $writer = Log::WRITER_ZEROMQ,
            $interv = 0 sec,
            $config = table(["endpoint"] = "tcp://10.1.2.3:44000")
        ];

        Log::add_filter(DNS::LOG, remote_filter_dns);
        Log::add_filter(HTTP::LOG, remote_filter_http);
        }

After running "broctl deploy", Bro will write the "HTTP", "DNS", and "Files"
logs to a subscriber on the localhost, and the "HTTP" and "DNS" logs will
also be written to a subscriber on another host.

Keep in mind that when you add a log filter with the same log path as an
existing filter (this is applicable to the DNS and HTTP log streams in
this example), then Bro will append a string of the form "-N", where N is
an integer, to the end of the log path so that each filter has its own
unique log path (in the example above, the remote subscriber would see
log paths of "dns-2" and "http-2").


How to Avoid Losing Log Messages
--------------------------------

ZeroMQ will drop messages before a connection to a subscriber has been
successfully established.  This means if you start Bro before starting
a subscriber, then log records intended to be written via ZeroMQ will be
dropped until Bro is able to connect to the subscriber.  Therefore, in
order to avoid losing log records, it is important to make
sure your subscribers are running before attempting to start Bro.

Once a connection to a subscriber is established, then if it is interrupted,
ZeroMQ will queue unsent messages in memory and they will all be sent when the
connection is re-established.  The maximum number of messages that will be
queued in memory in such circumstances is called the high water mark (once
the limit is reached, messages are dropped).  The default value is 1000 but
can be changed like this:

    redef LogZeroMQ::zmq_hwm = 25000;

However, if Bro terminates before an interrupted connection is
re-established, then all unsent log records are discarded.  In order to
avoid this, make sure your subscribers are running before issuing
the BroControl "stop", "restart", or "deploy" commands.  You can specify
the maximum amount of time that Bro will wait before terminating until all
unsent logs are sent by specifying the linger time (specified in milliseconds):

    redef LogZeroMQ::zmq_linger = 3000;

In this example, if there are any unsent logs when Bro decides to
terminate, then Bro will wait up to 3 seconds before giving up and discarding
all remaining unsent logs.