go-kafka-counter

Count kafka events from multiple topics.

This small utility reads kafka messages from multiple topics and provides you a running count of the number of seen events. If the message is a json document, you can also break down the counts for a topic by a specific key indicated by a JSON path.

When you quit the program pressing Ctrl-C, it also provides some statistics of the events seen broken down by topics.

NOTE: This is a Golang rewrite of https://github.com/sandipb/kafka-counter which I abandoned because of my application packaging requirements.

Usage

$ ./kafka-count -h
Usage of ./kafka-count:
  -c, --config string      Path to YAML config file (default "config.yaml")
  -d, --debug              Run in debug mode
  -n, --max-count int      How many messages to process before exiting. < 0 means infinite. (default -1)
  -N, --no-progress        Do not show running progress. Only the stats at the end.
  -u, --update-every int   How many ms between display updates (default 1000)
pflag: help requested

Example config

common:
  broker: kafka.default.svc.cluster.local:9092

count:

  k8sNginxIngressLogRaw:
    name: ingress            # Use this name to refer to this topic
    path: kubernetes_cluster # break down the numbers by the string value at this jsonpath location

  k8sNginxModSecurityLogRaw:
    name: modsec
    countOnly: True # Only count the total events

Example Usage

Without real-time display (-N):

$ ./kafka-count -c config.yaml -n 1000 -N

Total events received: 1000
	                       ingress:     957 (95.70%)
	                    containerd:      36 (3.60%)
	                       systemd:       6 (0.60%)
	                     k8sevents:       1 (0.10%)

Details:
                  ingress[   957] (cluster1=510(53.3)%, cluster2=447(46.7)%)
               containerd[    36] (cluster2=22(61.1)%, cluster1=14(38.9)%)
                  systemd[     6] (cluster2=3(50.0)%, cluster1=3(50.0)%)
                k8sevents[     1] (cluster1=1(100.0)%)

Installation

From source

Needs librdkafka development packages for compilation.

$ git clone https://github.com/sandipb/go-kafka-counter.git
Cloning into 'go-kafka-counter'...
...

$ cd go-kafka-counter/

$ make
go build ./cmd/kafka-count

$ ./kafka-count -h

Cross compilation

Because of its cgo requirements, the binary has to be compiled on the target OS.

If you are ok using docker for compilation, you can get a Linux binary compiled for centos7 by running make builddockerbuild buildindocker.

To change the target OS version, you can edit Dockerfile.build.