Skip to content
q language bindings for Apache Kafka based on librdkafka
C
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
LICENSE
README.md
k.h
kafka.q
kafkaq.c

README.md

kafkaq

q language bindings for Apache Kafka based on librdkafka

Installation

  • Install librdkafka
  • Download/clone this repo and navigate to the kafkaq directory
  • On Linux, compile with the following:
gcc -O2 -fPIC -shared kafkaq.c -o kafkaq.so -lrdkafka -lpthread

Kakfa Installation

To test with a local instance of kafka (more details here - the box should have an up to date version of java installed)

> # Download kafka package
> wget http://apache.mirrors.nublue.co.uk/kafka/0.10.2.0/kafka_2.11-0.10.2.0.tgz
> # Extract package and navigate to directory
> tar -xzf kafka_2.11-0.10.2.0.tgz
> cd kafka_2.11-0.10.2.0
> # start zookeeper server
> bin/zookeeper-server-start.sh -daemon config/zookeeper.properties
> # start kafka server
> bin/kafka-server-start.sh -daemon config/server.properties
> # create a topic called 'test'
> bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test

Usage Example

The included q file kafka.q assumes the kafkaq.so library is available in the same folder In this example q is acting as both producer and consumer via the local kafka instance started above, a more realistic scenario q would be either producer or consumer but not both.

q)\l kafka.q
q)initconsumer[`localhost:9092;()]
q)initproducer[`localhost:9092;()]
q)kupd / print default definition for incoming data - ignore key, print message as ascii
{[k;x] -1 `char$x;}
q)subscribe[`test;0] / subscribe to topic test, partition 0
q)pub:{publish[`test;0;`;`byte$x]} / define pub to publish text input to topic test on partition 0 with no key defined
q)pub"hello world"
q)hello world

q)

Notes

  • Currently the library supports a single consumer thread subscribed to one topic at a time
You can’t perform that action at this time.