Skip to content
kdb+ to Apache Kafka adapter, for pub/sub
Branch: master
Clone or download
sv Merge pull request #13 from sv/master
managing offsets api
Latest commit 9032b8e Nov 7, 2018
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
.travis.yml
LICENSE
Makefile
README.md Merge pull request #13 from sv/master Nov 7, 2018
k.h
kfk.c
kfk.q
socketpair.c initial release May 31, 2017
test_consumer.q
test_offsetc.q add example of working with offsets Sep 12, 2018
test_offsetp.q add example of working with offsets Sep 12, 2018
test_producer.q initial release May 31, 2017

README.md

kfk – Kafka for kdb+

kfk is a thin wrapper for kdb+ around librdkadka C API for Kafka. It is part of the Fusion for kdb+ interface collection.

Please report issues in this repository.

See code.kx.com/q/interfaces/kafka for full documentation.

Build and install

Step 1

Build and install the latest version of librdkafka. The minimum required version is v0.11.0.

Step 1

Build and install latest version of librdkafka. Minimum required version is v0.11.0.

Requirements

Follow requirements for librdkafka compilation noted on librdkafka page https://github.com/edenhill/librdkafka#requirements

To build 32-bit versions on 64-bit OS you need to have 32-bit versions of libraries and a toolchain.

#CentOS/RHEL
sudo yum install glibc-devel.i686 libgcc.i686 libstdc++.i686 zlib-devel.i686
# Ubuntu
sudo apt-get install gcc-multilib

Librdkafka

Package installation

macOS

brew install librdkafka

Ubuntu/Debian(unstable)

sudo apt-get install librdkafka-dev

RHEL/CentOS

sudo yum install librdkafka-devel

Building from source

macOS and Linux

git clone https://github.com/edenhill/librdkafka.git
cd librdkafka
make clean  # to make sure nothing left from previous build or if upgrading/rebuilding
# If using OpenSSL, remove --disable-ssl from configure command below
# On macOS with OpenSSL you might need to set `export OPENSSL_ROOT_DIR=/usr/local/Cellar/openssl/1.0.2k` before proceeding


# 32 bit
./configure --prefix=$HOME --disable-sasl --disable-lz4 --disable-ssl --mbits=32 
# 64 bits
./configure --prefix=$HOME --disable-sasl --disable-lz4 --disable-ssl --mbits=64

make
make install

Step 2

Compile and install a shared object (it will be installed to $QHOME/). Make sure you have QHOME environment set.

// in kfk source folder
make
make install

Note: If compiling dynamically linked libkfk.so make sure you have librdkafka.so.1 in your LD_LIBRARY_PATH.

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$HOME/lib

Documentation

See code.kx.com/q/interfaces/kafka.

https://docs.confluent.io/2.0.0/clients/consumer.html#synchronous-commits

To have launchd start kafka now and restart at login: brew services start kafka Or, if you don't want/need a background service you can just run: zookeeper-server-start /usr/local/etc/kafka/zookeeper.properties & kafka-server-start /usr/local/etc/kafka/server.properties

You can’t perform that action at this time.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.