The Apache Kafka C/C++ library
Switch branches/tags
Clone or download
edenhill test 0033: Fix "invalid regex" to be truly invalid
musl regex is list strict with certain undefined behaviour
regexes that glibc warns about.
Latest commit b677b45 Sep 17, 2018
Permalink
Failed to load latest commit information.
.github Issue template update Dec 11, 2017
examples rdkafka_zookeeper_example: fix conf_set("broker..") error string (#2002) Sep 14, 2018
mklove Added `make uninstall` Jun 19, 2018
packaging Add support for system C11 threads, e.g. from alpine/musl (#1998) Sep 17, 2018
src-cpp Bump version to v0.11.6 Sep 7, 2018
src Add support for system C11 threads, e.g. from alpine/musl (#1998) Sep 17, 2018
tests test 0033: Fix "invalid regex" to be truly invalid Sep 17, 2018
win32 Add support for system C11 threads, e.g. from alpine/musl (#1998) Sep 17, 2018
.appveyor.yml appveyor: Fix Win32OpenSSL 1.0.2p download link Aug 15, 2018
.dir-locals.el emacs default compile command Jan 7, 2017
.doozer.json Add mips build so we can test builds on big endian systems Nov 12, 2017
.gitignore Add msvcr120 for nuget packaging Oct 5, 2017
.travis.yml Add support for system C11 threads, e.g. from alpine/musl (#1998) Sep 17, 2018
CMakeLists.txt cmake build improve on windows (#1638 by @raulbocanegra) Jan 22, 2018
CODE_OF_CONDUCT.md Create CODE_OF_CONDUCT.md Oct 18, 2017
CONFIGURATION.md CONFIGURATION.md: changed '<' to 'before' Sep 11, 2018
CONTRIBUTING.md CONTRIBUTING: 8 space indent, not 4 Nov 15, 2016
Doxyfile Move statistics from wiki to STATISTICS.md May 31, 2018
INTRODUCTION.md Update batch.num.messages description Aug 25, 2018
LICENSE LICENSE date update May 31, 2018
LICENSE.crc32c Import of crc32c from http://stackoverflow.com/a/17646775/1821055 May 31, 2017
LICENSE.hdrhistogram Ported Coda Hale's Golang HdrHistogram to C May 31, 2018
LICENSE.lz4 Add lz4/lib sources for in-tree building May 10, 2017
LICENSE.murmur2 Refactor murmur2 code and add more unit tests (unaligned access) Jan 3, 2018
LICENSE.pycrc Extracted rdkafka from librd to its own library. Sep 19, 2012
LICENSE.queue Native Win32 (VS) support (take one) Mar 24, 2015
LICENSE.regexp regexp import from ccxvii/minilibs sha 875c33568b5a4aa4fb3dd0c52ea98f… Sep 28, 2016
LICENSE.snappy snappy-c upstream update: Clean import of andikleen/snappy-c@8015f2d Nov 5, 2015
LICENSE.tinycthread clean tinycthread import tinycthread.c at ce82058b05180ce, tinycthrea… Mar 23, 2015
LICENSE.wingetopt clean wingetopt import of wingetopt.c and wingetopt.h at version 281d… Mar 23, 2015
LICENSES.txt LICENSE date update May 31, 2018
Makefile Added `make uninstall` Jun 19, 2018
README.md Fix compilation error when ZLIB is not available Jul 5, 2018
README.win32 README.win32: SASL is supported now Sep 3, 2018
STATISTICS.md Update STATISTICS.md to match code (@ankon, #1936) Aug 12, 2018
configure mklove: fix ./configure --reconfigure Nov 24, 2016
configure.librdkafka Add support for system C11 threads, e.g. from alpine/musl (#1998) Sep 17, 2018
dev-conf.sh dev-conf: reorg -fsanitize Jun 5, 2018
lds-gen.py Dont include static functions in linker script (issue #662) May 18, 2016
mainpage.doxy Doxygenification: C API, C++ API stub, INTRODUCTION, CONFIGURATION an… Dec 8, 2015

README.md

librdkafka - the Apache Kafka C/C++ client library

Copyright (c) 2012-2018, Magnus Edenhill.

https://github.com/edenhill/librdkafka

Gitter chat Build status

librdkafka is a C library implementation of the Apache Kafka protocol, containing both Producer and Consumer support. It was designed with message delivery reliability and high performance in mind, current figures exceed 1 million msgs/second for the producer and 3 million msgs/second for the consumer.

librdkafka is licensed under the 2-clause BSD license.

For an introduction to the performance and usage of librdkafka, see INTRODUCTION.md

See the wiki for a FAQ.

NOTE: The master branch is actively developed, use latest release for production use.

Overview

  • High-level producer
  • High-level balanced KafkaConsumer (requires broker >= 0.9)
  • Simple (legacy) consumer
  • Compression: snappy, gzip, lz4
  • SSL support
  • SASL (GSSAPI/Kerberos/SSPI, PLAIN, SCRAM) support
  • Broker version support: >=0.8 (see Broker version compatibility)
  • Stable C & C++ APIs (ABI safety guaranteed for C)
  • Statistics metrics
  • Debian package: librdkafka1 and librdkafka-dev in Debian and Ubuntu
  • RPM package: librdkafka and librdkafka-devel
  • Gentoo package: dev-libs/librdkafka
  • Portable: runs on Linux, OSX, Win32, Solaris, FreeBSD, AIX, ...

Language bindings

Users of librdkafka

Usage

Requirements

The GNU toolchain
GNU make
pthreads
zlib-dev (optional, for gzip compression support)
libssl-dev (optional, for SSL and SASL SCRAM support)
libsasl2-dev (optional, for SASL GSSAPI support)

Instructions

Building

  ./configure
  make
  sudo make install

NOTE: See README.win32 for instructions how to build on Windows with Microsoft Visual Studio.

NOTE: See CMake instructions for experimental CMake build (unsupported).

Usage in code

See examples/rdkafka_example.c for an example producer and consumer.

Link your program with -lrdkafka -lz -lpthread -lrt.

Documentation

The public APIs are documented in their respective header files:

To generate Doxygen documents for the API, type:

make docs

Configuration properties are documented in CONFIGURATION.md

For a librdkafka introduction, see INTRODUCTION.md

Examples

See the examples/sub-directory.

Tests

See the tests/sub-directory.

Support

File bug reports, feature requests and questions using GitHub Issues

Questions and discussions are also welcome on irc.freenode.org, #apache-kafka, nickname Snaps.

Commercial support

Commercial support is available from Edenhill services