Python API for the Kafka Message Queue
Python
Switch branches/tags
Nothing to show
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
examples
kafka
tests
.gitignore
LICENSE
README.md
TODO
setup.py

README.md

pykafka

pykafka allows you to produce messages to the Kafka distributed publish/subscribe messaging service.

Requirements

You need to have access to your Kafka instance and be able to connect through TCP. You can obtain a copy and instructions on how to setup kafka at https://github.com/kafka-dev/kafka

Installation

pip install pykafka

Usage

Sending a simple message

import kafka

producer = kafka.producer.Producer('test')
message  = kafka.message.Message("Foo!")
producer.send(message)

Sending a sequence of messages

import kafka

producer = kafka.producer.Producer('test')
message1 = kafka.message.Message("Foo!")
message2 = kafka.message.Message("Bar!")
producer.send([message1, message2])

Batching a bunch of messages using a context manager.

import kafka
producer = kafka.producer.Producer('test')

with producer.batch() as messages:
  print "Batching a send of multiple messages.."
  messages.append(kafka.message.Message("first message to send")
  messages.append(kafka.message.Message("second message to send")
  • they will be sent all at once, after the context manager execution.

Consuming messages one by one

import kafka
consumer = kafka.consumer.Consumer('test')
messages = consumer.consume()

Consuming messages using a generator loop

import kafka

consumer = kafka.consumer.Consumer('test')

for message in consumer.loop():
  print message

Contact:

Please use the GitHub issues: https://github.com/dsully/pykafka/issues