Skip to content
Simple multi-threaded Kinesis Poster and Worker Python examples
Branch: master
Clone or download
brettf Merge pull request #6 from anduslim/master
Allow for region name as input argument
Latest commit cbb6323 May 22, 2015
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
.gitignore
LICENSE.txt
NOTICE.txt
README.md
long-poster.sh
long-worker.sh
poster.py
worker.py

README.md

kinesis-poster-worker

A simple Python-based Kinesis Poster and Worker example (aka The Egg Finder)

Poster is a multi-threaded client that creates --poster_count poster threads to:

  1. generate random characters, and then
  2. put the generated random characters into the stream as records

Worker is a thread-per-shard client that:

  1. gets batches of records, and then
  2. seeks through the records for the word 'egg' or simply echoes the record to the screen

Multiple Poster or Worker clients can be run simultaneously to generate multi-threaded load on a Kinesis stream.


Getting Started

To get this example working with Python 2.7+, first install boto 2.23+ using:

$ pip install boto

and follow the instructions here to get your credentials setup for use. The credentials you use should permit at least these Kinesis actions: CreateStream, DescribeStream, GetRecords, GetShardIterator, ListStreams & PutRecord. Both the MergeShards and SplitShard actions are unused in this example.

Once boto is configured with your credentials run:

$ python poster.py my-first-stream

and the Poster will attempt to create the Kinesis stream named my-first-stream. In a matter of a few minutes the stream will have been created and you can again run:

$ python poster.py my-first-stream

again. The Poster will then use multiple threads to pump records into the stream.

Once the Poster is pumping records into the stream, then run:

$ python worker.py my-first-stream

which will start the Worker. The Worker will then begin reading records from the my-first-stream Kinesis stream.

For detailed help and configuration options: python poster.py --help ..or python worker.py --help, respectively.

Related Resources

You can’t perform that action at this time.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.