Skip to content

aws-samples/kinesis-poster-worker

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

72 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

kinesis-poster-worker

A simple Python-based Kinesis Poster and Worker example (aka The Egg Finder)

Poster is a multi-threaded client that creates --poster_count poster threads to:

  1. generate random characters, and then
  2. put the generated random characters into the stream as records

Worker is a thread-per-shard client that:

  1. gets batches of records, and then
  2. seeks through the records for the word 'egg' or simply echoes the record to the screen

Multiple Poster or Worker clients can be run simultaneously to generate multi-threaded load on a Kinesis stream.


Getting Started

To get this example working with Python 2.7+, first install boto 2.23+ using:

$ pip install boto

and follow the instructions here to get your credentials setup for use. The credentials you use should permit at least these Kinesis actions: CreateStream, DescribeStream, GetRecords, GetShardIterator, ListStreams & PutRecord. Both the MergeShards and SplitShard actions are unused in this example.

Once boto is configured with your credentials run:

$ python poster.py my-first-stream

and the Poster will attempt to create the Kinesis stream named my-first-stream. In a matter of a few minutes the stream will have been created and you can again run:

$ python poster.py my-first-stream

again. The Poster will then use multiple threads to pump records into the stream.

Once the Poster is pumping records into the stream, then run:

$ python worker.py my-first-stream

which will start the Worker. The Worker will then begin reading records from the my-first-stream Kinesis stream.

For detailed help and configuration options: python poster.py --help ..or python worker.py --help, respectively.

Related Resources

About

Simple multi-threaded Kinesis Poster and Worker Python examples

Resources

License

Code of conduct

Security policy

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published