Implementing Efficient and Reliable Producers with the Amazon Kinesis Producer Library
This repository contains the code samples used in the AWS Big Data Blog post of the same name.
The code contains a single java package with various implementations of the base class
AbstractClickEventsToKinesis as described in the blog post. There is a driver class called
ClickEventsToKinesisTestDriver that can be executed to test out the implementations.
Maven is used for building and dependency management.
Running the Sample
You'll need to first create a stream to put data into. This stream should have a sufficient number of shards if you wish to try out the high-throughput implementations.
You can use any stream name and region, but you'll have to modify
AbstractClickEventsToKinesis to use the name and region you have chosen.
To execute the test driver:
mvn compile exec:java -Dexec.mainClass=com.amazonaws.services.kinesis.producer.demo.ClickEventsToKinesisTestDriver
You should see output that looks like this:
[INFO] --- exec-maven-plugin:1.4.0:java (default-cli) @ aws-blog-kinesis-producer-library --- 0 seconds, 0 records total, 0.00 RPS (avg last 10s) 1 seconds, 0 records total, 0.00 RPS (avg last 10s) 2 seconds, 4 records total, 0.40 RPS (avg last 10s) 3 seconds, 7 records total, 0.70 RPS (avg last 10s) 4 seconds, 12 records total, 1.20 RPS (avg last 10s) 5 seconds, 16 records total, 1.60 RPS (avg last 10s) 6 seconds, 22 records total, 2.20 RPS (avg last 10s) 7 seconds, 28 records total, 2.80 RPS (avg last 10s) 8 seconds, 31 records total, 3.10 RPS (avg last 10s) 9 seconds, 36 records total, 3.60 RPS (avg last 10s) 10 seconds, 46 records total, 4.60 RPS (avg last 10s) 11 seconds, 55 records total, 5.50 RPS (avg last 10s) ...
To try out a different implementation of the producer, find and change the following line in
// Change this line to use a different implementation final AbstractClickEventsToKinesis worker = new BasicClickEventsToKinesis(events);