From ecf962f99f61229716f818b82d2cff8f1c16ecd6 Mon Sep 17 00:00:00 2001 From: Devender Gollapally Date: Tue, 19 Nov 2019 09:18:07 -0800 Subject: [PATCH] Create README.md --- README.md | 56 +++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 56 insertions(+) create mode 100644 README.md diff --git a/README.md b/README.md new file mode 100644 index 0000000..386f386 --- /dev/null +++ b/README.md @@ -0,0 +1,56 @@ +# aws-kinesis-demo + +Simple Demo App, when you run the unit test it will write 10 messages to a Kinesis data stream and read from it. + +## Before you run +Please `application.properties` in src/main/resources, add your aws credentials and change the kinesis stream. + +## AWS IAM Roles Required + +* AmazonDynamoDBFullAccess +* AmazonKinesisFullAccess + +Expected output when the test is run. +``` +INFO ... Received Message: {Time=1574119709211} +INFO ... Received Message: {Time=1574120113272} +INFO ... Received Message: {Time=1574120114302} +INFO ... Received Message: {Time=1574120115306} +INFO ... Received Message: {Time=1574120116308} +INFO ... Received Message: {Time=1574120117314} +INFO ... Received Message: {Time=1574120118317} +INFO ... Received Message: {Time=1574120119321} +INFO ... Received Message: {Time=1574120120322} +INFO ... Received Message: {Time=1574120121325} +``` + +## AWS Kinesis Streams Notes: +* Shards + * A stream consist of a one or more shards. + * Each shard represents a fixed unit of capacity. + * What this means is: the more capacity you need for a stream, the more shards you will need to add. + * You can increase/decrease the number of shards at any time. + * Each shard can support: + * 1000 writes/sec OR + * Write rate of 1 MB/sec OR + * 5 reads/sec OR + * Read rate of 2 MB/sec + +* Record + * A record consist of a sequence number, a partition key and data blob. + * Data blob can be upto 1 MB. + * Max retention period of 7 days, defaults to 24 hours. + * Can change the retention period after creating the stream + +* Behind the scenes + * Unlike Apache Kafka the AWS Kinesis does not provide out-of-the-box support for consumer groups. + * The support of this feature is implemented as a part of MetadataStore key for shard checkpoints in the KinesisMessageDrivenChannelAdapter + * And uses Amazon DynamoDB under the covers + * Amazon Kinesis Client Library (KCL) automatically creates an Amazon DynamoDB table for each Amazon Kinesis Application to track and maintain state information such as resharding events and sequence number checkpoints + * In addition the LockRegistry is used to ensure exclusive access to each shard. This way only one channel adapter in the same consumer group will consumer messages from a single shard in the stream it is configured for. + +* Links + * [Common Pitfalls](https://www.youtube.com/watch?v=MELPeni0p04) + * [Limits](https://aws.amazon.com/kinesis/data-streams/faqs/) + * [Spring Cloud Kinesis binder](https://github.com/spring-cloud/spring-cloud-stream-binder-aws-kinesis/blob/master/spring-cloud-stream-binder-kinesis-docs/src/main/asciidoc/overview.adoc) + * [Lessons Learnt](https://tech.trivago.com/2018/07/13/aws-kinesis-with-lambdas-lessons-learned/)