Easily identify hot shard and hot key on your Kinesis data streams.
The Amazon Kinesis Hot Shard Advisor is a CLI tool that simplifies identifying whether you have hot key or hot shard issues on your Kinesis data streams. The tool can also identify whether you are hitting the shard level throughput limit per-second basis.
An Amazon Kinesis Data Stream consists of one or more shards. Records with the same partition key is always written to the same shard. Each shard has a maximum throughput it can support. When a producer application attempts to put records at a higher rate than the maximum throughput available, they receive
WriteThroughputExceeded error. This is commonly known as the hot shard problem.
When you experience a hot shard problem, you will notice an increase of
WriteProvisionedThroughputExceeded metric in CloudWatch. However, for bursty workloads you may not notice the corresponding data in
IncomingBytes metrics due to per minute aggregation. You can get the visibility to bursty writes with khs because it reads the records from your stream and aggregate the record size per second based on
Typically when there's a hot shard problem, you can increase the number of shards in the stream to increase available throughput (or change the capacity mode of your stream to on-demand). Increasing the number of shards does not solve the problem if it's caused by the producer using a key (or set of keys) more frequently than others. This is known as the hot key problem and requires an adjutment to parition key scheme to solve it. khs helps you identify hot key issues by showing the frequency of keys that are written to your stream within a given time period.
In order to perform analysis on your Kinesis Data Stream, khs must be executed under the context of an IAM entity with access to the following actions.
** khs will use the AWSCli configuration to access Kinesis data streams on a particular AWS region. If the default region is not configured in the current session, you must also specify the region of the Kinesis Data Stream via AWS_DEFAULT_REGION environment variable.
- download the compatible binary from releases and rename the file name to khs
- Use the command line and navigate to the directory where you downloaded the binary earlier
- run the below command
khs -stream=[YOUR STREAM NAME]] -from="yyyy-mm-dd hh:mm" -to="yyyy-mm-dd hh:mm"
Example: khs -stream=lab3 -from="2022-02-24 10:07" -to="2022-02-24 10:09"
Note: The date range should be within the retention period of your Kinesis data streams.
Listing shards for stream lab3...OK!
10 / 10 [====================================================] 100.00% 2m52s
output is written to out.html
Once you see the output as above, open the file pointed by
out option (by default this is set to out.html file from your current directory) with your default browser to view the report.
Usage of khs:
Use count-min-sketch (Optional) algorithm for counting key distribution (Optional). Default is false. Use this method to avoid OOM condition when analysing busy streams with high cardinality.
Start time in yyyy-mm-dd hh:mm format (Optional). Default value is current time - 5 minutes.
Number of keys to output in key distribution graph (Optional). (default 10)
Path to output file (Optional). Default is out.html. (default "out.html")
Comma separated list of shard ids to analyse.
End time in yyyy-mm-dd hh:mm format (Optional). Default value is current time.
Number of shards to emit to the report(Optional). Use 0 to emit all shards. Emitting all shards can result in a large file that may take a lot of system resources to view in the browser. (default 10)
If you are interested in contributing to this project, here are the steps you can follow.