Operation Limit

[mjguisado]

Hi,

first, thank you for your code, it's very useful.

I have modified it because in my case I'm sending a lot of events to Kinesis but their payload are only a few bytes, and my application exceeds the number of records limit (1000 PUT Request/second) before the 1MB/s limit.

Now I can configure if your application has to monitorize the number of bytes or the number of records.

Maybe this modification is useful for someone else.

Bye.

[IanMeyers]

Hello,

Thanks for this, and I'll go through the details to see if a merge will work with some other stuff that's currently in development. However, for your use case I would highly recommend you start using the Kinesis Producer Library (http://docs.aws.amazon.com/kinesis/latest/dev/developing-producers-with-kpl.html) which is specifically designed for your use case. With this library you will be able to keep your event rate, or go much faster, without having to scale up the number of Shards in the stream.

Regards,

Ian

[mjguisado]

Hi Ian,
I know the KPL, but I can't use it in this case.
I take advantage of this e-mail to comment you another thing.
I have modified this method to avoid make a resize if the new number of shards is equal to the current one. I didn't request a push because I'm not pretty sure if here is the most right place.

com.amazonaws.services.kinesis.scaling.StreamScaler.doResize(String, int, Integer, Integer)

...
final int currentShards = StreamScalingUtils.getOpenShardCount(
kinesisClient, streamName);
if (currentShards == targetShardCount) {
LOG.info("Skipping scaling operation. The new number of shards is equal to the current one.");
return null;
}
final double pct = 1d / targetShardCount;
...

[IanMeyers]

I believe that this issue is now resolved with the push for monitoring put count as well as bytes