No user-record aggregation

[mattford63]

Hi,

Unlike the KPL there does not seem to be any userRecord aggregation. This means that we very quickly get throttled on events. Am I missing something? Without this the client is too expensive in terms of kinesis calls across the fleet at high demand.

[chaochenq]

The data aggregation feature is not currently available but is on our roadmap. Is that the main reason that you always got throttled in #29?

[mattford63]

Yeah pretty much.

[pcurry]

+1 please get to record aggregation. It should help with the cost arguments I'm about to have.

[rgarcia]

+1 record aggregation would be big for us

[nikegp]

Same here. Any news re the roadmap plans?

Thanks!

[cohi]

+1 Same here - interested in record aggregation in order to increase throughput.

[chaochenq]

For users who stream data for a Kinesis Firehose deliveryStream, we have this option that you can add to your fileFlow

"aggregatedRecordSizeBytes"

Agent will parse until the last line of your file to which the aggregated bytes is less than or equal to the specified aggregatedRecordSizeBytes.

See

https://github.com/awslabs/amazon-kinesis-agent/blob/master/src/com/amazon/kinesis/streaming/agent/tailing/AggregationSplitter.java

https://github.com/awslabs/amazon-kinesis-agent/blob/master/src/com/amazon/kinesis/streaming/agent/tailing/FirehoseFileFlow.java#L42

But we still don't have a solution to Kinesis data streams (kinesisStream) similar to KPL. Having Agent enable KPL is something that we haven't been able to prioritize.

[jeisinge]

Any update here?