Kinesis - Helper method for batching put record request entries?

[Yeggstry]

Hi there,

Our team has recently written the following code for ensuring that the PutRecordsRequest we send to the AmazonKinesisClient does not exceed the maximum entry limit of 500.

Is there any reason why this kind of helper method isn't included in the SDK already? It would mean one less thing to worry about when putting the records onto the kinesis stream :)

private static final int KINESIS_MAX_BATCH_SIZE = 500;

private void putRecords( List<PutRecordsRequestEntry> putRecordsRequestEntries,
    String streamName )
{
    if ( !putRecordsRequestEntries.isEmpty() )
    {
        int size = putRecordsRequestEntries.size();
        int fullChunks = ( size - 1 ) / KINESIS_MAX_BATCH_SIZE;
        Stream<List<PutRecordsRequestEntry>> requestEntriesStream = IntStream.range( 0, fullChunks + 1 )
            .mapToObj( n -> putRecordsRequestEntries.subList( n * KINESIS_MAX_BATCH_SIZE,
                n == fullChunks ? size : ( n + 1 ) * KINESIS_MAX_BATCH_SIZE ) );

        requestEntriesStream.forEach( recordsEntryList ->
        {
            putRecords( new PutRecordsRequest().withRecords( recordsEntryList ).withStreamName( streamName ) );
        } );
    }
}

Thanks,

Lewis

[shorea]

I really like the idea of providing better support for batched operations. I'd like to solve this generically for all services as many services have similar patterns for batch requests. We could even have some automatic retries on failed batches. We are currently working on a new major version of the Java SDK and I see this as being a great addition to it. I'll keep you posted as we discuss this internally but meanwhile I'll leave this open so others can +1 if they are interested.

[Yeggstry]

Awesome, thanks :)

[debora-ito]

Hi @Yeggstry,

the SDK team has reviewed the feature request list for V1 and decided to not implement this one, since they're concentrating efforts on V2 new features. It's still being considered for V2, you can track the referenced issue above. I'll go ahead and close this one.

Please feel free to comment on the V2 issue with your use case, and reach out if you have further questions.