How to scale it

[ankurmitujjain]

Hi Team,

We are evaluating this connector library, as of now we are sending records in Kinesis at a rate of 3MP/s
and using this library we are saving data to S3.
Connector library works well when data ingestion is slow but as we increase speed, connector library continues with same speed of saving records...
Is there any way we can scale it out or auto scaling?

[darkcrawler01]

check these parameters -

1. whats the provisioned kinesis shard count ?
2. max_records per GetRecords ?
3. since the records are processed serially, how long does it take for s3 emitter to finish one batch ?

Using these, its possible to compute the max throughput of a shard, then provision as many shards as its required for ur use case. HTH