Kinesis Support for KPL and KCL Libraries

[jaymell]

Hello: I wanted to see whether anyone has investigated the feasibility of writing a Kinesis connector that supports AWS's Producer and Consumer libraries.

The advantage is that the producer library already supports aggregation of records both by batching records in requests and by aggregating multiple records into single requests. The consumer library supports the required de-aggregation as well as coordinating multiple consumers via checkpointing with DynamoDB.

We have been POC'ing Kafka with Akka streams apps for the past couple months at my company and are considering Kinesis as an alternative. Without the additional features that the KPL and KCL libraries offer, however, Kinesis is much less attractive.

[aserrallerios]

I've opened a PR of a Flow that uses the low-level sdk to PUT records. After that, I'll open a PR of a Source using KCL; i'ts already tested and implemented, I'm just finishing the migration to alpakka.

So the only missing piece would be a Flow that PUTs records using KPL. I guess I can have a look at it after finishing the KCL Source.

[aserrallerios]

The PR: #434

There's still work to do but feedback would be much appreciated :)

[jaymell]

@aserrallerios I was planning to work on the consumer side this week, so I will use your branch as a starting point. Thanks so much for the link!

[aserrallerios]

Oh, I'll keep working on it, just minor tweaks and near trivial tests are missing.

[jaymell]

@aserrallerios Sounds great. I will work to integrate our existing app using your branch -- will be glad to provide any feedback as I do so.

[eduardojld]

Hi @aserrallerios @jaymell , is there any progress on the sink stage with the put request, I have a working version that uses the sdk directly (no KPL) based on the sqs one, was about to rise the pull request when I bump into your ticket, I cant find a PR with that feature, is there one?

[aserrallerios]

Here it is the sink sdk PR (merged):
#408
KCL PR:
#434

[YikSanChan]

KCL support is released in 2.0.0. See https://akka.io/blog/news/2020/04/30/alpakka-2.0.0-released

[ennru]

Thank you for the ping.
Available as of 2.0.0.